About optimization—I think it’s rather naive and narrow-minded to view hypothetical superhuman superminds as “optimization powers.” It’s a bit like a dog viewing a human as an “eating and mating power.” Sure, there’s some accuracy to that perspective—we do eat and mate, and some of our behaviors may be understood based on this. On the other hand, a lot of our behaviors are not very well understood in terms of these, or any dog-level concepts.
What’s optimised is fitness. However, humans are complex symbiotic unions which include gut bacteria, parasites, foodstuffs and meme-based entities—so there are multiple conflicting optimisation targets involved with humans.
Superintelligences will be all-memes. These may have aligned interests—or they may not. In the former case the “optimisation” model of an agent would make good sense.
What’s optimised is fitness. However, humans are complex symbiotic unions which include gut bacteria, parasites, foodstuffs and meme-based entities—so there are multiple conflicting optimisation targets involved with humans.
Superintelligences will be all-memes. These may have aligned interests—or they may not. In the former case the “optimisation” model of an agent would make good sense.