it seems likely that most of the difference between people’s visions of the future stems from differing cultural/memetic backgrounds, character flaws, lack of information and time, etc.
Indeed, but our cultural background is the only thing that distinguishes us from cavemen. You can’t strip that off without eliminating much that we find of value. Also, take the luddite/technophile divide. That probably arises, in part, because of different innate abilities to perform technical tasks. You can’t easily strip that difference off without favouring some types of nuclear genetics over others.
Obviously, this isn’t a terminal problem—society today does its best to please some majority of the population—a superintelligence could act similarly—but ineviably there will be some who don’t like it. Some people don’t want a superintelligence in the first place.
It all seems rather hypothetical, anyway: this is the benevolent-agents-create-superintelligence-for-the-good-of-all-humanity scenario. I don’t list that as among the plausible outcomes on http://alife.co.uk/essays/the_awakening_marketplace/ Even the inventor of this scenario seems to assign it a low probability of playing out. The history of technology is full of instances of inventions being used to benefit the minorities best placed to take advantage of them. Is there any case to be made for such a utility function ever actually being used?
Indeed, but our cultural background is the only thing that distinguishes us from cavemen. You can’t strip that off without eliminating much that we find of value. Also, take the luddite/technophile divide. That probably arises, in part, because of different innate abilities to perform technical tasks. You can’t easily strip that difference off without favouring some types of nuclear genetics over others.
Obviously, this isn’t a terminal problem—society today does its best to please some majority of the population—a superintelligence could act similarly—but ineviably there will be some who don’t like it. Some people don’t want a superintelligence in the first place.
It all seems rather hypothetical, anyway: this is the benevolent-agents-create-superintelligence-for-the-good-of-all-humanity scenario. I don’t list that as among the plausible outcomes on http://alife.co.uk/essays/the_awakening_marketplace/ Even the inventor of this scenario seems to assign it a low probability of playing out. The history of technology is full of instances of inventions being used to benefit the minorities best placed to take advantage of them. Is there any case to be made for such a utility function ever actually being used?