The choice between an “averagist” and “totalist” model of optimal human welfare is a tough one. The averagist wants to maximize average happiness (or some such measure of welfare); the totalist wants to maximize total happiness. Both lead to unfortunate reductio arguments. Average human welfare can be improved by eliminating everyone who is below average. This process can be repeated successively until we have only one person left, the happiest man in the world. The totalist would proceed by increasing the population to the very edge of carrying capacity, so that everyone was just a small increment above being so unhappy that they would commit suicide.
Neither model seems to match with our intuitions. As Eliezer has frequently warned, anyone who may accidentally or intentionally create a super-intelligent Artificial Intelligence that might take over the world had better beware of it adopting one of these extremes, even if the intention is to program it to be beneficent.
I’m sure philosophers have considered these questions for centuries. I’d be curious to know if there is a principled model for optimal human happiness which does not conflict so violently with our moral instincts. Not to privilege our instincts unjustifiably; it’s possible that the AI might be right to adopt one of the views above, and we are wrong, with our muddled human thinking. I would feel better if there were a nice, consistent and relatively simple model which did not lead to a seemingly horrific outcome.
Average human welfare can be improved by eliminating everyone who is below average. This process can be repeated successively until we have only one person left, the happiest man in the world.
“Obvious” counterargument: If you kill everyone else, the happiest man in the world will become less happy.
Average human welfare can be improved by eliminating everyone who is below average.
Except that if they still believe their lives are worth living, then you are causing them disutility by violating their preference to survive. It also causes everyone everyone else disutility because they don’t want other people killed, and because they become worried about themselves or their families dying if they become unhappy. It also eliminates the future possibility of the killed people’s lives improving.
The choice between an “averagist” and “totalist” model of optimal human welfare is a tough one. The averagist wants to maximize average happiness (or some such measure of welfare); the totalist wants to maximize total happiness. Both lead to unfortunate reductio arguments. Average human welfare can be improved by eliminating everyone who is below average. This process can be repeated successively until we have only one person left, the happiest man in the world. The totalist would proceed by increasing the population to the very edge of carrying capacity, so that everyone was just a small increment above being so unhappy that they would commit suicide.
Neither model seems to match with our intuitions. As Eliezer has frequently warned, anyone who may accidentally or intentionally create a super-intelligent Artificial Intelligence that might take over the world had better beware of it adopting one of these extremes, even if the intention is to program it to be beneficent.
I’m sure philosophers have considered these questions for centuries. I’d be curious to know if there is a principled model for optimal human happiness which does not conflict so violently with our moral instincts. Not to privilege our instincts unjustifiably; it’s possible that the AI might be right to adopt one of the views above, and we are wrong, with our muddled human thinking. I would feel better if there were a nice, consistent and relatively simple model which did not lead to a seemingly horrific outcome.
“Obvious” counterargument: If you kill everyone else, the happiest man in the world will become less happy.
Except that if they still believe their lives are worth living, then you are causing them disutility by violating their preference to survive. It also causes everyone everyone else disutility because they don’t want other people killed, and because they become worried about themselves or their families dying if they become unhappy. It also eliminates the future possibility of the killed people’s lives improving.