I don’t think the idea is that happy moments are necessarily outweighed by suffering. It reads to me like it’s the idea that suffering is inherent in existence, not just for humans but for all life, combined with a kind of negative utilitarianism.
I think I would be very happy to see that first-half world, too. And depending on how we got it, yeah, it probably wouldn’t go wrong in the way this story portrays. But, the principles that generate that world might actually be underspecified in something like the ways described, meaning that they allow for multiple very different ethical frameworks and we couldn’t easily know in advance where such a world would evolve next. After all, Buddhism exists: Within human mindspace there is an attractor state for morality that aims at self-denial and cessation of consciousness as a terminal value. In some cases this includes venerating beings who vow to eternally intervene/remain in the world until everyone achieves such cessation; in others it includes honoring or venerating those who self-mummify through poisoning, dehydrating, and/or starving themselves.
Humans are very bad at this kind of self-denial in practice, except for a very small minority. AIs need not have that problem. Imagine if, additionally, they did not inherit the pacifism generally associated with Buddhist thought but instead believed, like medieval Catholics, in crusades, inquisitions, and forced conversion. If you train an AI on human ethical systems, I don’t know what combination of common-among-humans-and-good-in-context ideas it might end up generalizing or universalizing.
I don’t think the idea is that happy moments are necessarily outweighed by suffering. It reads to me like it’s the idea that suffering is inherent in existence, not just for humans but for all life, combined with a kind of negative utilitarianism.
I think I would be very happy to see that first-half world, too. And depending on how we got it, yeah, it probably wouldn’t go wrong in the way this story portrays. But, the principles that generate that world might actually be underspecified in something like the ways described, meaning that they allow for multiple very different ethical frameworks and we couldn’t easily know in advance where such a world would evolve next. After all, Buddhism exists: Within human mindspace there is an attractor state for morality that aims at self-denial and cessation of consciousness as a terminal value. In some cases this includes venerating beings who vow to eternally intervene/remain in the world until everyone achieves such cessation; in others it includes honoring or venerating those who self-mummify through poisoning, dehydrating, and/or starving themselves.
Humans are very bad at this kind of self-denial in practice, except for a very small minority. AIs need not have that problem. Imagine if, additionally, they did not inherit the pacifism generally associated with Buddhist thought but instead believed, like medieval Catholics, in crusades, inquisitions, and forced conversion. If you train an AI on human ethical systems, I don’t know what combination of common-among-humans-and-good-in-context ideas it might end up generalizing or universalizing.