Honestly, I find it most likely that an FAI would kill all non-human animals, because sustaining multiple species with very different needs and preferences is inefficient and/or because of meta-ethical or psychological uncertainty about the value of non-humans.
If it wanted to maximize positive states of consciousness, it would probably kill all sentient beings and attempt to convert all the matter in the universe into beings that efficiently experience large amounts of happiness. I find it plausible that this would be a good thing. See here for more discussion.
I don’t find that unlikely. (I think I’m a little less confident than Eliezer that something CEV-like would produce values actual humans would recognize, from their own limited perspectives, as preferable. Maybe my extrapolations are extrapolateder, and he places harder limits on how much we’re allowed to modify humans to make them more knowledgeable and rational for the purpose of determining what’s good.)
But I’m less confident that a correctly-constructed (i.e., Friendly) CEV calculation would replace humans with something radically nonhuman, than that CEV would kill all or most non-humans. Humans care a lot more about themselves than about other species, and are less confident about non-human subjectivity.
Of course, I suppose the reverse is a possibility. Maybe some existing non-human terrestrial species has far greater capacities for well-being, or is harder to inflict suffering on, than humans are, and an FAI would kill humans and instead work on optimizing that other species. I find that scenario much less plausible than yours, though.
But I’m less confident that a correctly-constructed (i.e., Friendly) CEV calculation would replace humans with something radically nonhuman, than that CEV would kill all or most non-humans.
If a CEV did this then I believe it would be acting unethically—at the very least, I find it highly implausible that, among the hundreds of thousands(?) of sentient species, homo sapiens is capable of producing the most happiness per unit of resources. This is a big reason why I feel uneasy about the idea of creating a CEV from human values. If we do create a CEV, it should take all existing interests into account, not just the interests of humans.
It also seems highly implausible that any extant species is optimized for producing pleasure. After all, evolution produces organisms that are good at carrying on genes, not feeling happy. A superintelligent AI could probably create much more effective happiness-experiencers than any currently-living beings. This seems to be similar to what you’re getting at in your last paragraph.
If it wanted to maximize positive states of consciousness, it would probably kill all sentient beings and attempt to convert all the matter in the universe into beings that efficiently experience large amounts of happiness. I find it plausible that this would be a good thing. See here for more discussion.
I don’t find that unlikely. (I think I’m a little less confident than Eliezer that something CEV-like would produce values actual humans would recognize, from their own limited perspectives, as preferable. Maybe my extrapolations are extrapolateder, and he places harder limits on how much we’re allowed to modify humans to make them more knowledgeable and rational for the purpose of determining what’s good.)
But I’m less confident that a correctly-constructed (i.e., Friendly) CEV calculation would replace humans with something radically nonhuman, than that CEV would kill all or most non-humans. Humans care a lot more about themselves than about other species, and are less confident about non-human subjectivity.
Of course, I suppose the reverse is a possibility. Maybe some existing non-human terrestrial species has far greater capacities for well-being, or is harder to inflict suffering on, than humans are, and an FAI would kill humans and instead work on optimizing that other species. I find that scenario much less plausible than yours, though.
If a CEV did this then I believe it would be acting unethically—at the very least, I find it highly implausible that, among the hundreds of thousands(?) of sentient species, homo sapiens is capable of producing the most happiness per unit of resources. This is a big reason why I feel uneasy about the idea of creating a CEV from human values. If we do create a CEV, it should take all existing interests into account, not just the interests of humans.
It also seems highly implausible that any extant species is optimized for producing pleasure. After all, evolution produces organisms that are good at carrying on genes, not feeling happy. A superintelligent AI could probably create much more effective happiness-experiencers than any currently-living beings. This seems to be similar to what you’re getting at in your last paragraph.