But also: ants can actually do heaps of things we can’t, whereas (arguably) at some point that won’t be true for us relative to AI systems.
Devil’s advocate: by comparative advantage, even if the AI was strictly superior to humans at all tasks, it might still make sense for it to trade with humans.
By comparative advantage, the relevant threshold isn’t “AI can do everything strictly better than a human”; it’s “AI is able to kill the humans and use our matter-energy to build infrastructure that’s more useful than humanity”.
(Or “AI is able to kill the humans and the expected gain from trading with us is lower than the expected loss from us possibly shutting it down, building a rival AGI, etc.”)
How do you describe the impulse that leads humans around the world to collect antiques, attempt to preserve endangered species, re-enact past time periods, etc?
Whatever that is, there seems to be more of it in humans than in non-human animals. Why do you imagine we’d see less rather than more of it in something built to have more than we do of those traits which distinguish us from other creatures?
Also, should be noted that the value of human atoms is very small: these atoms constitute around 10e-20 of all atoms in Solar system. Any small positive utility of human existence would overweight atom’s usefulness.
Yeah. It’s conceivable you have an AI with some sentimental attachment to humans that leaves part of the universe as a “nature preserve” for humans. (Less analogous to our relationship with ants and more to charismatic flora and megafauna.)
Devil’s advocate: by comparative advantage, even if the AI was strictly superior to humans at all tasks, it might still make sense for it to trade with humans.
By comparative advantage, the relevant threshold isn’t “AI can do everything strictly better than a human”; it’s “AI is able to kill the humans and use our matter-energy to build infrastructure that’s more useful than humanity”.
(Or “AI is able to kill the humans and the expected gain from trading with us is lower than the expected loss from us possibly shutting it down, building a rival AGI, etc.”)
How do you describe the impulse that leads humans around the world to collect antiques, attempt to preserve endangered species, re-enact past time periods, etc?
Whatever that is, there seems to be more of it in humans than in non-human animals. Why do you imagine we’d see less rather than more of it in something built to have more than we do of those traits which distinguish us from other creatures?
Gwern’s “What Is The Collecting Mindset?” is relevant to your question.
Also, should be noted that the value of human atoms is very small: these atoms constitute around 10e-20 of all atoms in Solar system. Any small positive utility of human existence would overweight atom’s usefulness.
Yeah. It’s conceivable you have an AI with some sentimental attachment to humans that leaves part of the universe as a “nature preserve” for humans. (Less analogous to our relationship with ants and more to charismatic flora and megafauna.)
I think that there is a small instrumental value in preserving humans. They could be exchange with Alien friendly AI, for example.