By comparative advantage, the relevant threshold isn’t “AI can do everything strictly better than a human”; it’s “AI is able to kill the humans and use our matter-energy to build infrastructure that’s more useful than humanity”.
(Or “AI is able to kill the humans and the expected gain from trading with us is lower than the expected loss from us possibly shutting it down, building a rival AGI, etc.”)
How do you describe the impulse that leads humans around the world to collect antiques, attempt to preserve endangered species, re-enact past time periods, etc?
Whatever that is, there seems to be more of it in humans than in non-human animals. Why do you imagine we’d see less rather than more of it in something built to have more than we do of those traits which distinguish us from other creatures?
By comparative advantage, the relevant threshold isn’t “AI can do everything strictly better than a human”; it’s “AI is able to kill the humans and use our matter-energy to build infrastructure that’s more useful than humanity”.
(Or “AI is able to kill the humans and the expected gain from trading with us is lower than the expected loss from us possibly shutting it down, building a rival AGI, etc.”)
How do you describe the impulse that leads humans around the world to collect antiques, attempt to preserve endangered species, re-enact past time periods, etc?
Whatever that is, there seems to be more of it in humans than in non-human animals. Why do you imagine we’d see less rather than more of it in something built to have more than we do of those traits which distinguish us from other creatures?
Gwern’s “What Is The Collecting Mindset?” is relevant to your question.