The most compelling-to-me argument I’ve seen in that vein is that human civilization is currently, even without AI, on a trajectory to demand more and more energy, and eventually that will involve doing things on a scale sufficient to significantly change the amount of sunlight that reaches the surface of the Earth.
Humans probably won’t do that, because we live here (though even there, emphasis on “probably”—we’re not exactly doing great in terms of handling climate change from accidentally changing the amount of CO2 in the atmosphere, and while that’s unlikely to be an existential threat it’s also not a very good sign for what will happen when humans eventually scale up to using 1000x as much energy).
An AI that runs on silicon can survive in conditions that humans can’t survive in, and so its long-term actions probably look bad for life on Earth unless it specifically cares about leaving the Earth habitable.
This argument probably holds even in the absence of a single coherent AI that seizes control of the future, as long as things-which-need-earthlike-conditions don’t retain enough control of the future.
My model is that the relevant analogy is not “human relationship with mice in general”, it’s with “human relationship with mice that live on a patch of ground where we want to build a chip fab, and also there’s nowhere else for the mice to go”.
A lot of my hope for “humans do not go extinct within the next 50 years” looks something like that, yeah (a lot of the rest is in “it turns out that language models are just straightforwardly easy to align, and that it’s just straightforwardly easy to teach them to use powerful tools”). If it turns out that “learn a heuristic that you should avoid irreversible actions that destroy complex and finely-tuned systems” is convergent that could maybe look like the “human reserve”.
There’s an anthropic argument that if that’s what the future looks like, most humans that ever live would live on a human reserve, and as such we should be surprised that we’re not. But I’m kinda suspicious of anthropic arguments.
Although it might be possible for various cyborg scenarios, where humans and AI co-exist, co-evolve, co-modify, etc., to follow the space expansion paradigm.
The most compelling-to-me argument I’ve seen in that vein is that human civilization is currently, even without AI, on a trajectory to demand more and more energy, and eventually that will involve doing things on a scale sufficient to significantly change the amount of sunlight that reaches the surface of the Earth.
Humans probably won’t do that, because we live here (though even there, emphasis on “probably”—we’re not exactly doing great in terms of handling climate change from accidentally changing the amount of CO2 in the atmosphere, and while that’s unlikely to be an existential threat it’s also not a very good sign for what will happen when humans eventually scale up to using 1000x as much energy).
An AI that runs on silicon can survive in conditions that humans can’t survive in, and so its long-term actions probably look bad for life on Earth unless it specifically cares about leaving the Earth habitable.
This argument probably holds even in the absence of a single coherent AI that seizes control of the future, as long as things-which-need-earthlike-conditions don’t retain enough control of the future.
My model is that the relevant analogy is not “human relationship with mice in general”, it’s with “human relationship with mice that live on a patch of ground where we want to build a chip fab, and also there’s nowhere else for the mice to go”.
Earth could be turned into one huge nature reserve. Analogous to what present day nature reserves are to mice.
A lot of my hope for “humans do not go extinct within the next 50 years” looks something like that, yeah (a lot of the rest is in “it turns out that language models are just straightforwardly easy to align, and that it’s just straightforwardly easy to teach them to use powerful tools”). If it turns out that “learn a heuristic that you should avoid irreversible actions that destroy complex and finely-tuned systems” is convergent that could maybe look like the “human reserve”.
There’s an anthropic argument that if that’s what the future looks like, most humans that ever live would live on a human reserve, and as such we should be surprised that we’re not. But I’m kinda suspicious of anthropic arguments.
Although it might be possible for various cyborg scenarios, where humans and AI co-exist, co-evolve, co-modify, etc., to follow the space expansion paradigm.