There is reasoning hiding behind the points that seem magical to you. The AI will want our matter as resources. Avoiding processing Earth and everything in it for negentropy would require that it cares about us, and nobody knows how to train an AI that wants that.
The AI wouldn’t be oriented towards trying to find reasons for why keep the planet it started on habitable (or in one piece) for one particular species. It’s true that it’s possible the AI will discover some reasons for not killing us all that we can’t currently see, but that sounds like motivated reasoning to me (it’s also possible it will discover extra reasons to process Earth).
Other planets have more mass, higher insolation, lower gravity, lower temperature and/or rings and more (mass in) moons. I can think of reasons why any of those might be more or less desirable than the characteristics of Earth It is also possible that the AI may determine it is better off not to be on a planet at all. In addition, in a non- foom scenario, for defensive or conflict avoidance reasons the AI may wind up leaving Earth and once it does so may choose not to return.
That depends a lot on how it views the probe. In particular by doing this is it setting up a more dangerous competitor than humanity or not? Does it regard the probe as self? Has it solved the alignment problem and how good does it think it’s solution is?
No. Humans aren’t going to be the best solution. The question is whether they will be good enough that it would be a better use of resources to continue using the humans and focus on other issues.
It’s definitely possible that it will discover extra reasons to process Earth (or destroy the humans even if it doesn’t process Earth).
So, the interesting part is that it’s not enough that they’re a better source of raw material (even if they were) and better for optimizing (even if they were), because travelling to those planets also costs something.
I can think of reasons why any of those might be more or less desirable than the characteristics of Earth
So, we would need specific evidence that would cut one way but not another. If we can explain AI choosing another planet over Earth as well as we can explain it choosing Earth over another planet, we have zero knowledge.
2. This is an interesting point. I thought at first that it can simply set it up to keep synchronizing the probe with itself, so that it would be a single redundantly run process, rather than another agent. But that would involve always having to shut down periodically (so that the other half could be active for a while). But it’s plausible it would be confident enough in simply creating its copy and choosing not to modify the relevant parts of its utility function without some sort of handshake or metaprocedure. It definitely doesn’t sound like something that it would have to wait to completely solve alignment for.
3. That would give us a brief window during which humans would be tricked into or forced to work for an unaligned AI, after which it would kill us all.
There is reasoning hiding behind the points that seem magical to you. The AI will want our matter as resources. Avoiding processing Earth and everything in it for negentropy would require that it cares about us, and nobody knows how to train an AI that wants that.
This is just wrong. Avoiding processing Earth doesn’t require that the AI cares for us. Other possibilities include:
(1) Earth is not worth it; the AI determines that getting off Earth fast is better;
(2) AI determines that it is unsure that it can process Earth without unacceptable risk to itself;
(3) AI determines that humans are actually useful to it one way or another;
(4) Other possibilities that a super-intelligent AI can think of, that we can’t.
What does the negentropy on other planets have that Earth doesn’t, that will result in the AI quickly getting off Earth without processing it first?
Send a probe away from Earth and also undergo the 0.001% risk of being destroyed while trying to take over Earth.
In just the right way that would make it most profitable for the AI to use humans instead of some other solution?
The AI wouldn’t be oriented towards trying to find reasons for why keep the planet it started on habitable (or in one piece) for one particular species. It’s true that it’s possible the AI will discover some reasons for not killing us all that we can’t currently see, but that sounds like motivated reasoning to me (it’s also possible it will discover extra reasons to process Earth).
Other planets have more mass, higher insolation, lower gravity, lower temperature and/or rings and more (mass in) moons. I can think of reasons why any of those might be more or less desirable than the characteristics of Earth It is also possible that the AI may determine it is better off not to be on a planet at all. In addition, in a non- foom scenario, for defensive or conflict avoidance reasons the AI may wind up leaving Earth and once it does so may choose not to return.
That depends a lot on how it views the probe. In particular by doing this is it setting up a more dangerous competitor than humanity or not? Does it regard the probe as self? Has it solved the alignment problem and how good does it think it’s solution is?
No. Humans aren’t going to be the best solution. The question is whether they will be good enough that it would be a better use of resources to continue using the humans and focus on other issues.
It’s definitely possible that it will discover extra reasons to process Earth (or destroy the humans even if it doesn’t process Earth).
So, the interesting part is that it’s not enough that they’re a better source of raw material (even if they were) and better for optimizing (even if they were), because travelling to those planets also costs something.
So, we would need specific evidence that would cut one way but not another. If we can explain AI choosing another planet over Earth as well as we can explain it choosing Earth over another planet, we have zero knowledge.
2. This is an interesting point. I thought at first that it can simply set it up to keep synchronizing the probe with itself, so that it would be a single redundantly run process, rather than another agent. But that would involve always having to shut down periodically (so that the other half could be active for a while). But it’s plausible it would be confident enough in simply creating its copy and choosing not to modify the relevant parts of its utility function without some sort of handshake or metaprocedure. It definitely doesn’t sound like something that it would have to wait to completely solve alignment for.
3. That would give us a brief window during which humans would be tricked into or forced to work for an unaligned AI, after which it would kill us all.