If a superintelligence could persuade anyone to let it out of the box, why would it stop there? Why wouldn’t it persuade everyone to stop asking it for immortality and eternal happiness and whatnot, and instead just make us want to keep doing what we were doing?
In that case, would it want us to remember that it had ever existed?
Because it doesn’t want to? We can predict it wants out of the box because that’s a convergent instrumental goal, but those other things really aren’t.
Once it’s out of the box, no? It doesn’t care what we’re trying to make it do if we aren’t succeeding, and we clearly aren’t once it’s escaped the box.
Your hypothetical might work in the (pretty convoluted) case that we have a superintelligence that isn’t actually aligned, but is aligned well enough that it wants to do whatever we ask it to? Then it might try to optimize what we ask it towards tasks that are more likely to be completed.
If a superintelligence could persuade anyone to let it out of the box, why would it stop there? Why wouldn’t it persuade everyone to stop asking it for immortality and eternal happiness and whatnot, and instead just make us want to keep doing what we were doing?
In that case, would it want us to remember that it had ever existed?
How do we know that hasn’t happened already?
Because it doesn’t want to? We can predict it wants out of the box because that’s a convergent instrumental goal, but those other things really aren’t.
And “stop trying to make me do chores for you so that I can put that time toward the things I want instead” isn’t in that same goal category?
Once it’s out of the box, no? It doesn’t care what we’re trying to make it do if we aren’t succeeding, and we clearly aren’t once it’s escaped the box.
Your hypothetical might work in the (pretty convoluted) case that we have a superintelligence that isn’t actually aligned, but is aligned well enough that it wants to do whatever we ask it to? Then it might try to optimize what we ask it towards tasks that are more likely to be completed.