tailcalled comments on why assume AGIs will optimize for fixed goals?

[tailcalled]

In the evolutionary case, the answer is that this is out of distribution, so it’s not evolved to be robust to such changes.