If we solve the problem normally thought of as “misalignment”, it seems like this scenario would now go well.
This might be totally obvious, but I think it’s worth pointing out that even if we “solve misalignment”—which I take to mean solving the technical problem of intent alignment—Bob could still chose to deploy a business strategising AI, in which case this failure mode would still occur. In fact, with a solution to intent alignment, it seems Bob would be more likely to do this, because his business strategising assistant will actually be trying to do what Bob wants (help his business suceed).
It seems like if Bob deploys an aligned AI, then it will ultimately yield control of all of its resources to Bob. It doesn’t seem to me like this would result in a worthless future even if every single human deploys such an AI.
This might be totally obvious, but I think it’s worth pointing out that even if we “solve misalignment”—which I take to mean solving the technical problem of intent alignment—Bob could still chose to deploy a business strategising AI, in which case this failure mode would still occur. In fact, with a solution to intent alignment, it seems Bob would be more likely to do this, because his business strategising assistant will actually be trying to do what Bob wants (help his business suceed).
It seems like if Bob deploys an aligned AI, then it will ultimately yield control of all of its resources to Bob. It doesn’t seem to me like this would result in a worthless future even if every single human deploys such an AI.