I think in a world with multiple superintelligent agents that have read access to each others’ code, I expect that agents ‘change their own goals’ for the social signalling/bargaining reasons that Bostrom mentions. Although it’s unclear whether this would look more like spawning a new successor system with different values and architecture.
I think in a world with multiple superintelligent agents that have read access to each others’ code, I expect that agents ‘change their own goals’ for the social signalling/bargaining reasons that Bostrom mentions. Although it’s unclear whether this would look more like spawning a new successor system with different values and architecture.