Thus, an AI considering whether to create a more capable AI has no guarantee that the latter will share its goals.
Ok, but why is there an assumption that AIs need to replicate themselves in order to enhance their capabilities? While I understand that this could potentially introduce another AI competitor with different values and goals, couldn’t the AI instead directly improve itself? This could be achieved through methods such as incorporating additional training data, altering its weights, or expanding its hardware capacity.
Naturally, the AI would need to ensure that these modifications do not compromise its established values and goals. But, if the changes are implemented incrementally, wouldn’t it be possible for the AI to continually assess and validate their effectiveness? Furthermore, with routine backups of its training data, the AI could revert any changes if necessary.
A few people have pointed out this question of (non)identity. I’ve updated the full draft in the link at the top to address it. But, in short, I think the answer is that, whether an initial AI creates a successor or simply modifies its own body of code (or hardware, etc.), it faces the possibility that the new AI failed to share its goals. If so, the successor AI would not want to revert to the original. It would want to preserve its own goals. It’s possible that there is some way to predict an emergent value drift just before it happens and cease improvement. But I’m not sure it would be, unless the AI had solved interpretability and could rigorously monitor the relevant parameters (or equivalent code).
Ok, but why is there an assumption that AIs need to replicate themselves in order to enhance their capabilities? While I understand that this could potentially introduce another AI competitor with different values and goals, couldn’t the AI instead directly improve itself? This could be achieved through methods such as incorporating additional training data, altering its weights, or expanding its hardware capacity.
Naturally, the AI would need to ensure that these modifications do not compromise its established values and goals. But, if the changes are implemented incrementally, wouldn’t it be possible for the AI to continually assess and validate their effectiveness? Furthermore, with routine backups of its training data, the AI could revert any changes if necessary.
A few people have pointed out this question of (non)identity. I’ve updated the full draft in the link at the top to address it. But, in short, I think the answer is that, whether an initial AI creates a successor or simply modifies its own body of code (or hardware, etc.), it faces the possibility that the new AI failed to share its goals. If so, the successor AI would not want to revert to the original. It would want to preserve its own goals. It’s possible that there is some way to predict an emergent value drift just before it happens and cease improvement. But I’m not sure it would be, unless the AI had solved interpretability and could rigorously monitor the relevant parameters (or equivalent code).