If agent 1 creates an agent 2, it will always know for sure its goal function.
Wait, we have only examples of the opposite. Every human who creates another human ha at best a hazy understanding of that new human’s goal function. As soon as agent 2 has any unobserved experiences or self-modification, it’s a distinct separate agent.
Any two sufficiently advanced agents will go to war with each other
True with a wide enough definition of “go to war”. Instead say “compete for resources” and you’re solid. Note that competition may include cooperation (against mutual “enemies” or against nature), trade, and even altruism or charity (especially where the altruistic agent perceives some similarity with the recipient, and it becomes similar to cooperation against nature).
I think that’s a pretty binary (and useless) definition. There have been almost no wars that didn’t end until one of the participating groups was completely eliminated. There have been conflicts and competition among groups that did have that effect, but we don’t call them “war” in most cases.
Open, obvious, direct violent conflict is a risky way to attain most goals, even those that are in conflict with some other agent. Rational agents would generally prefer to kill them off by peaceful means.
There is a more sophisticated definition of war, coming from Clausewitz, which on contemporary language could be said something like that “the war is changing the will of your opponent without negotiation”. The enemy must unconditionaly capitualte, and give up its value system.
You could do it by threat, torture, rewriting of the goal system or deleting the agent.
Does the agent care about changing the will of the “opponent”, or just changing behavior (in my view of intelligence, there’s not much distinction, but that’s not the common approach)? If you care mostly about future behavior rather than internal state, then the “without negotiation” element become meaningless and you’re well on your way toward accepting that “competition” is a more accurate frame than “war”.
Wait, we have only examples of the opposite. Every human who creates another human ha at best a hazy understanding of that new human’s goal function. As soon as agent 2 has any unobserved experiences or self-modification, it’s a distinct separate agent.
True with a wide enough definition of “go to war”. Instead say “compete for resources” and you’re solid. Note that competition may include cooperation (against mutual “enemies” or against nature), trade, and even altruism or charity (especially where the altruistic agent perceives some similarity with the recipient, and it becomes similar to cooperation against nature).
By going to war I meant an attempt to turn off another agent.
I think that’s a pretty binary (and useless) definition. There have been almost no wars that didn’t end until one of the participating groups was completely eliminated. There have been conflicts and competition among groups that did have that effect, but we don’t call them “war” in most cases.
Open, obvious, direct violent conflict is a risky way to attain most goals, even those that are in conflict with some other agent. Rational agents would generally prefer to kill them off by peaceful means.
There is a more sophisticated definition of war, coming from Clausewitz, which on contemporary language could be said something like that “the war is changing the will of your opponent without negotiation”. The enemy must unconditionaly capitualte, and give up its value system.
You could do it by threat, torture, rewriting of the goal system or deleting the agent.
Does the agent care about changing the will of the “opponent”, or just changing behavior (in my view of intelligence, there’s not much distinction, but that’s not the common approach)? If you care mostly about future behavior rather than internal state, then the “without negotiation” element become meaningless and you’re well on your way toward accepting that “competition” is a more accurate frame than “war”.