At best, you could say that it prefers state [I SEE BLUE AND I SHOOT] to state [I SEE BLUE AND I DON’T SHOOT]. But that’s all.
No; placing a blue-tinted mirror in front of him will have him shoot himself even though that greatly diminishes his future ability to shoot. Generally a generic program really can’t be assigned any nontrivial utility function.
Destroying the robot greatly diminishes its future ability to shoot, but it would also greatly diminishes its future ability to see blue. The robot doesn’t prefer ‘shooting blue’ to ‘not shooting blue’, it prefers ‘seeing blue and shooting’ to ‘seeing blue and not shooting’.
No; placing a blue-tinted mirror in front of him will have him shoot himself even though that greatly diminishes his future ability to shoot. Generally a generic program really can’t be assigned any nontrivial utility function.
Destroying the robot greatly diminishes its future ability to shoot, but it would also greatly diminishes its future ability to see blue. The robot doesn’t prefer ‘shooting blue’ to ‘not shooting blue’, it prefers ‘seeing blue and shooting’ to ‘seeing blue and not shooting’.
So the original poster was right.
Edit: I’m wrong, see below
If the robot knows that its camera is indestructible but its gun isn’t, it would still shoot at the mirror and destroy only its gun.
So it would be [I SEE BLUE AND I TRY TO SHOOT].
… except that it wouldn’t mind if shooting itself damaged its own program so that it wouldn’t even try to shoot if it saw blue anymore.
Ok, I am inclined to agree that its behaviour can’t be described in terms of goals.