An excellent video. But in my traditional role as “guy who is not embarrassed to ask stupid questions”, I have a nitpick:
I didn’t instantly see why the expected utility satisficer would turn itself into a maximiser.
After a few minutes of thinking I’ve got as far as:
Out of all the packets I could send, sending something that is a copy of me (with a tiny hack to make it a maximiser) onto another computer will create an agent which (since I am awesome) will almost certainly acquire a vast number of stamps.
Which runs into a personal-identity-type problem:
*I’m* an agent. If my goal is to acquire a fairly nice girlfriend, then the plan “create a copy of me that wants the best possible girlfriend”, doesn’t really help me achieve my goal. In fact it might be a bit of a distraction.
I’m pretty sure there’s a real problem here, but the argument in the video doesn’t slam the door on it for me. Can anyone come up with something better?
An excellent video. But in my traditional role as “guy who is not embarrassed to ask stupid questions”, I have a nitpick:
I didn’t instantly see why the expected utility satisficer would turn itself into a maximiser.
After a few minutes of thinking I’ve got as far as:
Out of all the packets I could send, sending something that is a copy of me (with a tiny hack to make it a maximiser) onto another computer will create an agent which (since I am awesome) will almost certainly acquire a vast number of stamps.
Which runs into a personal-identity-type problem:
*I’m* an agent. If my goal is to acquire a fairly nice girlfriend, then the plan “create a copy of me that wants the best possible girlfriend”, doesn’t really help me achieve my goal. In fact it might be a bit of a distraction.
I’m pretty sure there’s a real problem here, but the argument in the video doesn’t slam the door on it for me. Can anyone come up with something better?