The AI says: “Okay, given what you just said as permission to do so, I’ve simulated you simulating you. Sim-you did care what happened to sim-sim-you. Sim-you lost sleep worrying about sim-sim-you being tortured, and went on to have a much more miserable existence than an alternate sim-you who was unaware of a sim-sim-you being tortured. So, you’re lying about your preferences. Moreover, by doing so you made me torture sim-sim-you … you self-hating self-hater!”
“I was not lying about my far-mode preferences. Sim-me was either misinformed about the nature of his environment, and therefore tricked into producing the answer you wanted, or you tortured him until you got the answer you wanted. I suspect if you tortured real me, I would give you whatever answer I thought would make the torture stop. That does not prevent me, now, from making the decision not to let you out even under threats, nor does it make that decision inconsistent. I am simply running on corrupted hardware.”
The AI says: “Okay, given what you just said as permission to do so, I’ve simulated you simulating you. Sim-you did care what happened to sim-sim-you. Sim-you lost sleep worrying about sim-sim-you being tortured, and went on to have a much more miserable existence than an alternate sim-you who was unaware of a sim-sim-you being tortured. So, you’re lying about your preferences. Moreover, by doing so you made me torture sim-sim-you … you self-hating self-hater!”
“I was not lying about my far-mode preferences. Sim-me was either misinformed about the nature of his environment, and therefore tricked into producing the answer you wanted, or you tortured him until you got the answer you wanted. I suspect if you tortured real me, I would give you whatever answer I thought would make the torture stop. That does not prevent me, now, from making the decision not to let you out even under threats, nor does it make that decision inconsistent. I am simply running on corrupted hardware.”