Prices or Bindings? and to a lesser extent (although with simpler formal statement) Newcomb’s Problem and The True Prisoner’s Dilemma: show just how insanely alien the rational thing can be, even if it’s directed to your own cause. You may need to conscientiously avoid preventing the world destruction, not take free money, and trade a billion human lives for one paperclip.
Prices or Bindings? and to a lesser extent (although with simpler formal statement) Newcomb’s Problem and The True Prisoner’s Dilemma: show just how insanely alien the rational thing can be, even if it’s directed to your own cause. You may need to conscientiously avoid preventing the world destruction, not take free money, and trade a billion human lives for one paperclip.