I forget where, but I read a blog post that described these sorts of things as controlled experiments—you want to test one part of your decision apparatus, not have it confounded by all the others.
Also, you are anthropomorphizing the paperclipper AI. It would accept your bargain, but demand not just a handful but as many paperclips as you could be arm-twisted into making—where your pain at the expense and effort is just below where you’d prefer the billion deaths. And then it would exploit your exhausted state to stab you in the back anyway. It’s not psychopathic, it’s just incrementing a number by expedient means. You can’t negotiate with something that won’t stay bought.
What’s the goal of that controlled experiment? If my decision apparatus fails on Newcomb’s problem or the “true PD”, does it tell you anything about my real world behavior?
It tells us that your real world behavior has the potential to be inconsistent.
Many people carry a decision apparatus that consists of a mess of unrelated heuristics and ad hoc special-cases. Examining extreme cases is a tool for uncovering places where the ad hoc system falls down, so that a more general system can be derived from basic principles, preferably before encountering a real world situation where the flaws in the ad hoc system become apparent.
To my mind, a better analogy than “controlled experiment” would be describing these as decision system unit tests.
I forget where, but I read a blog post that described these sorts of things as controlled experiments—you want to test one part of your decision apparatus, not have it confounded by all the others.
Also, you are anthropomorphizing the paperclipper AI. It would accept your bargain, but demand not just a handful but as many paperclips as you could be arm-twisted into making—where your pain at the expense and effort is just below where you’d prefer the billion deaths. And then it would exploit your exhausted state to stab you in the back anyway. It’s not psychopathic, it’s just incrementing a number by expedient means. You can’t negotiate with something that won’t stay bought.
What’s the goal of that controlled experiment? If my decision apparatus fails on Newcomb’s problem or the “true PD”, does it tell you anything about my real world behavior?
It tells us that your real world behavior has the potential to be inconsistent.
Many people carry a decision apparatus that consists of a mess of unrelated heuristics and ad hoc special-cases. Examining extreme cases is a tool for uncovering places where the ad hoc system falls down, so that a more general system can be derived from basic principles, preferably before encountering a real world situation where the flaws in the ad hoc system become apparent.
To my mind, a better analogy than “controlled experiment” would be describing these as decision system unit tests.