Frank: Suppose FDT(situation) = “push the button”. Then all psychopaths die, which includes me. Suppose instead FDT(situation) = “don’t push the button”. Then no psychopaths die. Since I prefer living in a world with psychopaths to dying, FDT(situation) = “don’t push the button”.
The main controversial piece is from the problem specification: “Paul is quite confident that only a psychopath would press such a button.” I think this mixes up P(button|psychopath) and P(psychopath|button), but since the problem specification is our only source of how the button determines who is or isn’t a psychopath, it seems fine to trust it on that point.
Another related problem is one where there’s a button who kills everyone who would, given the option, press it. You might expect that such people are bad neighbors and prefer a world without them without having any way to act on that belief (and if you come to believe that FDT pushes that button, what it really means is that you shouldn’t be so confident people who would press the button are bad neighbors!).
[In general, your decision theory should save you from claims in the problem specification of the form “and then you make a bad decision”, but it can’t be expected to save you from having incorrect empirical beliefs.]
In my view, FDT handles the problem as follows:
The main controversial piece is from the problem specification: “Paul is quite confident that only a psychopath would press such a button.” I think this mixes up
P(button|psychopath)andP(psychopath|button), but since the problem specification is our only source of how the button determines who is or isn’t a psychopath, it seems fine to trust it on that point.Another related problem is one where there’s a button who kills everyone who would, given the option, press it. You might expect that such people are bad neighbors and prefer a world without them without having any way to act on that belief (and if you come to believe that FDT pushes that button, what it really means is that you shouldn’t be so confident people who would press the button are bad neighbors!).
[In general, your decision theory should save you from claims in the problem specification of the form “and then you make a bad decision”, but it can’t be expected to save you from having incorrect empirical beliefs.]