Pausing reading at “Paul’s simple argument” to jot this down: The expected values are identical when you’re conditioning on all the parent nodes of the action (i.e. you have full knowledge of your own decision-making process, in your decision-making process). But if you can’t do that, then it seems like EDT goes nuts—e.g. if there’s a button you won’t want to press, and you’re not conditioning on your own brain activity, then EDT might evaluate the expected utility of pressing the button by assuming you have a harmful seizure that makes you hit the button, since it just looks for the most likely route, irrespective of utility. This seems like it might be related to problems of embodied cognition—not conditioning on all parent nodes of the action leads the EDT algorithm to treat the seizure as “just part of the plan,” when in fact it’s a breakdown of the cognition implementing EDT in the first place.
After reading more of the post, what seems to be going on is exploiting the difference between the output of the decision algorithm and what gets counted as an action. Sticking with the seizure example—in the Dutch book scenario, the CDT agent happily buys “seizure insurance conditional on A=a.” Then when it’s making its choice to not press the button, it notices that it gets the highest utility from “sell back the seizure insurance and don’t press the button,” so it tries to do that—but some small percentage of the time it still has the seizure before accidentally pressing the button. I’m not sure that a decision theory that models the distinction between its choice and the “actual action” sells back the seizure insurance there. In fact, we could probably fix the problem entirely within CDT by modeling the decision algorithm output as the “actual action” and the intervening seizure a stochastic part of the environment.
Pausing reading at “Paul’s simple argument” to jot this down: The expected values are identical when you’re conditioning on all the parent nodes of the action (i.e. you have full knowledge of your own decision-making process, in your decision-making process). But if you can’t do that, then it seems like EDT goes nuts—e.g. if there’s a button you won’t want to press, and you’re not conditioning on your own brain activity, then EDT might evaluate the expected utility of pressing the button by assuming you have a harmful seizure that makes you hit the button, since it just looks for the most likely route, irrespective of utility. This seems like it might be related to problems of embodied cognition—not conditioning on all parent nodes of the action leads the EDT algorithm to treat the seizure as “just part of the plan,” when in fact it’s a breakdown of the cognition implementing EDT in the first place.
After reading more of the post, what seems to be going on is exploiting the difference between the output of the decision algorithm and what gets counted as an action. Sticking with the seizure example—in the Dutch book scenario, the CDT agent happily buys “seizure insurance conditional on A=a.” Then when it’s making its choice to not press the button, it notices that it gets the highest utility from “sell back the seizure insurance and don’t press the button,” so it tries to do that—but some small percentage of the time it still has the seizure before accidentally pressing the button. I’m not sure that a decision theory that models the distinction between its choice and the “actual action” sells back the seizure insurance there. In fact, we could probably fix the problem entirely within CDT by modeling the decision algorithm output as the “actual action” and the intervening seizure a stochastic part of the environment.