Chris asked me via PM, “I’m curious, have you written any posts about why you hold that position?”
I don’t think I have, but I’ll give the reasons here:
“evidential-style conditioning on a logically uncertain distribution” seems simpler / more elegant to me.
I’m not aware of a compelling argument for “causal-graph-style counterpossible reasoning”. There are definitely some unresolved problems with evidential-style UDT and I do endorse people looking into causal-style FDT as an alternative but I’m not convinced the solutions actually lie in that direction. (https://sideways-view.com/2018/09/30/edt-vs-cdt-2-conditioning-on-the-impossible/ and links therein are relevant here.)
Part of it is just historical, in that UDT was originally specified as “evidential-style conditioning on a logically uncertain distribution” and if I added my name as a co-author to a paper that focuses on causal-style decision theory, people would naturally wonder if something made me change my mind.
I’m actually still quite confused by the necessity of logical uncertainty for UDT. Most of the common problems like Newcomb’s or Parfit’s Hitchhiker don’t seem to require it. Where does it come in?
(The only reference to it that I could find was on the LW wiki)
I think it’s needed just to define what it means to condition on an action, i.e., if an agent conditions on “I make this decision” in order to compute its expected utility, what does that mean formally? You could make “I” a primitive element in the agent’s ontology, but I think that runs into all kinds of problems. My solution was to make it a logical statement of the form “source code X outputs action/policy Y”, and then to condition on it you need a logically uncertain distribution.
Hmm, I’m still confused. I can’t figure out why we would need logical uncertainty in the typical case to figure out the consequences of “source code X outputs action/policy Y”. Is there a simple problem where this is necessary or is this just a result of trying to solve for the general case?
Agents need to consider multiple actions and choose the one that has the best outcome. But we’re supposing that the code representing the agent’s decision only has one possible output. E.g., perhaps an agent is going to choose between action A and action B, and will end up choosing A. Then a sufficiently close examination of the agent’s source code will reveal that the scenario “the agent chooses B” is logically inconsistent. But then it’s not clear how the agent can reason about the desirability of “the agent chooses B” while evaluating its outcomes, if not via some mechanism for nontrivially reasoning about outcomes of logically inconsistent situations.
Do we need the ability to reason about logically inconsistent situations? Perhaps we could attempt to transform the question of logical counterfactuals into a question about consistent situations instead as I describe in this post? Or to put it another way, is the idea of logical counterfactuals an analogy or something that is supposed to be taken literally?
You can formalize UDT in a more standard game-theoretic setting, which allows many problems like Parfit’s Hitchhiker to be dealt with, if that is enough for what you’re interested in. However, the formalism assumes a lot about the world (such as the identity of the agent being a nonproblematic given, as Wei Dai mentions), so if you want to address questions of where that structure is coming from, you have to do something else.
Chris asked me via PM, “I’m curious, have you written any posts about why you hold that position?”
I don’t think I have, but I’ll give the reasons here:
“evidential-style conditioning on a logically uncertain distribution” seems simpler / more elegant to me.
I’m not aware of a compelling argument for “causal-graph-style counterpossible reasoning”. There are definitely some unresolved problems with evidential-style UDT and I do endorse people looking into causal-style FDT as an alternative but I’m not convinced the solutions actually lie in that direction. (https://sideways-view.com/2018/09/30/edt-vs-cdt-2-conditioning-on-the-impossible/ and links therein are relevant here.)
Part of it is just historical, in that UDT was originally specified as “evidential-style conditioning on a logically uncertain distribution” and if I added my name as a co-author to a paper that focuses on causal-style decision theory, people would naturally wonder if something made me change my mind.
I’m actually still quite confused by the necessity of logical uncertainty for UDT. Most of the common problems like Newcomb’s or Parfit’s Hitchhiker don’t seem to require it. Where does it come in?
(The only reference to it that I could find was on the LW wiki)
I think it’s needed just to define what it means to condition on an action, i.e., if an agent conditions on “I make this decision” in order to compute its expected utility, what does that mean formally? You could make “I” a primitive element in the agent’s ontology, but I think that runs into all kinds of problems. My solution was to make it a logical statement of the form “source code X outputs action/policy Y”, and then to condition on it you need a logically uncertain distribution.
Hmm, I’m still confused. I can’t figure out why we would need logical uncertainty in the typical case to figure out the consequences of “source code X outputs action/policy Y”. Is there a simple problem where this is necessary or is this just a result of trying to solve for the general case?
Agents need to consider multiple actions and choose the one that has the best outcome. But we’re supposing that the code representing the agent’s decision only has one possible output. E.g., perhaps an agent is going to choose between action A and action B, and will end up choosing A. Then a sufficiently close examination of the agent’s source code will reveal that the scenario “the agent chooses B” is logically inconsistent. But then it’s not clear how the agent can reason about the desirability of “the agent chooses B” while evaluating its outcomes, if not via some mechanism for nontrivially reasoning about outcomes of logically inconsistent situations.
Do we need the ability to reason about logically inconsistent situations? Perhaps we could attempt to transform the question of logical counterfactuals into a question about consistent situations instead as I describe in this post? Or to put it another way, is the idea of logical counterfactuals an analogy or something that is supposed to be taken literally?
See “Example 1: Counterfactual Mugging” in Towards a New Decision Theory.
You can formalize UDT in a more standard game-theoretic setting, which allows many problems like Parfit’s Hitchhiker to be dealt with, if that is enough for what you’re interested in. However, the formalism assumes a lot about the world (such as the identity of the agent being a nonproblematic given, as Wei Dai mentions), so if you want to address questions of where that structure is coming from, you have to do something else.