Thanks, Eliezer—that’s a clear explanation of an elegant theory. So far, TDT (I haven’t looked carefully at UDT) strikes me as more promising than any other decision theory I’m aware of (including my own efforts, past and pending). Congratulations are in order!
I agree, of course, that TDT doesn’t make the A6/A7 mistake. That was just a simple illustration of the need, in counterfactual reasoning (broadly construed), to specify somehow what to hold fixed and what not to, and that different ways of doing so specify different senses of counterfactual inference (i.e., that there are different kinds of ‘if-counterfactually’). If counterfactual inference is construed a la Pearl, for example, then such inferences (causal-counterfactual) correspond to causal links (if-causally).
As you say, TDT’s utility formula doesn’t perform general logical inferences (or evidential-counterfactual inferences) from the antecedents it evaluates (i.e. the candidate outputs of the Platonic computation). Rather, the utility formula performs causal-counterfactual inferences from the set of nodes that designate the outputs of the Platonic computation, in all places where that Platonic computation is approximately physically instantiated.
However, it seems to me we can, if we wish, use TDT to define what we can call a TDT-counterfactual that tells us would be true ‘if-timelessly’ a particular physical agent’s particular physical action were to occur. In particular, whereas CDT says that what would be true (if-causally) consists of what’s causally downstream from that action, TDT says that what would be true (if-timelessly) consists of what’s causally downstream from the output of the suitably-specified Platonic computation that the particular physical agent approximately implements, and also what’s causally downstream from that same Platonic computation in all other places where that computation is approximately physically instantiated. (And the physical TDT agent argmaxes over the utilities of the TDT-counterfactual consequences of that agent’s candidate actions.)
I think there are a few reasons we might sometimes find it useful to think in terms of the TDT-counterfactual consequences of a physical agent’s actions, rather than directly in terms of the standard TDT formulation (even though they’re merely two different ways of expressing the same decision theory, unless I’ve misunderstood).
The TDT-counterfactual perspective places TDT in a common framework with other decision theories that (implicitly or explicitly) use other kinds of counterfactual reasoning, starting with a physical agent’s action as the antecedent. Then we can apply some meta-criterion to ask which of those alternative theories is correct, and why. (That was the intuition behind my MCDT proposal, although MCDT itself was hastily specified and too simpleminded to be correct.)
Plausibly, people are agents who think in terms of the counterfactual consequences of an action, rather than being hardwired to use TDT. If we are to choose to act in accordance with TDT from now on (or, equivalently, if we are to build AIs who act in accordance with TDT), we need to be persuaded that doing so is for the best (even if e.g. a Newcomb snapshot was already taken before we became persuaded). (I’m assuming here that our extant choice machinery allows us the flexibility to be persuaded about what sort of counterfactual to use; if not, alas, we can’t necessarily get there from here).
In the standard formulation of TDT, you effectively view yourself as an abstract computation with one or more approximate physical instantiations, and you ask what you (thus construed) cause (i.e. what follows causal-counterfactually). In the alternative formulation, I view myself as a particular physical agent that is among one or more approximate instantiations of an abstract computation, and I ask what follows TDT-counterfactually from what I (thus construed) choose.
The original formulation seems to require a precommitment to identify oneself with all instantiations (in the causal net) of the abstract computation (or at least seems to require that in order for us non-TDT agents to decide to emulate TDT). And that identification is indeed plausible in the case of fairly exact replication. But consider, say, a 1-shot PD game between Eliezer and me. Our mutual understanding of reflexive consistency would let us win. And I agree that we both approximately instantiate, at some level of abstraction, a common decision computation, which is what lets the TDT framework apply and lets us both win.
But (in contrast with an exact-simulation case) that common computation is at a level of abstraction that does not preserve our respective personal identities. (That’s kind of the point of the abstraction. My utility function for the game places value on Gary’s points and not Eliezer’s points; the common abstract computation lacks that bias.) So I would hesitate to identify either of us with the common abstraction. (And I see in other comments that Eliezer explicitly agrees.) Rather, I’d like to reason that if-timelessly I, Gary, choose ‘Cooperate’, then so does Eliezer. That way, “I am you as you are me” emerges as a (metaphorical) conclusion about the situation (we each have a choice about the other’s action in the game, and are effectively acting together) rather than being needed as the point of departure.
Again, the foregoing is just an alternative but equivalent (unless I’ve erred) way of viewing TDT, an alternative that may be useful for some purposes.
Thanks, Eliezer—that’s a clear explanation of an elegant theory. So far, TDT (I haven’t looked carefully at UDT) strikes me as more promising than any other decision theory I’m aware of (including my own efforts, past and pending). Congratulations are in order!
I agree, of course, that TDT doesn’t make the A6/A7 mistake. That was just a simple illustration of the need, in counterfactual reasoning (broadly construed), to specify somehow what to hold fixed and what not to, and that different ways of doing so specify different senses of counterfactual inference (i.e., that there are different kinds of ‘if-counterfactually’). If counterfactual inference is construed a la Pearl, for example, then such inferences (causal-counterfactual) correspond to causal links (if-causally).
As you say, TDT’s utility formula doesn’t perform general logical inferences (or evidential-counterfactual inferences) from the antecedents it evaluates (i.e. the candidate outputs of the Platonic computation). Rather, the utility formula performs causal-counterfactual inferences from the set of nodes that designate the outputs of the Platonic computation, in all places where that Platonic computation is approximately physically instantiated.
However, it seems to me we can, if we wish, use TDT to define what we can call a TDT-counterfactual that tells us would be true ‘if-timelessly’ a particular physical agent’s particular physical action were to occur. In particular, whereas CDT says that what would be true (if-causally) consists of what’s causally downstream from that action, TDT says that what would be true (if-timelessly) consists of what’s causally downstream from the output of the suitably-specified Platonic computation that the particular physical agent approximately implements, and also what’s causally downstream from that same Platonic computation in all other places where that computation is approximately physically instantiated. (And the physical TDT agent argmaxes over the utilities of the TDT-counterfactual consequences of that agent’s candidate actions.)
I think there are a few reasons we might sometimes find it useful to think in terms of the TDT-counterfactual consequences of a physical agent’s actions, rather than directly in terms of the standard TDT formulation (even though they’re merely two different ways of expressing the same decision theory, unless I’ve misunderstood).
The TDT-counterfactual perspective places TDT in a common framework with other decision theories that (implicitly or explicitly) use other kinds of counterfactual reasoning, starting with a physical agent’s action as the antecedent. Then we can apply some meta-criterion to ask which of those alternative theories is correct, and why. (That was the intuition behind my MCDT proposal, although MCDT itself was hastily specified and too simpleminded to be correct.)
Plausibly, people are agents who think in terms of the counterfactual consequences of an action, rather than being hardwired to use TDT. If we are to choose to act in accordance with TDT from now on (or, equivalently, if we are to build AIs who act in accordance with TDT), we need to be persuaded that doing so is for the best (even if e.g. a Newcomb snapshot was already taken before we became persuaded). (I’m assuming here that our extant choice machinery allows us the flexibility to be persuaded about what sort of counterfactual to use; if not, alas, we can’t necessarily get there from here).
In the standard formulation of TDT, you effectively view yourself as an abstract computation with one or more approximate physical instantiations, and you ask what you (thus construed) cause (i.e. what follows causal-counterfactually). In the alternative formulation, I view myself as a particular physical agent that is among one or more approximate instantiations of an abstract computation, and I ask what follows TDT-counterfactually from what I (thus construed) choose.
The original formulation seems to require a precommitment to identify oneself with all instantiations (in the causal net) of the abstract computation (or at least seems to require that in order for us non-TDT agents to decide to emulate TDT). And that identification is indeed plausible in the case of fairly exact replication. But consider, say, a 1-shot PD game between Eliezer and me. Our mutual understanding of reflexive consistency would let us win. And I agree that we both approximately instantiate, at some level of abstraction, a common decision computation, which is what lets the TDT framework apply and lets us both win.
But (in contrast with an exact-simulation case) that common computation is at a level of abstraction that does not preserve our respective personal identities. (That’s kind of the point of the abstraction. My utility function for the game places value on Gary’s points and not Eliezer’s points; the common abstract computation lacks that bias.) So I would hesitate to identify either of us with the common abstraction. (And I see in other comments that Eliezer explicitly agrees.) Rather, I’d like to reason that if-timelessly I, Gary, choose ‘Cooperate’, then so does Eliezer. That way, “I am you as you are me” emerges as a (metaphorical) conclusion about the situation (we each have a choice about the other’s action in the game, and are effectively acting together) rather than being needed as the point of departure.
Again, the foregoing is just an alternative but equivalent (unless I’ve erred) way of viewing TDT, an alternative that may be useful for some purposes.