I continue to be puzzled as to why many people on LW are very confident in the “algorithmic ontology” about decision theory:
So I see all axes except the “algorithm” axis as “live debates”—basically anyone who has thought about it very much seems to agree that you control “the policy of agents who sufficiently resemble you” (rather than something more myopic like “your individual action”)
Can someone point to resources that clearly argue for this position? (I don’t think that, e.g., the intuition that you ought to cooperate with your exact copy in a Prisoner’s Dilemma — much as I share it — is an argument for this ontology. You could endorse the physicalist ontology + EDT, for example.)
I can’t point you to existing resources, but from my perspective, I assumed an algorithmic ontology because it seemed like the only way to make decision theory well defined (at least potentially, after solving various open problems). That is, for an AI that knows its own source code S, you could potentially define the “consequences of me doing X” as the logical consequences of the logical statement “S outputs X”. Whereas I’m not sure how this could even potentially be defined under a physicalist ontology, since it seems impossible for even an ASI to know the exact details of itself as a physical system.
This does lead to the problem that I don’t know how to apply LDT to humans (who do not know their own source code), which does make me somewhat suspicious that the algorithmic ontology might be a wrong approach (although physicalist ontology doesn’t seem to help). I mentioned this as problem #6 in UDT shows that decision theory is more puzzling than ever.
I am indeed interested in decision theory that applies to agents other than AIs that know their own source code. Though I’m not sure why it’s a problem for the physicalist ontology that the agent doesn’t know the exact details of itself — seems plausible to me that “decisions” might just be a vague concept, which we still want to be able to reason about under bounded rationality. E.g. under physicalist EDT, what I ask myself when I consider a decision to do X is, “What consequences do I expect conditional on my brain-state going through the process that I call ‘deciding to do X’ [and conditional on all the other relevant info I know including my own reasoning about this decision, per the Tickle Defense]?” But I might miss your point.
Re: mathematical universe hypothesis: I’m pretty unconvinced, though I at least see the prima facie motivation (IIUC: we want an explanation for why the universe we find ourselves in has the dynamical laws and initial conditions it does, rather than some others). Not an expert here, this is just based on some limited exploration of the topic. My main objections:
The move from “fundamental physics is very well described by mathematics” to “physics is (some) mathematical structure” seems like a map-territory error. I just don’t see the justification for this.
I worry about giving description-length complexity a privileged status when setting priors / judging how “simple” a hypothesis is. The Great Meta-Turing Machine in the Sky as described by Schmidhuber scores very poorly by the speed prior.
It’s very much not obvious to me that conscious experience is computable. (This is a whole can of worms in this community, presumably :).)
Not sure what you mean by “the math” exactly. I’ve heard people cite the algorithmic ontology as a motivation for, e.g., logical updatelessness, or for updateless decision theory generally. In the case of logical updatelessness, I think (low confidence!) the idea is that if you don’t see yourself as this physical object that exists in “the real world,” but rather see yourself as an algorithm instantiated in a bunch of possible worlds, then it might be sensible to follow a policy that doesn’t update on e.g. the first digit of pi being odd.
query rephrase: taboo both “algorithmic ontology” and “physicalist ontology”. describe how each of them constructs math to describe things in the world, and how that math differs. That is, if you’re saying you have an ontology, presumably this means you have some math and some words describing how the math relates to reality. I’m interested in a comparison of that math and those words; so far you’re saying things about a thing I don’t really understand as being separate from physicalism. Why can’t you just see yourself as multiple physical objects and still have a physicalist ontology? what makes these things different in some, any, math, as opposed to only being a difference in how the math connects to reality?
I think I just don’t understand / probably disagree with the premise of your question, sorry. I’m taking as given whatever distinction between these two ontologies is noted in the post I linked. These don’t need to be mathematically precise in order to be useful concepts.
There isn’t a difference, though it’s only because algorithms can simulate all of the physical stuff that’s in the physical ontology, and thus the physical ontology is a special case of the algorithmic ontology.
Thanks — do you have a specific section of the paper in mind? Is the idea that this ontology is motivated by “finding a decision theory that recommends verdicts in such and such decision problems that we find pre-theoretically intuitive”?
shrug — I guess it’s not worth rehashing pretty old-on-LW decision theory disagreements, but: (1) I just don’t find the pre-theoretic verdicts in that paper nearly as obvious as the authors do, since these problems are so out-of-distribution. Decision theory is hard. Also, some interpretations of logical decision theories give the pre-theoretically “wrong” verdict on “betting on the past.” (2) I pre-theoretically find the kind of logical updatelessness that some folks claim follows from the algorithmic ontology pretty bizarre. (3) On its face it seems more plausible to me that algorithms just aren’t ontologically basic, they’re abstractions we use to represent (physical) input-output processes.
I continue to be puzzled as to why many people on LW are very confident in the “algorithmic ontology” about decision theory:
Can someone point to resources that clearly argue for this position? (I don’t think that, e.g., the intuition that you ought to cooperate with your exact copy in a Prisoner’s Dilemma — much as I share it — is an argument for this ontology. You could endorse the physicalist ontology + EDT, for example.)
I can’t point you to existing resources, but from my perspective, I assumed an algorithmic ontology because it seemed like the only way to make decision theory well defined (at least potentially, after solving various open problems). That is, for an AI that knows its own source code S, you could potentially define the “consequences of me doing X” as the logical consequences of the logical statement “S outputs X”. Whereas I’m not sure how this could even potentially be defined under a physicalist ontology, since it seems impossible for even an ASI to know the exact details of itself as a physical system.
This does lead to the problem that I don’t know how to apply LDT to humans (who do not know their own source code), which does make me somewhat suspicious that the algorithmic ontology might be a wrong approach (although physicalist ontology doesn’t seem to help). I mentioned this as problem #6 in UDT shows that decision theory is more puzzling than ever.
ETA: I was (and still am) also under strong influence of Tegmark’s Mathematical universe hypothesis. What’s your view on it?
Thanks, that’s helpful!
I am indeed interested in decision theory that applies to agents other than AIs that know their own source code. Though I’m not sure why it’s a problem for the physicalist ontology that the agent doesn’t know the exact details of itself — seems plausible to me that “decisions” might just be a vague concept, which we still want to be able to reason about under bounded rationality. E.g. under physicalist EDT, what I ask myself when I consider a decision to do X is, “What consequences do I expect conditional on my brain-state going through the process that I call ‘deciding to do X’ [and conditional on all the other relevant info I know including my own reasoning about this decision, per the Tickle Defense]?” But I might miss your point.
Re: mathematical universe hypothesis: I’m pretty unconvinced, though I at least see the prima facie motivation (IIUC: we want an explanation for why the universe we find ourselves in has the dynamical laws and initial conditions it does, rather than some others). Not an expert here, this is just based on some limited exploration of the topic. My main objections:
The move from “fundamental physics is very well described by mathematics” to “physics is (some) mathematical structure” seems like a map-territory error. I just don’t see the justification for this.
I worry about giving description-length complexity a privileged status when setting priors / judging how “simple” a hypothesis is. The Great Meta-Turing Machine in the Sky as described by Schmidhuber scores very poorly by the speed prior.
It’s very much not obvious to me that conscious experience is computable. (This is a whole can of worms in this community, presumably :).)
how does a physicalist ontology differ from an algorithmic ontology in terms of the math?
Not sure what you mean by “the math” exactly. I’ve heard people cite the algorithmic ontology as a motivation for, e.g., logical updatelessness, or for updateless decision theory generally. In the case of logical updatelessness, I think (low confidence!) the idea is that if you don’t see yourself as this physical object that exists in “the real world,” but rather see yourself as an algorithm instantiated in a bunch of possible worlds, then it might be sensible to follow a policy that doesn’t update on e.g. the first digit of pi being odd.
query rephrase: taboo both “algorithmic ontology” and “physicalist ontology”. describe how each of them constructs math to describe things in the world, and how that math differs. That is, if you’re saying you have an ontology, presumably this means you have some math and some words describing how the math relates to reality. I’m interested in a comparison of that math and those words; so far you’re saying things about a thing I don’t really understand as being separate from physicalism. Why can’t you just see yourself as multiple physical objects and still have a physicalist ontology? what makes these things different in some, any, math, as opposed to only being a difference in how the math connects to reality?
I think I just don’t understand / probably disagree with the premise of your question, sorry. I’m taking as given whatever distinction between these two ontologies is noted in the post I linked. These don’t need to be mathematically precise in order to be useful concepts.
ah my bad, my attention missed the link! that does in fact answer my whole question, and if I hadn’t missed it I’d have had nothing to ask :)
There isn’t a difference, though it’s only because algorithms can simulate all of the physical stuff that’s in the physical ontology, and thus the physical ontology is a special case of the algorithmic ontology.
Helpful link here to build intuition:
http://www.amirrorclear.net/academic/ideas/simulation/index.html
That’s what I already believed, but OP seems to disagree, so I’m trying to understand what they mean
The argument for this is spelled out in Eliezer and Nate’s Functional Decision Theory: A New Theory of Instrumental Rationality. See also the LessWrong wiki tag page
Thanks — do you have a specific section of the paper in mind? Is the idea that this ontology is motivated by “finding a decision theory that recommends verdicts in such and such decision problems that we find pre-theoretically intuitive”?
That sounds like a good description of my understanding, but I’d also say the pre-theoretic intuitions are real damn convincing!
There’s a table of contents which you can use to read relevant sections of the paper. You know your cruxes better than I do.
shrug — I guess it’s not worth rehashing pretty old-on-LW decision theory disagreements, but: (1) I just don’t find the pre-theoretic verdicts in that paper nearly as obvious as the authors do, since these problems are so out-of-distribution. Decision theory is hard. Also, some interpretations of logical decision theories give the pre-theoretically “wrong” verdict on “betting on the past.” (2) I pre-theoretically find the kind of logical updatelessness that some folks claim follows from the algorithmic ontology pretty bizarre. (3) On its face it seems more plausible to me that algorithms just aren’t ontologically basic, they’re abstractions we use to represent (physical) input-output processes.