I’m more interested in idealized decision theories than in heuristics, because until we figure out the idealized part, we don’t know what we’re trying to approximate or how well we’re approximating it. All my decision theory posts on LW, as well as many of other people’s posts, have followed this approach.
Also I think that discussing things at the level of “heuristics” might lead people to misconceptions. For example, UDT is not just for interacting with other agents. It’s necessary in scenarios where you are the only agent, like the Absent-Minded Driver problem.
I’m open to the idea that the idealised form is more worth studying. I still think that a substantial fraction of discussion relates to heuristics (for instance the linked post saying that Newcomblike problems are common), and that having a way to notice this and separate it off would improve dialogue.
The Absent-Minded Driver problem is a good example of a problem which doesn’t seem to succumb to confusion about idealised/heuristics.
I should clarify that I didn’t want to claim that UDT was just for interacting with agents. I wanted to show the space of statements we could start discussing. From a heuristic level, there at least seem to be some cases where it will be unnecessary to use UDT, and more complex than is needed. It would be nice if there were a simple characterisation of how to recognise if you were in a scenario where CDT might fail.
I’m not sure that “Newcomblike problems are common” was intended as an argument about heuristics. To me it’s more of an argument about which idealization we want to be studying in the long run.
A simpler version of the argument could go like this. When we build an AI, it will be able to make indistinguishable copies of itself, and the existence of these copies might depend on the AI’s decisions and on facts about the world. VNM utility maximization doesn’t cover such situations, so we need a more general theory that’s as mathematically nice and reduces to VNM utility maximization when you have only one copy.
To be more precise, by “copies” I mean multiple decision nodes belonging to the same information set, as described in this post. These can also arise from mechanisms other than copying, such as forgetting things (AMD problem), having one agent simulate another (Newcomb’s problem), or just putting multiple agents in the same situation (anthropic problems).
Does that answer your question about when UDT should be used?
I’m more interested in idealized decision theories than in heuristics, because until we figure out the idealized part, we don’t know what we’re trying to approximate or how well we’re approximating it. All my decision theory posts on LW, as well as many of other people’s posts, have followed this approach.
Also I think that discussing things at the level of “heuristics” might lead people to misconceptions. For example, UDT is not just for interacting with other agents. It’s necessary in scenarios where you are the only agent, like the Absent-Minded Driver problem.
I’m open to the idea that the idealised form is more worth studying. I still think that a substantial fraction of discussion relates to heuristics (for instance the linked post saying that Newcomblike problems are common), and that having a way to notice this and separate it off would improve dialogue.
The Absent-Minded Driver problem is a good example of a problem which doesn’t seem to succumb to confusion about idealised/heuristics.
I should clarify that I didn’t want to claim that UDT was just for interacting with agents. I wanted to show the space of statements we could start discussing. From a heuristic level, there at least seem to be some cases where it will be unnecessary to use UDT, and more complex than is needed. It would be nice if there were a simple characterisation of how to recognise if you were in a scenario where CDT might fail.
I’m not sure that “Newcomblike problems are common” was intended as an argument about heuristics. To me it’s more of an argument about which idealization we want to be studying in the long run.
A simpler version of the argument could go like this. When we build an AI, it will be able to make indistinguishable copies of itself, and the existence of these copies might depend on the AI’s decisions and on facts about the world. VNM utility maximization doesn’t cover such situations, so we need a more general theory that’s as mathematically nice and reduces to VNM utility maximization when you have only one copy.
To be more precise, by “copies” I mean multiple decision nodes belonging to the same information set, as described in this post. These can also arise from mechanisms other than copying, such as forgetting things (AMD problem), having one agent simulate another (Newcomb’s problem), or just putting multiple agents in the same situation (anthropic problems).
Does that answer your question about when UDT should be used?