The idea that there is an “ideal” decision theory.
There are many classes of decision problems that allow optimal solutions, but none of them can cover all of reality, because in reality an AI can be punished for having any given decision theory. That said, the design space of decision theories has sweet spots. For example, future AIs will likely face an environment where copying and simulation is commonplace, and we’ve found simple decision theories that allow for copies and simulations. Looking for more such sweet spots is fun and fruitful.
Imo we haven’t found a simple decision theory that allows for copies and simulations. We’ve found a simple rule that works in limiting cases, but is only well-defined for identical copies (modulo stochasticity). My expectation that FDT will be rigorously extended from this setting is low, for much the same reason that I don’t expect a rigorous definition of CDT. You understand FDT much better than I do, though—would you say that’s a fair summary?
If all agents involved in a situation share the same utility function over outcomes, we should be able to make them coordinate despite having different source code. I think that’s where one possible boundary will settle, and I expect the resulting theory to be simple. Whereas in case of different utility functions we enter the land of game theory, where I’m pretty sure there can be no theory of unilateral decision making.
I’m not convinced by the distinction you draw. Suppose you simulate me at slightly less than perfect fidelity. The simulation is an agent with a (slightly) different utility function to me. Yet this seems like a case where FDT should be able to say relevant things.
FDT requires a notion of logical causality, which hasn’t appeared yet.
I expect that logical causality will be just as difficult to formalise as normal causality, and in fact that no “correct” formalisation exists for either.
The Pearl- Rubin- Sprites-Glymour- and others I theory of causality is a very powerful framework for causality that satisfies pretty what one intuitively understand as ‘causality’. It is moreover powerful enough to make definite computations and even the much- craved for ‘real applications’.
It is ‘a very correct’ formalisation of ‘normal’ causality.
I say ‘very correct’ instead of ‘correct’ because there are still areas of improvements—but this is more like GR improving on Newtonian gravity rather than Newtonian gravity being incorrect.
Got a link to the best overview/defense of that claim? I’m open to this argument but have some cached thoughts about Pearl’s framework being unsatisfactory—would be useful to do some more reading and see if I still believe them.
There are some cases where Pearl and others’ causality framework can be improved—supposedly Factored Sets will, although I personally don’t understand it. I was recently informed that certain abductive counterfactual phrases due to David Lewis are not well-captured by Pearl’s system. I believe there are also other ways—all of this is actively being researched.
What do you find unsatisfactory about Pearl?
All of this is besides the point which is that there is a powerful well-developed, highly elegant theory of causality with an enourmous range of applications.
Rubin’s framework (which I am told is equivalent to Pearl) is used throughout econometrics—indeed econometrics is best understand as the Science of Causality.
I am not an expert—I am trying to learn much of this theory right now. I am probably not the best person to ask about theory of causality. That said:
I am not sure to what degree you are already familiar with Pearl’s theory of causality but I recommend
For a much more leisurely argument for Pearl’s viewpoint, I recommend his “book of why”. In a pinch you could take a look at the book review on the causality bloglist on LW.
There are many classes of decision problems that allow optimal solutions, but none of them can cover all of reality, because in reality an AI can be punished for having any given decision theory. That said, the design space of decision theories has sweet spots. For example, future AIs will likely face an environment where copying and simulation is commonplace, and we’ve found simple decision theories that allow for copies and simulations. Looking for more such sweet spots is fun and fruitful.
Imo we haven’t found a simple decision theory that allows for copies and simulations. We’ve found a simple rule that works in limiting cases, but is only well-defined for identical copies (modulo stochasticity). My expectation that FDT will be rigorously extended from this setting is low, for much the same reason that I don’t expect a rigorous definition of CDT. You understand FDT much better than I do, though—would you say that’s a fair summary?
If all agents involved in a situation share the same utility function over outcomes, we should be able to make them coordinate despite having different source code. I think that’s where one possible boundary will settle, and I expect the resulting theory to be simple. Whereas in case of different utility functions we enter the land of game theory, where I’m pretty sure there can be no theory of unilateral decision making.
I’m not convinced by the distinction you draw. Suppose you simulate me at slightly less than perfect fidelity. The simulation is an agent with a (slightly) different utility function to me. Yet this seems like a case where FDT should be able to say relevant things.
In Abram’s words,
I expect that logical causality will be just as difficult to formalise as normal causality, and in fact that no “correct” formalisation exists for either.
What. This seems obviously incorrect?
The Pearl- Rubin- Sprites-Glymour- and others I theory of causality is a very powerful framework for causality that satisfies pretty what one intuitively understand as ‘causality’. It is moreover powerful enough to make definite computations and even the much- craved for ‘real applications’.
It is ‘a very correct’ formalisation of ‘normal’ causality.
I say ‘very correct’ instead of ‘correct’ because there are still areas of improvements—but this is more like GR improving on Newtonian gravity rather than Newtonian gravity being incorrect.
Got a link to the best overview/defense of that claim? I’m open to this argument but have some cached thoughts about Pearl’s framework being unsatisfactory—would be useful to do some more reading and see if I still believe them.
There are some cases where Pearl and others’ causality framework can be improved—supposedly Factored Sets will, although I personally don’t understand it. I was recently informed that certain abductive counterfactual phrases due to David Lewis are not well-captured by Pearl’s system. I believe there are also other ways—all of this is actively being researched.
What do you find unsatisfactory about Pearl?
All of this is besides the point which is that there is a powerful well-developed, highly elegant theory of causality with an enourmous range of applications.
Rubin’s framework (which I am told is equivalent to Pearl) is used throughout econometrics—indeed econometrics is best understand as the Science of Causality.
I am not an expert—I am trying to learn much of this theory right now. I am probably not the best person to ask about theory of causality. That said:
I am not sure to what degree you are already familiar with Pearl’s theory of causality but I recommend
https://michaelnielsen.org/ddi/if-correlation-doesnt-imply-causation-then-what-does/
for an excellent introduction.
THere is EY’s
https://www.lesswrong.com/posts/hzuSDMx7pd2uxFc5w/causal-diagrams-and-causal-models
which you may or may not find convincing
For a much more leisurely argument for Pearl’s viewpoint, I recommend his “book of why”. In a pinch you could take a look at the book review on the causality bloglist on LW.
https://www.lesswrong.com/tag/causality