Sorry, following the SIAI decision theory workshop, I’ve been working with some of the participants to write a better formulation of UDT to avoid some of the problems that were pointed out during the workshop. It’s a bit hard for me to switch back to thinking about the old formulation and try to explain that, so you might want to wait a bit for the “new version” to come out.
Be sure to consider the possibility of the worlds spontaneously constructing the agent in some epistemic state, or dissolving it. Also, when a (different) agent thinks about our agent, it might access a statement about the agent’s strategy that involves many different epistemic states. For this reason, the agent’s strategy controls many more worlds than where the agent is instantiated “normally”. This makes the problem of figuring out which of the world programs contain the agent very non-trivial, depending on what state of the agent are we talking about, and what kind of worlds are we considering, and not just by the order in which the agent program expects observations.
These considerations made me write off Bayesian updating as a non-fundamental technique that shouldn’t be shoehorned into a more general decision theory for working with arbitrary preference. I currently suspect that there is no generally applicable simple trick, and FAI decision theory should instead seek to clarify the conceptual issues, and then work on optimizing brute force algorithms that follow from that picture. Think abstract interpretation, not variational mean field.
Sorry, following the SIAI decision theory workshop, I’ve been working with some of the participants to write a better formulation of UDT to avoid some of the problems that were pointed out during the workshop. It’s a bit hard for me to switch back to thinking about the old formulation and try to explain that, so you might want to wait a bit for the “new version” to come out.
Be sure to consider the possibility of the worlds spontaneously constructing the agent in some epistemic state, or dissolving it. Also, when a (different) agent thinks about our agent, it might access a statement about the agent’s strategy that involves many different epistemic states. For this reason, the agent’s strategy controls many more worlds than where the agent is instantiated “normally”. This makes the problem of figuring out which of the world programs contain the agent very non-trivial, depending on what state of the agent are we talking about, and what kind of worlds are we considering, and not just by the order in which the agent program expects observations.
These considerations made me write off Bayesian updating as a non-fundamental technique that shouldn’t be shoehorned into a more general decision theory for working with arbitrary preference. I currently suspect that there is no generally applicable simple trick, and FAI decision theory should instead seek to clarify the conceptual issues, and then work on optimizing brute force algorithms that follow from that picture. Think abstract interpretation, not variational mean field.
I look forward to it.
I should probably be studying for a linear models exam tomorrow anyway...