What exactly is the optimization problem to which UDT is the solution?
The question is not trivial, because of the way we define UDT. It assumes a prior over possible programs and a utility function over their execution histories. But once you fix these two mathematical structures, there’s nothing left to optimize. Whatever happens, happens. So an answer to the question is bound to involve some new formal tricks. Any ideas to what they may be?
After that I went on to invent just such a formal trick (W/U/A), but it failed to clear things up.
But once you fix these two mathematical structures, there’s nothing left to optimize. Whatever happens, happens.
It’s a free will/epistemology (morality/truth) clash problem, expressed perhaps in agent-provability. What you’ll do is defined by the laws of physics, but you can’t infer what you’ll do by considering the laws of physics, since there are other relevant (moral) considerations that go into deciding what to do. So you can’t really say in the context of discussing decision theory that “whatever happens, happens”. It’s not a relevant consideration in arriving at a decision.
What do you mean? Whatever happens, happens, if you are not deciding. A normative idea of a correct decision can be thought of from the inside, even if it’s generally uncomputable, and so only glimpses of the answer can be extracted from it.
From the outside, counterfactual consequences don’t appear consistent. If the agent actually chooses action A, the idealized UDT-AIXI thingy will see that choosing action B would have given the agent a billion dollars, and choosing C would have given a trillion. Do you see a way around that?
UDT-AIXI could ask which moral arguments the agent would discover if it had more time to think. It won’t of course examine the counterfactuals of a fact known to the context in which the resulting mathematical structure is to be interpreted. You can only use a normative consideration from the inside, so whenever you step outside, you must also shift the decision problem to allow thinking about moral considerations.
My email from Nov 15 may be relevant here:
After that I went on to invent just such a formal trick (W/U/A), but it failed to clear things up.
It’s a free will/epistemology (morality/truth) clash problem, expressed perhaps in agent-provability. What you’ll do is defined by the laws of physics, but you can’t infer what you’ll do by considering the laws of physics, since there are other relevant (moral) considerations that go into deciding what to do. So you can’t really say in the context of discussing decision theory that “whatever happens, happens”. It’s not a relevant consideration in arriving at a decision.
But it seems to be a relevant consideration when looking at the situation “from the outside” like your proposed UDT-AIXI does, right?
What do you mean? Whatever happens, happens, if you are not deciding. A normative idea of a correct decision can be thought of from the inside, even if it’s generally uncomputable, and so only glimpses of the answer can be extracted from it.
From the outside, counterfactual consequences don’t appear consistent. If the agent actually chooses action A, the idealized UDT-AIXI thingy will see that choosing action B would have given the agent a billion dollars, and choosing C would have given a trillion. Do you see a way around that?
UDT-AIXI could ask which moral arguments the agent would discover if it had more time to think. It won’t of course examine the counterfactuals of a fact known to the context in which the resulting mathematical structure is to be interpreted. You can only use a normative consideration from the inside, so whenever you step outside, you must also shift the decision problem to allow thinking about moral considerations.