I think a lot of the problems come from starting with final states. Instead, we can let Y1, Y2, … be the possible outputs of the system S, each with a certain utility attached. There is no such thing as mixed states: you can’t half-output of Y1 and half-output Y2 (in some cases, you might have a probabilistic strategy which outputs Y1 50% of the time, but the utility calculation for that is different). Furthermore, there is no need to deal with the coarse-graining issue.
Presumably. It’s debatable whether or not this captures risk aversion, but in my opinion it does (and risk aversion falls out of nonlinear utility).
But one thing to keep in mind is that if there is an output Y1 that leads to a final state X1, and an output Y2 that leads to a final state X2, then there is not necessarily an output leading to 0.5X1+0.5X2.
I think a lot of the problems come from starting with final states. Instead, we can let Y1, Y2, … be the possible outputs of the system S, each with a certain utility attached. There is no such thing as mixed states: you can’t half-output of Y1 and half-output Y2 (in some cases, you might have a probabilistic strategy which outputs Y1 50% of the time, but the utility calculation for that is different). Furthermore, there is no need to deal with the coarse-graining issue.
How do you assign utility to an output that is a mixed final state? As a weighted sum of utilities?
Presumably. It’s debatable whether or not this captures risk aversion, but in my opinion it does (and risk aversion falls out of nonlinear utility).
But one thing to keep in mind is that if there is an output Y1 that leads to a final state X1, and an output Y2 that leads to a final state X2, then there is not necessarily an output leading to 0.5X1+0.5X2.