One more, because one of my posts presented two open problems, and I only listed one of them above:
15. Our current theoretical foundations for rationality all assume a fully specified utility function (or the equivalent), or at least a probability distribution on utility functions (to express moral/value uncertainty). But to the extent that humans can be considered to have a utility function at all, it’s may best be viewed as a partial function that returns “unknown” for most of the input domain. Our current decision theories can’t handle this because they would end up trying to add “unknown” to a numerical value during expected utility computation. Forcing humans to come up with an utility function or even a probability distribution on utility functions in order to use decision theory seems highly unsafe so we need an alternative.
If you just do it the straightforward way, any option you can choose would have a non-zero probability of producing an outcome with “unknown” or NaN utility. If multiply those two numbers together you get NaN, then if you add that to other probability*utility values as part of your expected utility computation you will end up with NaN as your final expected utility. I don’t see how to avoid this, hence my question.
One more, because one of my posts presented two open problems, and I only listed one of them above:
15. Our current theoretical foundations for rationality all assume a fully specified utility function (or the equivalent), or at least a probability distribution on utility functions (to express moral/value uncertainty). But to the extent that humans can be considered to have a utility function at all, it’s may best be viewed as a partial function that returns “unknown” for most of the input domain. Our current decision theories can’t handle this because they would end up trying to add “unknown” to a numerical value during expected utility computation. Forcing humans to come up with an utility function or even a probability distribution on utility functions in order to use decision theory seems highly unsafe so we need an alternative.
Does it help to propagate “unknown” through computations by treating it like NaN? Or would that tend to turn the answer to every question into NaN?
If you just do it the straightforward way, any option you can choose would have a non-zero probability of producing an outcome with “unknown” or NaN utility. If multiply those two numbers together you get NaN, then if you add that to other probability*utility values as part of your expected utility computation you will end up with NaN as your final expected utility. I don’t see how to avoid this, hence my question.