I would think a satisficer would maximize E(g(U)) not g(E(U)).
I assume you are avoiding maximizing E(f(U) because doing so would result in the AI seeking super-high certainty that U is at the maximum of f, leading to side effects?
Edit: it seems to me that once the AI got the expected value it wanted, it would be incentivized to not seek new information since that would adjust the expected value away from that value. So e.g. it might arrange things so that the expected value conditional on it committing suicide is at the intended level, then commit suicide. Or maybe that’s a feature not a bug if we want self-limiting AI?
Yes of course. But generally you would expect a state with maximal f(U) to be different than a state with maximal U—max U state is particularly likely to be a “marketing world” but max f(U) is not, since any world with U at the maximum of f qualifies.
I would think a satisficer would maximize E(g(U)) not g(E(U)).
I assume you are avoiding maximizing E(f(U) because doing so would result in the AI seeking super-high certainty that U is at the maximum of f, leading to side effects?
Edit: it seems to me that once the AI got the expected value it wanted, it would be incentivized to not seek new information since that would adjust the expected value away from that value. So e.g. it might arrange things so that the expected value conditional on it committing suicide is at the intended level, then commit suicide. Or maybe that’s a feature not a bug if we want self-limiting AI?
Maximising E(f(U)) is just expected utility maximisation on the utility function f(U)
Yes of course. But generally you would expect a state with maximal f(U) to be different than a state with maximal U—max U state is particularly likely to be a “marketing world” but max f(U) is not, since any world with U at the maximum of f qualifies.