Ooookay. The whole “loop” thing feels like a leaky abstraction to me. If you had to do that much work to explain the loopiness (which I’m still not sold on) and why it’s a problem, perhaps saying it’s “loopy” isn’t adding much.
This loses the sight of the original purpose: the evaluating criteria should be acceptable to the original person
I think I may still be misunderstanding you, but this seems wrong. The whole point is that even if you’re on some kind of weird drugs that make you think that drinking bleach would be great, the idealised version of you would not be under such an influence, etc. Hence it might well be that the idealised advisors evaluate things in ways that you would find unaccepable. That’s WAD.
Also, I find your other proposal hard to follow: surely if you’ve got a well-defined utility function already, then none of this is necessary?
I wasn’t trying to solve the whole CEV and FAI issue in 5 min, was only giving an example of how breaking a feedback loop avoids some of the complications.
Ooookay. The whole “loop” thing feels like a leaky abstraction to me. If you had to do that much work to explain the loopiness (which I’m still not sold on) and why it’s a problem, perhaps saying it’s “loopy” isn’t adding much.
I think I may still be misunderstanding you, but this seems wrong. The whole point is that even if you’re on some kind of weird drugs that make you think that drinking bleach would be great, the idealised version of you would not be under such an influence, etc. Hence it might well be that the idealised advisors evaluate things in ways that you would find unaccepable. That’s WAD.
Also, I find your other proposal hard to follow: surely if you’ve got a well-defined utility function already, then none of this is necessary?
I wasn’t trying to solve the whole CEV and FAI issue in 5 min, was only giving an example of how breaking a feedback loop avoids some of the complications.