This comment reads to me like: “Haha, I think there are problems with your argument, but I’m not going to tell you what they are, I’m just going to hint obliquely in a way that makes me look clever.”
If you actually do have issues with Sobel’s arguments, do you think you could actually say what they are?
Sorry if this came across as a status game. Let me give you one example.
experiencing one life can leave you incapable of experiencing another in an unbiased way.
This is a loop Sobel solves with the amnesia model. (A concurrent clone model would be a better description, to avoid any problems with influences between lives, such as physical changes). There is still however the issue of giving advice to your past self after removing amnesia, even though you ” might be incapable of adequately evaluating the lives they’ve experienced based on their current, more knowledgeable, evaluative perspective.” This loses the sight of the original purpose: the evaluating criteria should be acceptable to the original person, and no such criteria have been set in advance. Same with the parliament: the evaluation depends on the future experiences, feeding into the loop. To remedy the issue, you can decide to create and freeze the arbitration rules in advance. For example, you might choose as your utility function some weighted average of longevity, happiness, procreation, influence on the world around you, etc. Then score the utility of each simulated life, and then pick one of, say, top 10 as your “initial dynamic”. Or the top life you find acceptable. (Not restricting to automatically picking the highest-utility one, in order to avoid the “literal genie” pitfall.) You can repeat as you see fit as you go on, adjusting the criteria (hence “dynamic”).
While you are by no means guaranteed to end up with the “best life possible” life after breaking the reasoning loop, you at least are spared problems like “better off dead” and “insane parliament”, both of which result from a preference feedback loop.
Ooookay. The whole “loop” thing feels like a leaky abstraction to me. If you had to do that much work to explain the loopiness (which I’m still not sold on) and why it’s a problem, perhaps saying it’s “loopy” isn’t adding much.
This loses the sight of the original purpose: the evaluating criteria should be acceptable to the original person
I think I may still be misunderstanding you, but this seems wrong. The whole point is that even if you’re on some kind of weird drugs that make you think that drinking bleach would be great, the idealised version of you would not be under such an influence, etc. Hence it might well be that the idealised advisors evaluate things in ways that you would find unaccepable. That’s WAD.
Also, I find your other proposal hard to follow: surely if you’ve got a well-defined utility function already, then none of this is necessary?
I wasn’t trying to solve the whole CEV and FAI issue in 5 min, was only giving an example of how breaking a feedback loop avoids some of the complications.
This comment reads to me like: “Haha, I think there are problems with your argument, but I’m not going to tell you what they are, I’m just going to hint obliquely in a way that makes me look clever.”
If you actually do have issues with Sobel’s arguments, do you think you could actually say what they are?
Sorry if this came across as a status game. Let me give you one example.
This is a loop Sobel solves with the amnesia model. (A concurrent clone model would be a better description, to avoid any problems with influences between lives, such as physical changes). There is still however the issue of giving advice to your past self after removing amnesia, even though you ” might be incapable of adequately evaluating the lives they’ve experienced based on their current, more knowledgeable, evaluative perspective.” This loses the sight of the original purpose: the evaluating criteria should be acceptable to the original person, and no such criteria have been set in advance. Same with the parliament: the evaluation depends on the future experiences, feeding into the loop. To remedy the issue, you can decide to create and freeze the arbitration rules in advance. For example, you might choose as your utility function some weighted average of longevity, happiness, procreation, influence on the world around you, etc. Then score the utility of each simulated life, and then pick one of, say, top 10 as your “initial dynamic”. Or the top life you find acceptable. (Not restricting to automatically picking the highest-utility one, in order to avoid the “literal genie” pitfall.) You can repeat as you see fit as you go on, adjusting the criteria (hence “dynamic”).
While you are by no means guaranteed to end up with the “best life possible” life after breaking the reasoning loop, you at least are spared problems like “better off dead” and “insane parliament”, both of which result from a preference feedback loop.
Ooookay. The whole “loop” thing feels like a leaky abstraction to me. If you had to do that much work to explain the loopiness (which I’m still not sold on) and why it’s a problem, perhaps saying it’s “loopy” isn’t adding much.
I think I may still be misunderstanding you, but this seems wrong. The whole point is that even if you’re on some kind of weird drugs that make you think that drinking bleach would be great, the idealised version of you would not be under such an influence, etc. Hence it might well be that the idealised advisors evaluate things in ways that you would find unaccepable. That’s WAD.
Also, I find your other proposal hard to follow: surely if you’ve got a well-defined utility function already, then none of this is necessary?
I wasn’t trying to solve the whole CEV and FAI issue in 5 min, was only giving an example of how breaking a feedback loop avoids some of the complications.