Ever since I worked, in the course of my PhD, with the Godel metric, a solution of the equations of GR which contains closed timlelike curves, I’ve been noticing how strange loops mess up arguments, calculations and intuition whenever they creep in. My approach has been to search and unwind them before proceeding any further. That’s one way to resolve the grandfather paradox, for example.
The issue you are discussing is rife with loops. Notice them. Unwind them. Restate the problem without them. This is not always an easy task, some loops can be pretty insidious. Here is an example from your post:
an ideal version of that agent (fully informed, perfectly rational, etc.) would advise the non-ideal version
One of the ways of removing a potential loop is already suggested in your post:
“our volition be extrapolated once and acted on.
“Once” is what breaks the loop.
Now, to list several loops in Sobel’s arguments. Some of these are not obvious, but they are there nonetheless, if you look carefully.
two of the idealized viewpoints disagree about what is to be preferred
experiencing one life can leave you incapable of experiencing another in an unbiased way.
the idealized agent, having experienced such a level of perfection, might come to the conclusion that their non-ideal counterpart is so limited as to be better off dead.
Some of these versions are then assigned as a parliament where they vote on various choices and make trades with one another.
Meditation. Find the loops in each of the above quotes and consider how they can be avoided.
This comment reads to me like: “Haha, I think there are problems with your argument, but I’m not going to tell you what they are, I’m just going to hint obliquely in a way that makes me look clever.”
If you actually do have issues with Sobel’s arguments, do you think you could actually say what they are?
Sorry if this came across as a status game. Let me give you one example.
experiencing one life can leave you incapable of experiencing another in an unbiased way.
This is a loop Sobel solves with the amnesia model. (A concurrent clone model would be a better description, to avoid any problems with influences between lives, such as physical changes). There is still however the issue of giving advice to your past self after removing amnesia, even though you ” might be incapable of adequately evaluating the lives they’ve experienced based on their current, more knowledgeable, evaluative perspective.” This loses the sight of the original purpose: the evaluating criteria should be acceptable to the original person, and no such criteria have been set in advance. Same with the parliament: the evaluation depends on the future experiences, feeding into the loop. To remedy the issue, you can decide to create and freeze the arbitration rules in advance. For example, you might choose as your utility function some weighted average of longevity, happiness, procreation, influence on the world around you, etc. Then score the utility of each simulated life, and then pick one of, say, top 10 as your “initial dynamic”. Or the top life you find acceptable. (Not restricting to automatically picking the highest-utility one, in order to avoid the “literal genie” pitfall.) You can repeat as you see fit as you go on, adjusting the criteria (hence “dynamic”).
While you are by no means guaranteed to end up with the “best life possible” life after breaking the reasoning loop, you at least are spared problems like “better off dead” and “insane parliament”, both of which result from a preference feedback loop.
Ooookay. The whole “loop” thing feels like a leaky abstraction to me. If you had to do that much work to explain the loopiness (which I’m still not sold on) and why it’s a problem, perhaps saying it’s “loopy” isn’t adding much.
This loses the sight of the original purpose: the evaluating criteria should be acceptable to the original person
I think I may still be misunderstanding you, but this seems wrong. The whole point is that even if you’re on some kind of weird drugs that make you think that drinking bleach would be great, the idealised version of you would not be under such an influence, etc. Hence it might well be that the idealised advisors evaluate things in ways that you would find unaccepable. That’s WAD.
Also, I find your other proposal hard to follow: surely if you’ve got a well-defined utility function already, then none of this is necessary?
I wasn’t trying to solve the whole CEV and FAI issue in 5 min, was only giving an example of how breaking a feedback loop avoids some of the complications.
Ever since I worked, in the course of my PhD, with the Godel metric, a solution of the equations of GR which contains closed timlelike curves, I’ve been noticing how strange loops mess up arguments, calculations and intuition whenever they creep in. My approach has been to search and unwind them before proceeding any further. That’s one way to resolve the grandfather paradox, for example.
The issue you are discussing is rife with loops. Notice them. Unwind them. Restate the problem without them. This is not always an easy task, some loops can be pretty insidious. Here is an example from your post:
One of the ways of removing a potential loop is already suggested in your post:
“Once” is what breaks the loop.
Now, to list several loops in Sobel’s arguments. Some of these are not obvious, but they are there nonetheless, if you look carefully.
Meditation. Find the loops in each of the above quotes and consider how they can be avoided.
This comment reads to me like: “Haha, I think there are problems with your argument, but I’m not going to tell you what they are, I’m just going to hint obliquely in a way that makes me look clever.”
If you actually do have issues with Sobel’s arguments, do you think you could actually say what they are?
Sorry if this came across as a status game. Let me give you one example.
This is a loop Sobel solves with the amnesia model. (A concurrent clone model would be a better description, to avoid any problems with influences between lives, such as physical changes). There is still however the issue of giving advice to your past self after removing amnesia, even though you ” might be incapable of adequately evaluating the lives they’ve experienced based on their current, more knowledgeable, evaluative perspective.” This loses the sight of the original purpose: the evaluating criteria should be acceptable to the original person, and no such criteria have been set in advance. Same with the parliament: the evaluation depends on the future experiences, feeding into the loop. To remedy the issue, you can decide to create and freeze the arbitration rules in advance. For example, you might choose as your utility function some weighted average of longevity, happiness, procreation, influence on the world around you, etc. Then score the utility of each simulated life, and then pick one of, say, top 10 as your “initial dynamic”. Or the top life you find acceptable. (Not restricting to automatically picking the highest-utility one, in order to avoid the “literal genie” pitfall.) You can repeat as you see fit as you go on, adjusting the criteria (hence “dynamic”).
While you are by no means guaranteed to end up with the “best life possible” life after breaking the reasoning loop, you at least are spared problems like “better off dead” and “insane parliament”, both of which result from a preference feedback loop.
Ooookay. The whole “loop” thing feels like a leaky abstraction to me. If you had to do that much work to explain the loopiness (which I’m still not sold on) and why it’s a problem, perhaps saying it’s “loopy” isn’t adding much.
I think I may still be misunderstanding you, but this seems wrong. The whole point is that even if you’re on some kind of weird drugs that make you think that drinking bleach would be great, the idealised version of you would not be under such an influence, etc. Hence it might well be that the idealised advisors evaluate things in ways that you would find unaccepable. That’s WAD.
Also, I find your other proposal hard to follow: surely if you’ve got a well-defined utility function already, then none of this is necessary?
I wasn’t trying to solve the whole CEV and FAI issue in 5 min, was only giving an example of how breaking a feedback loop avoids some of the complications.
Wow.