The central point is more that superintelligence won’t be able to help with your decisions, won’t be able to decide for you at some fundamental level, no matter how capable it is. It can help instrumentally, but not replace your ability to decide in order to find out what the decisions are, so in some sense it’s not able to help at all. I’m trying to capture the sense in which this holds regardless of its ability to precisely predict and determine outcomes in the physical world (if we only look at the state of the future, rather than full trajectories that get the world there).
When a program is given strange input, or if the computer it would be running on is destroyed, that event in the physical world usually doesn’t affect the semantics of the program that describes its behavior for all possible inputs. If you are weak and brittle, talking about what you’ll decide requires defining what we even mean in principle by a decision that is yours, only then can we ask if it does remain yours in actuality. Or if it does remain yours, even if you are yourself no longer present in actuality. Which is not very useful for keeping it (or yourself) present in actuality, but can be conceptually useful for formulating desiderata towards it being present in actuality.
So there are two claims. First, it’s in some sense natural for your decisions to remain yours, if you don’t start mindlessly parroting external inputs that dictate your actions, even in the face of superintelligence (in its aspect of capability, but not necessarily aimed in a way that disrupts you). Second, if you are strongly manipulated or otherwise overridden, this should in some sense mean that you are no longer present, that the resulting outcome doesn’t capture or simulate what we should define as being you (in order to talk about the decisions that are in principle yours). Thus presence of overpowering manipulation doesn’t contradict decisions usually remaining yours, it just requires that you are consequently no longer present to manifest them in actuality when that happens.
This seems like the third comment on the same concern, I’ve also answered it here and here, going into more detail on other related things. So there is a missing prerequisite post.
In case this helps you write that post, here are some things I am still confused about.
What, specifically, is the “you” that is no longer present in the scenario of strong manipulation?
In the “mindless parroting” scenario, what happens if the ASI magically disappeared? Does/can the “you” reappear? Under what circumstances?
Why is this not a fully general argument against other humans helping you make decisions? For example, if someone decides to mindlessly parrot everything a cult leader tells them, I agree there’s a sense in which they are no longer present. But the choice to obey is still theirs, and can be changed, and they can reappear if they change that one choice.
OTOH, if this is a fully general argument about anyone helping anyone else make decisions, that seems like a major (and underspecified) redefinition of both “help” and “decision.” It then seems like it’s premature to jump to focusing on ASI as a special case, and also I’m not sure why I’m supposed to care about these definitional changes?
“So it’s fair to say that we are already not physically present in the world, the things that are physically present are better described as kludgy and imperfect simulators”—I mean, yes, I see your point, but also, I am present implicitly in the structure of the physically-existing things. There is some set of arrangements-of-matter that I’d consider me, and others I would not. I don’t know if the set’s boundaries are quantitative or binary or what, but each member either encodes me, or not.
I think, under more conventional definitions of “help” and “decision,” that telling me what to do, or showing me what I’m going to do, is kinda beside the point. A superintelligence that wanted to help me choose the best spouse might very well do something completely different, like hack someone’s Waymo to bump my car right when we’re both looking for someone new and in the right headspace to have a meet cute. I think that’s mostly like a fancier version of a friend trying to set people up by just sitting them next to each other at a dinner party, which I would definitely classify as helping (if done skillfully). Real superintelligences help with butterflies.
The central point is more that superintelligence won’t be able to help with your decisions, won’t be able to decide for you at some fundamental level, no matter how capable it is. It can help instrumentally, but not replace your ability to decide in order to find out what the decisions are, so in some sense it’s not able to help at all. I’m trying to capture the sense in which this holds regardless of its ability to precisely predict and determine outcomes in the physical world (if we only look at the state of the future, rather than full trajectories that get the world there).
When a program is given strange input, or if the computer it would be running on is destroyed, that event in the physical world usually doesn’t affect the semantics of the program that describes its behavior for all possible inputs. If you are weak and brittle, talking about what you’ll decide requires defining what we even mean in principle by a decision that is yours, only then can we ask if it does remain yours in actuality. Or if it does remain yours, even if you are yourself no longer present in actuality. Which is not very useful for keeping it (or yourself) present in actuality, but can be conceptually useful for formulating desiderata towards it being present in actuality.
So there are two claims. First, it’s in some sense natural for your decisions to remain yours, if you don’t start mindlessly parroting external inputs that dictate your actions, even in the face of superintelligence (in its aspect of capability, but not necessarily aimed in a way that disrupts you). Second, if you are strongly manipulated or otherwise overridden, this should in some sense mean that you are no longer present, that the resulting outcome doesn’t capture or simulate what we should define as being you (in order to talk about the decisions that are in principle yours). Thus presence of overpowering manipulation doesn’t contradict decisions usually remaining yours, it just requires that you are consequently no longer present to manifest them in actuality when that happens.
This seems like the third comment on the same concern, I’ve also answered it here and here, going into more detail on other related things. So there is a missing prerequisite post.
In case this helps you write that post, here are some things I am still confused about.
What, specifically, is the “you” that is no longer present in the scenario of strong manipulation?
In the “mindless parroting” scenario, what happens if the ASI magically disappeared? Does/can the “you” reappear? Under what circumstances?
Why is this not a fully general argument against other humans helping you make decisions? For example, if someone decides to mindlessly parrot everything a cult leader tells them, I agree there’s a sense in which they are no longer present. But the choice to obey is still theirs, and can be changed, and they can reappear if they change that one choice.
OTOH, if this is a fully general argument about anyone helping anyone else make decisions, that seems like a major (and underspecified) redefinition of both “help” and “decision.” It then seems like it’s premature to jump to focusing on ASI as a special case, and also I’m not sure why I’m supposed to care about these definitional changes?
“So it’s fair to say that we are already not physically present in the world, the things that are physically present are better described as kludgy and imperfect simulators”—I mean, yes, I see your point, but also, I am present implicitly in the structure of the physically-existing things. There is some set of arrangements-of-matter that I’d consider me, and others I would not. I don’t know if the set’s boundaries are quantitative or binary or what, but each member either encodes me, or not.
I think, under more conventional definitions of “help” and “decision,” that telling me what to do, or showing me what I’m going to do, is kinda beside the point. A superintelligence that wanted to help me choose the best spouse might very well do something completely different, like hack someone’s Waymo to bump my car right when we’re both looking for someone new and in the right headspace to have a meet cute. I think that’s mostly like a fancier version of a friend trying to set people up by just sitting them next to each other at a dinner party, which I would definitely classify as helping (if done skillfully). Real superintelligences help with butterflies.