In case this helps you write that post, here are some things I am still confused about.
What, specifically, is the “you” that is no longer present in the scenario of strong manipulation?
In the “mindless parroting” scenario, what happens if the ASI magically disappeared? Does/can the “you” reappear? Under what circumstances?
Why is this not a fully general argument against other humans helping you make decisions? For example, if someone decides to mindlessly parrot everything a cult leader tells them, I agree there’s a sense in which they are no longer present. But the choice to obey is still theirs, and can be changed, and they can reappear if they change that one choice.
OTOH, if this is a fully general argument about anyone helping anyone else make decisions, that seems like a major (and underspecified) redefinition of both “help” and “decision.” It then seems like it’s premature to jump to focusing on ASI as a special case, and also I’m not sure why I’m supposed to care about these definitional changes?
“So it’s fair to say that we are already not physically present in the world, the things that are physically present are better described as kludgy and imperfect simulators”—I mean, yes, I see your point, but also, I am present implicitly in the structure of the physically-existing things. There is some set of arrangements-of-matter that I’d consider me, and others I would not. I don’t know if the set’s boundaries are quantitative or binary or what, but each member either encodes me, or not.
I think, under more conventional definitions of “help” and “decision,” that telling me what to do, or showing me what I’m going to do, is kinda beside the point. A superintelligence that wanted to help me choose the best spouse might very well do something completely different, like hack someone’s Waymo to bump my car right when we’re both looking for someone new and in the right headspace to have a meet cute. I think that’s mostly like a fancier version of a friend trying to set people up by just sitting them next to each other at a dinner party, which I would definitely classify as helping (if done skillfully). Real superintelligences help with butterflies.
In case this helps you write that post, here are some things I am still confused about.
What, specifically, is the “you” that is no longer present in the scenario of strong manipulation?
In the “mindless parroting” scenario, what happens if the ASI magically disappeared? Does/can the “you” reappear? Under what circumstances?
Why is this not a fully general argument against other humans helping you make decisions? For example, if someone decides to mindlessly parrot everything a cult leader tells them, I agree there’s a sense in which they are no longer present. But the choice to obey is still theirs, and can be changed, and they can reappear if they change that one choice.
OTOH, if this is a fully general argument about anyone helping anyone else make decisions, that seems like a major (and underspecified) redefinition of both “help” and “decision.” It then seems like it’s premature to jump to focusing on ASI as a special case, and also I’m not sure why I’m supposed to care about these definitional changes?
“So it’s fair to say that we are already not physically present in the world, the things that are physically present are better described as kludgy and imperfect simulators”—I mean, yes, I see your point, but also, I am present implicitly in the structure of the physically-existing things. There is some set of arrangements-of-matter that I’d consider me, and others I would not. I don’t know if the set’s boundaries are quantitative or binary or what, but each member either encodes me, or not.
I think, under more conventional definitions of “help” and “decision,” that telling me what to do, or showing me what I’m going to do, is kinda beside the point. A superintelligence that wanted to help me choose the best spouse might very well do something completely different, like hack someone’s Waymo to bump my car right when we’re both looking for someone new and in the right headspace to have a meet cute. I think that’s mostly like a fancier version of a friend trying to set people up by just sitting them next to each other at a dinner party, which I would definitely classify as helping (if done skillfully). Real superintelligences help with butterflies.