As such, we’d be unlikely to get what we really want if the world was re-engineered in accordance with a description of what we want that came from verbal introspective access to our motivations.
Interesting as these experimental results are, it sounds to me like you’re saying that there’s a license to be human (or a license to be yourself, or a license to be your current self).
Suppose I found out that many of my actions that seemed random were actually subtly aimed at invading Moldova, perhaps because aliens with weird preferences placed some functional equivalent of mind control lasers in my brain, and suppose that this fact was not introspectively accessible to me; e.g., a future where Moldova is invaded does not feel more utopian to imagine than the alternatives. Isn’t there an important sense in which, in that hypothetical, I don’t care about invading Moldova? What if the mind control laser was outside my brain, perhaps in orbit? At what point do I get to say, “I won’t let my so-called preferences stop me from doing what’s right?”
My impression is that this mindset, where you determine what to do by looking closely at the world to see what you’re already doing, and then giving that precedence over what seems right, would be seen as an alien mindset by anyone not affected by certain subtle misunderstandings of the exact sense in which value is subjective. My impression is that once these misunderstandings go away and people ask themselves what considerations they’re really moved by, they’ll find out that where their utility function (or preferences or whatever) disagrees with what, on reflection, seems right, they genuinely don’t care (at least in any straightforward way) what their preferences are, paradoxical as that sounds.
My impression is that once these misunderstandings go away and people ask themselves what considerations they’re really moved by, they’ll find out that where their utility function (or preferences or whatever) disagrees with what, on reflection, seems right, they genuinely don’t care (at least in any straightforward way) what their preferences are, paradoxical as that sounds.
I think you would have a strong point if the arguments that really move us forms a coherent ethical system, but what if when people find out what they’re really moved by, it turns out not to be anything coherent, but just a semi-random set of “considerations” that happen to move a hodgepodge of neural circuits?
That certainly seems to be to some extent true of real humans, but the point is that even if I’m to some extent a random hodgepodge, this does not obviously create in me an impulse to consult a brain scan readout or a table of my counterfactual behaviors and then follow those at the expense of whatever my other semi-random considerations are causing me to feel is right.
this does not obviously create in me an impulse to consult a brain scan readout or a table of my counterfactual behaviors
Sure, unless one of the semi-random considerations that moves you is “Crap, my EV is not coherent. Well I don’t want to lay down and wait to die, so let’s just make an AI that will serve my current desires.” :)
Incoherent considerations aren’t all that bad. Even if someone prefers A to B, B to C, and C to A, they’ll just spend a lot of time switching rather than waiting to die. I guess that people probably prefer changing their considerations in general, so your example of a semi-random consideration is sufficient but not at all unique or uncommon.
Agreed. But depending on exactly what’s meant I think lukeprog is still correct in the statement that “we’d be unlikely to get what we really want if the world was re-engineered in accordance with a description of what we want that came from verbal introspective access to our motivations”, simply because the descriptions that people actually produce from this are so incomplete. We’d have to compile something from asking “Would you prefer Moldova to be invaded or not? Would you prefer...”, etc., since people wouldn’t even think of that question themselves. (And we’d probably need specific scenarios, not just “Moldova is invaded vs. not”.)
And since verbal introspection is so unreliable, a better check might be somehow actually simulating you in a world where Moldova is invaded vs. not, and seeing which you prefer. That may be getting a little too close to “license to be human” territory, since that obviously would be revealed preference, but due to human inconsistency—specifically, the fact that our preferences over actions don’t seem to always follow from preferences over consequences like they should—I’m not certain it’s necessarily the sort that gives us problems. It’s when you go by our preferences over actions that you get the real problems...
I agree with you, but I think there are a lot of LW people who didn’t really like the meta-ethics sequence or liked it but got something odd out of it and who basically think that most of what they value comes from genetic-evolutionary pressures (the aliens in your scenario). Luke’s post is very important for them if not for the rest of us who are more interested in where we’re getting our notion of ‘right’ from if not entirely from the aliens.
Suppose I found out that many of my actions that seemed random were actually subtly aimed at invading Moldova, perhaps because aliens with weird preferences placed some functional equivalent of mind control lasers in my brain
I suspect you’d prefer the aliens turn off their mind-control lasers, and if you had a choice you would have preferred they did not turn on the lasers in the first place.
Once you’re corrupted, you’re corrupted. At that point we have a mind-controlled Steven wandering around and there’s not much point in trying to learn about human motivation from the behavior of humans who are mind-controlled by aliens.
what if it’s not space aliens, but an alien god [really evolution]?
Well, then its unlikely that your random unconscious actions have any ulterior motive as sophisticated as invading Moldova. Your true desires are probably just some combination of increasing your status, activities prone to make babies, and your conscious desires, assuming the conscious desires haven’t been subverted by bad philosophy.
I don’t see much harm in activities prone to make babies, so the real question here is “If I my unconscious desires lead me to have poor relationships because I’m gaming them for status, and I don’t consciously value status, would I want to fix that by changing the unconscious desires?” I think I would, if I could be sure my income wouldn’t be affected much, and the fix was well tested, preferably on other people.
But in any case, human volition is going to look like a clump of mud. It has a more-or-less well defined position, but not exactly, and the boundaries are unclear.
Personally I find having an inconsistent mind so intolerable that as far as I know, I’d face a choice between
A: blocking the aliens out of my head completely
B: Assimilating with them completely.
Correspondingly I have endeavoured to establish a rapport with evolution’s design deep enough that I can either
A: Consciously adapt it to the epoch of intelligent agency, for example, instilling within it a fear of solar collapse, a sense of the kinship linking all life on earth, and a cognizance of extra-solar hunting grounds for it to aspire towards. These might sound like rationalizations of noble goals we’d communally established post hoc.. well yes, they would either way, I think those goals were only able to be ennobled upon the favour of evolution’s old intents of surviving and spreading.
B: Truly accept as not horrible and perfectly normal, the subjectively horrible unacceptable things that would drive most people away from forging this kind of self-rapport. I’d give examples but these are by their nature hard to index, as if they’re communicated tactfully, they don’t seem horrible at all.
But then, I was drawn to this thread for a reason. I wonder if all my progress under A is just a mat of rationalizations and if the reality of Her Design is too ugly for me to publicly embrace, and if that very design has been built to anticipate that, and that is why our vocal selves are blanketed with confusion as to our intents.
Thank you for pointing out the recurrent threat of empiric stupidity in these sorts of matters, namely the guiding empiricist assumption in this case that an empirical determination of our desires by outside instrumentation is going to result in an improvement to human affairs. We cannot overlook the way scientific empiricism can sometimes make people stupider at affairs requiring developed skill at reflective judgment.
Interesting as these experimental results are, it sounds to me like you’re saying that there’s a license to be human (or a license to be yourself, or a license to be your current self).
Suppose I found out that many of my actions that seemed random were actually subtly aimed at invading Moldova, perhaps because aliens with weird preferences placed some functional equivalent of mind control lasers in my brain, and suppose that this fact was not introspectively accessible to me; e.g., a future where Moldova is invaded does not feel more utopian to imagine than the alternatives. Isn’t there an important sense in which, in that hypothetical, I don’t care about invading Moldova? What if the mind control laser was outside my brain, perhaps in orbit? At what point do I get to say, “I won’t let my so-called preferences stop me from doing what’s right?”
My impression is that this mindset, where you determine what to do by looking closely at the world to see what you’re already doing, and then giving that precedence over what seems right, would be seen as an alien mindset by anyone not affected by certain subtle misunderstandings of the exact sense in which value is subjective. My impression is that once these misunderstandings go away and people ask themselves what considerations they’re really moved by, they’ll find out that where their utility function (or preferences or whatever) disagrees with what, on reflection, seems right, they genuinely don’t care (at least in any straightforward way) what their preferences are, paradoxical as that sounds.
Or am I somehow confused here?
I think you would have a strong point if the arguments that really move us forms a coherent ethical system, but what if when people find out what they’re really moved by, it turns out not to be anything coherent, but just a semi-random set of “considerations” that happen to move a hodgepodge of neural circuits?
That certainly seems to be to some extent true of real humans, but the point is that even if I’m to some extent a random hodgepodge, this does not obviously create in me an impulse to consult a brain scan readout or a table of my counterfactual behaviors and then follow those at the expense of whatever my other semi-random considerations are causing me to feel is right.
Sure, unless one of the semi-random considerations that moves you is “Crap, my EV is not coherent. Well I don’t want to lay down and wait to die, so let’s just make an AI that will serve my current desires.” :)
Incoherent considerations aren’t all that bad. Even if someone prefers A to B, B to C, and C to A, they’ll just spend a lot of time switching rather than waiting to die. I guess that people probably prefer changing their considerations in general, so your example of a semi-random consideration is sufficient but not at all unique or uncommon.
This is also a reason why looking closely at neuroscience seems like a dubious way of making progress on metaethics.
Agreed. But depending on exactly what’s meant I think lukeprog is still correct in the statement that “we’d be unlikely to get what we really want if the world was re-engineered in accordance with a description of what we want that came from verbal introspective access to our motivations”, simply because the descriptions that people actually produce from this are so incomplete. We’d have to compile something from asking “Would you prefer Moldova to be invaded or not? Would you prefer...”, etc., since people wouldn’t even think of that question themselves. (And we’d probably need specific scenarios, not just “Moldova is invaded vs. not”.)
And since verbal introspection is so unreliable, a better check might be somehow actually simulating you in a world where Moldova is invaded vs. not, and seeing which you prefer. That may be getting a little too close to “license to be human” territory, since that obviously would be revealed preference, but due to human inconsistency—specifically, the fact that our preferences over actions don’t seem to always follow from preferences over consequences like they should—I’m not certain it’s necessarily the sort that gives us problems. It’s when you go by our preferences over actions that you get the real problems...
I agree with you, but I think there are a lot of LW people who didn’t really like the meta-ethics sequence or liked it but got something odd out of it and who basically think that most of what they value comes from genetic-evolutionary pressures (the aliens in your scenario). Luke’s post is very important for them if not for the rest of us who are more interested in where we’re getting our notion of ‘right’ from if not entirely from the aliens.
I suspect you’d prefer the aliens turn off their mind-control lasers, and if you had a choice you would have preferred they did not turn on the lasers in the first place.
Once you’re corrupted, you’re corrupted. At that point we have a mind-controlled Steven wandering around and there’s not much point in trying to learn about human motivation from the behavior of humans who are mind-controlled by aliens.
So the next question is, what if it’s not space aliens, but an alien god?
Well, then its unlikely that your random unconscious actions have any ulterior motive as sophisticated as invading Moldova. Your true desires are probably just some combination of increasing your status, activities prone to make babies, and your conscious desires, assuming the conscious desires haven’t been subverted by bad philosophy.
I don’t see much harm in activities prone to make babies, so the real question here is “If I my unconscious desires lead me to have poor relationships because I’m gaming them for status, and I don’t consciously value status, would I want to fix that by changing the unconscious desires?” I think I would, if I could be sure my income wouldn’t be affected much, and the fix was well tested, preferably on other people.
But in any case, human volition is going to look like a clump of mud. It has a more-or-less well defined position, but not exactly, and the boundaries are unclear.
Personally I find having an inconsistent mind so intolerable that as far as I know, I’d face a choice between
A: blocking the aliens out of my head completely
B: Assimilating with them completely.
Correspondingly I have endeavoured to establish a rapport with evolution’s design deep enough that I can either
A: Consciously adapt it to the epoch of intelligent agency, for example, instilling within it a fear of solar collapse, a sense of the kinship linking all life on earth, and a cognizance of extra-solar hunting grounds for it to aspire towards. These might sound like rationalizations of noble goals we’d communally established post hoc.. well yes, they would either way, I think those goals were only able to be ennobled upon the favour of evolution’s old intents of surviving and spreading.
B: Truly accept as not horrible and perfectly normal, the subjectively horrible unacceptable things that would drive most people away from forging this kind of self-rapport. I’d give examples but these are by their nature hard to index, as if they’re communicated tactfully, they don’t seem horrible at all.
But then, I was drawn to this thread for a reason. I wonder if all my progress under A is just a mat of rationalizations and if the reality of Her Design is too ugly for me to publicly embrace, and if that very design has been built to anticipate that, and that is why our vocal selves are blanketed with confusion as to our intents.
Thank you for pointing out the recurrent threat of empiric stupidity in these sorts of matters, namely the guiding empiricist assumption in this case that an empirical determination of our desires by outside instrumentation is going to result in an improvement to human affairs. We cannot overlook the way scientific empiricism can sometimes make people stupider at affairs requiring developed skill at reflective judgment.