First, I think you underestimate the selection pressures your “caring about your brother” function has been under. That’s not a mechanistic argument, I know, but bear with me here.
That function wasn’t produced by evolution, which you know. It wasn’t produced by the reward circuitry either, nor your own deliberations. Rather, it was produced by thousands of years of culture and adversity and trial-and-error.
A Stone Age or a medieval human, if given superintelligent power, would probably make life miserable for their loved ones, because they don’t have the sophisticated insights into psychology and moral philosophy and meta-cognition that we use to implement our “caring” function. Stone-Agers/medievals would get hijacked by some bright idea or ideology or a mistaken conception of what they imagine their loved ones want or ought to want, and would just inflict that on them. You might object that this means they don’t “really” care — well yes, that’s the point!
And that sort of thing has been happening for generations, on a smaller scale. And is still happening: I expect that if you chose a random pair of people who’d profess to (and genuinely believe to) be each other’s loved ones, and then you amplified one of them to superintelligent godhood, they’d likely make the second one’s life fairly miserable.
Most people have not actually thought all of this through to the extent you or other LW-types have; they’re not altruistic in the correct nuanced way, and don’t have enough meta-cognition to recognize when they’re screwing up on that. Their utopias would be fairly dystopian or lethal for most people. They would not course-correct after their victims object, either: they don’t have enough meta-cognition for that either. They’d dig their heels in. If not impulsively self-modify into a monster on the spot.
The reason some of the modern people, who’d made a concentrated effort to become kind, can fairly credibly claim to genuinely care for others, is because their caring functions are perfected. They’d been perfected by generations of victims of imperfect caring, who’d pushed back on the imperfections, and by scientists and philosophers who took such feedback into account and compiled ever-better ways to care about people in a way that care-receivers would endorse. And care-receivers having the power to force the care-givers to go along with their wishes was a load-bearing part in this process.
It doesn’t answer the question of how it works, yes. But the argument that it’s trivial or simple is invalid. Plausibly, there might not actually be a billion examples of this.
But to the extent that there are people who genuinely care about others in a way we here would recognize as “real”, in a way that’s robust to intelligence amplification, I think they do have perfect caring functions, or a perfect pointer to a caring function, or something. Some part of the system is perfect.
The claim about Stone Age people seems probably false to me—I think if Stone Age people could understand what they were actually doing (not at the level of psychology or morality, but at the purely “physical” level), they would probably do lots of very nice things for their friends and family, in particular give them a lot of resources. However, even if it is true, I don’t think the reason we have gotten better is because of philosophy—I think it’s because we’re smarter in a more general way. Stone Age people were uneducated and had less good nutrition than us; they were literally just stupid.
Education is part of what I’m talking about. Modern humans iterate on the output of thousands of years of cultural evolution, their basic framework of how the world works is drastically different from the ancestral ones. Implicit and explicit lessons of how to care about people without e. g. violating their agency come part and parcel with it.
I think if Stone Age people could understand what they were actually doing (not at the level of psychology or morality, but at the purely “physical” level), they would probably do lots of very nice things for their friends and family
At the basic level, why do you think that their idea of “nice things” would be nuanced enough to understand that, say, non-consensual wireheading is not a nice thing? Some modern people don’t.
Stone Age people didn’t live a very comfortable life by modern standards, the experience of pleasure and escape from physical ailments would be common aspirations, while the high-cognitive-tech ideas of “self-actualization” would make no native sense to them. Why would a newly-ascended Stone Age god not assume that making everyone experience boundless pleasure free of pain forever is not the greatest thing there could possibly be? Would it occur to that god to even think carefully about whether such assumptions are right?
Edit: More than that, ancient people’s world-models are potentially much more alien and primitive than we can easily imagine. I refer you to the speculations in the section 2 here. The whole “voices of the gods” thing in the rest of the post is probably wrong, but I find it easy to believe that the basic principles of theory-of-mind that we take for granted are not something any human would independently invent. And if so, who knows what people without it would consider the maximally best way to be nice to someone?
I think the world modelling improvements from modern science and IQ raising social advances can be analytically separated from changes in our approach to welfare. As for non consensual wireheading, I am uncertain as to the moral status of this, so it seems like partially we just disagree about values. I am also uncertain as to the attitude of Stone Age people towards this—while your argument seems plausible, the fact that early philosophers like the Ancient Greeks were not pure hedonists in the wireheading sense but valued flourishing seems like evidence against this, suggesting that favoring non consensual wireheading is downstream of modern developments in utilitarianism.
favoring non consensual wireheading is downstream of modern developments in utilitarianism
Fair enough, I suppose. My point is more— Okay, let’s put the Stone Age people aside for a moment and think about the medieval people instead. Many of them were religious and superstitious and nationalistic, as the result of being raised on the diet of various unwholesome ideologies. These ideologies often had their own ideas of “the greater good” that they tried to sell people, ideas no less un-nice than non-consensual wireheading. Thus, a large fraction of humanity for the majority of its history endorsed views that would be catastrophic if scaled up.
I just assume this naturally extrapolates backwards to the Stone Age. Stone-age people had their own superstitions and spiritual traditions, and rudimentary proto-ideologies. I assume that these would also be catastrophic if scaled up.
Note that I’m not saying that the people in the past were literally alien, to the extent that they wouldn’t be able to converge towards modern moral views if we e. g. resurrected and educated one of them (and slightly intelligence-amplified them to account for worse nutrition, though I’m not as convinced that it’d be necessary as you), then let them do value reflection. But this process of “education” would need to be set up very carefully, in a way that might need to be “perfect”.
My argument is simply that if we granted godhood to one of these people and let them manage this process themselves, that will doom the light cone.
First, I think you underestimate the selection pressures your “caring about your brother” function has been under. That’s not a mechanistic argument, I know, but bear with me here.
That function wasn’t produced by evolution, which you know. It wasn’t produced by the reward circuitry either, nor your own deliberations. Rather, it was produced by thousands of years of culture and adversity and trial-and-error.
A Stone Age or a medieval human, if given superintelligent power, would probably make life miserable for their loved ones, because they don’t have the sophisticated insights into psychology and moral philosophy and meta-cognition that we use to implement our “caring” function. Stone-Agers/medievals would get hijacked by some bright idea or ideology or a mistaken conception of what they imagine their loved ones want or ought to want, and would just inflict that on them. You might object that this means they don’t “really” care — well yes, that’s the point!
And that sort of thing has been happening for generations, on a smaller scale. And is still happening: I expect that if you chose a random pair of people who’d profess to (and genuinely believe to) be each other’s loved ones, and then you amplified one of them to superintelligent godhood, they’d likely make the second one’s life fairly miserable.
Most people have not actually thought all of this through to the extent you or other LW-types have; they’re not altruistic in the correct nuanced way, and don’t have enough meta-cognition to recognize when they’re screwing up on that. Their utopias would be fairly dystopian or lethal for most people. They would not course-correct after their victims object, either: they don’t have enough meta-cognition for that either. They’d dig their heels in. If not impulsively self-modify into a monster on the spot.
The reason some of the modern people, who’d made a concentrated effort to become kind, can fairly credibly claim to genuinely care for others, is because their caring functions are perfected. They’d been perfected by generations of victims of imperfect caring, who’d pushed back on the imperfections, and by scientists and philosophers who took such feedback into account and compiled ever-better ways to care about people in a way that care-receivers would endorse. And care-receivers having the power to force the care-givers to go along with their wishes was a load-bearing part in this process.
It doesn’t answer the question of how it works, yes. But the argument that it’s trivial or simple is invalid. Plausibly, there might not actually be a billion examples of this.
But to the extent that there are people who genuinely care about others in a way we here would recognize as “real”, in a way that’s robust to intelligence amplification, I think they do have perfect caring functions, or a perfect pointer to a caring function, or something. Some part of the system is perfect.
The claim about Stone Age people seems probably false to me—I think if Stone Age people could understand what they were actually doing (not at the level of psychology or morality, but at the purely “physical” level), they would probably do lots of very nice things for their friends and family, in particular give them a lot of resources. However, even if it is true, I don’t think the reason we have gotten better is because of philosophy—I think it’s because we’re smarter in a more general way. Stone Age people were uneducated and had less good nutrition than us; they were literally just stupid.
Education is part of what I’m talking about. Modern humans iterate on the output of thousands of years of cultural evolution, their basic framework of how the world works is drastically different from the ancestral ones. Implicit and explicit lessons of how to care about people without e. g. violating their agency come part and parcel with it.
At the basic level, why do you think that their idea of “nice things” would be nuanced enough to understand that, say, non-consensual wireheading is not a nice thing? Some modern people don’t.
Stone Age people didn’t live a very comfortable life by modern standards, the experience of pleasure and escape from physical ailments would be common aspirations, while the high-cognitive-tech ideas of “self-actualization” would make no native sense to them. Why would a newly-ascended Stone Age god not assume that making everyone experience boundless pleasure free of pain forever is not the greatest thing there could possibly be? Would it occur to that god to even think carefully about whether such assumptions are right?
Edit: More than that, ancient people’s world-models are potentially much more alien and primitive than we can easily imagine. I refer you to the speculations in the section 2 here. The whole “voices of the gods” thing in the rest of the post is probably wrong, but I find it easy to believe that the basic principles of theory-of-mind that we take for granted are not something any human would independently invent. And if so, who knows what people without it would consider the maximally best way to be nice to someone?
I think the world modelling improvements from modern science and IQ raising social advances can be analytically separated from changes in our approach to welfare. As for non consensual wireheading, I am uncertain as to the moral status of this, so it seems like partially we just disagree about values. I am also uncertain as to the attitude of Stone Age people towards this—while your argument seems plausible, the fact that early philosophers like the Ancient Greeks were not pure hedonists in the wireheading sense but valued flourishing seems like evidence against this, suggesting that favoring non consensual wireheading is downstream of modern developments in utilitarianism.
Fair enough, I suppose. My point is more— Okay, let’s put the Stone Age people aside for a moment and think about the medieval people instead. Many of them were religious and superstitious and nationalistic, as the result of being raised on the diet of various unwholesome ideologies. These ideologies often had their own ideas of “the greater good” that they tried to sell people, ideas no less un-nice than non-consensual wireheading. Thus, a large fraction of humanity for the majority of its history endorsed views that would be catastrophic if scaled up.
I just assume this naturally extrapolates backwards to the Stone Age. Stone-age people had their own superstitions and spiritual traditions, and rudimentary proto-ideologies. I assume that these would also be catastrophic if scaled up.
Note that I’m not saying that the people in the past were literally alien, to the extent that they wouldn’t be able to converge towards modern moral views if we e. g. resurrected and educated one of them (and slightly intelligence-amplified them to account for worse nutrition, though I’m not as convinced that it’d be necessary as you), then let them do value reflection. But this process of “education” would need to be set up very carefully, in a way that might need to be “perfect”.
My argument is simply that if we granted godhood to one of these people and let them manage this process themselves, that will doom the light cone.