favoring non consensual wireheading is downstream of modern developments in utilitarianism
Fair enough, I suppose. My point is more— Okay, let’s put the Stone Age people aside for a moment and think about the medieval people instead. Many of them were religious and superstitious and nationalistic, as the result of being raised on the diet of various unwholesome ideologies. These ideologies often had their own ideas of “the greater good” that they tried to sell people, ideas no less un-nice than non-consensual wireheading. Thus, a large fraction of humanity for the majority of its history endorsed views that would be catastrophic if scaled up.
I just assume this naturally extrapolates backwards to the Stone Age. Stone-age people had their own superstitions and spiritual traditions, and rudimentary proto-ideologies. I assume that these would also be catastrophic if scaled up.
Note that I’m not saying that the people in the past were literally alien, to the extent that they wouldn’t be able to converge towards modern moral views if we e. g. resurrected and educated one of them (and slightly intelligence-amplified them to account for worse nutrition, though I’m not as convinced that it’d be necessary as you), then let them do value reflection. But this process of “education” would need to be set up very carefully, in a way that might need to be “perfect”.
My argument is simply that if we granted godhood to one of these people and let them manage this process themselves, that will doom the light cone.
Fair enough, I suppose. My point is more— Okay, let’s put the Stone Age people aside for a moment and think about the medieval people instead. Many of them were religious and superstitious and nationalistic, as the result of being raised on the diet of various unwholesome ideologies. These ideologies often had their own ideas of “the greater good” that they tried to sell people, ideas no less un-nice than non-consensual wireheading. Thus, a large fraction of humanity for the majority of its history endorsed views that would be catastrophic if scaled up.
I just assume this naturally extrapolates backwards to the Stone Age. Stone-age people had their own superstitions and spiritual traditions, and rudimentary proto-ideologies. I assume that these would also be catastrophic if scaled up.
Note that I’m not saying that the people in the past were literally alien, to the extent that they wouldn’t be able to converge towards modern moral views if we e. g. resurrected and educated one of them (and slightly intelligence-amplified them to account for worse nutrition, though I’m not as convinced that it’d be necessary as you), then let them do value reflection. But this process of “education” would need to be set up very carefully, in a way that might need to be “perfect”.
My argument is simply that if we granted godhood to one of these people and let them manage this process themselves, that will doom the light cone.