I just thought about something. Could it be that we implicitly assign negative value to the existence of human-like minds who’s utilities are just slightly off in a obvious way? Is part of the aversion to wireheading a uncanny valley effect?
Thinking of a creature that evolved or was selected to enjoy being used as a beast of labour by caretakers seems more ok the more different its mind is from ours (lets say for the sake of argument it does have human or superhuman level intelligence).
Why is my sympathy tied to this? Is this a case of my neural circutry being incapable of emulating what it would be like to be such a creature? Or is the failure in the first example, since I try to use my mind to emulate a mind that while otherwise similar has something vital I can’t understand changed?
I’d expect that the Less Wrong community would tend to be unusually independent and averse to hierarchies. But much of humanity is accustomed to obedience, and even regards obedience to authority as a positive good. Some societies are more authoritarian than others, but duty, and humility and respect to superiors are commonly praised as virtues. Disobedience, whining, and malingering are considered bad.
Lots and lots of parents devote a lot of effort to raising their children to be obedient and respectful, not for cynical reasons, but out of love. They deliberately train their children to be servile, not only within the family, but in the context of school and adult life. I take it that some degree of this is traditional throughout the world.
I just thought about something. Could it be that we implicitly assign negative value to the existence of human-like minds who’s utilities are just slightly off in a obvious way? Is part of the aversion to wireheading a uncanny valley effect?
Thinking of a creature that evolved or was selected to enjoy being used as a beast of labour by caretakers seems more ok the more different its mind is from ours (lets say for the sake of argument it does have human or superhuman level intelligence).
Why is my sympathy tied to this? Is this a case of my neural circutry being incapable of emulating what it would be like to be such a creature? Or is the failure in the first example, since I try to use my mind to emulate a mind that while otherwise similar has something vital I can’t understand changed?
I’d expect that the Less Wrong community would tend to be unusually independent and averse to hierarchies. But much of humanity is accustomed to obedience, and even regards obedience to authority as a positive good. Some societies are more authoritarian than others, but duty, and humility and respect to superiors are commonly praised as virtues. Disobedience, whining, and malingering are considered bad.
Lots and lots of parents devote a lot of effort to raising their children to be obedient and respectful, not for cynical reasons, but out of love. They deliberately train their children to be servile, not only within the family, but in the context of school and adult life. I take it that some degree of this is traditional throughout the world.