Suppose that the more powerful being is aligned to the less powerful: that is to say that (as should be the case in the babysitting example you give) the more powerful being’s fundamental motive is the well-being of the less powerful being.. Assume also that a lot of the asymmetry is of intellectual capacity: the more powerful being is also a great deal smarter. I think the likely and correct outcome is that there isn’t always consent, the less powerful being is frequently being manipulated into actions and reactions that they haven’t actually consented to, and might not even be capable of realizing why they should consent to — but ones that, if they were as intellectually capable as the more powerful being, they would in fact consent to.
I also think that,. for situations where the less powerful being is able to understand the alternatives and make an rational and informed decision, and wants to, the more powerful should give them the option and let them do so.. That’s the polite, respectful way to do things But often that isn’t going to be practical, or desirable. and the baby sitter should just distract the baby before they get into the dangerous situation.
Consent is a concept that fundamentally assumes that I am the best person available to make decisions about my own well-being. Outside parental situations, for interactions between evolved intelligence like humans, that’s almost invariably true. But if I had a superintelligence aligned to me, then yes, I would want it to keep me away from dangers so complex that I’m not capable of making an informed decision about them.
Let’s consider three ways you can be altruistic towards another agent:
You care about their welfare: some metric of how good their life is (as defined by you). I’ll call this care-morality—it endorses things like promoting their happiness, reducing their suffering, and hedonic utilitarian behavior (if you care about many agents).
You care about their agency: their ability to achieve their goals (as defined by them). I’ll call this cooperation-morality—it endorses things like honesty, fairness, deontological behavior towards others, and some virtues (like honor).
You care about obedience to them. I’ll call this deference-morality—it endorses things like loyalty, humility, and respect for authority.
[...]
Care-morality mainly makes sense as an attitude towards agents who are much less capable than you, and/or can’t make decisions for themselves—for example animals, future people, and infants.
[...]
Cooperation-morality mainly makes sense as an attitude towards agents whose capabilities are comparable to yours—for example others around us who are trying to influence the world.
[...]
Deference-morality mainly makes sense as an attitude towards trustworthy agents who are much more capable than you—for example effective leaders, organizations, communities, and sometimes society as a whole.
Thanks for this! I think the categories of morality is a useful framework. I am very wary of the judgement that care-morality is appropriate for less capable subjects—basically because of paternalism.
I think at some level, maybe a crux is that I believe that the harder version of the problem is more useful to solve, where we cannot remove the power differential, or at best cannot remove it totally, or at least do better than society does under such power differentials.
Also, maybe I view paternalism in a more positive context, especially as it relates to parenting, especially for legal guardians, as well as raising animals, where I’d argue that the power differential shouldn’t be removed.
Suppose that the more powerful being is aligned to the less powerful: that is to say that (as should be the case in the babysitting example you give) the more powerful being’s fundamental motive is the well-being of the less powerful being.. Assume also that a lot of the asymmetry is of intellectual capacity: the more powerful being is also a great deal smarter. I think the likely and correct outcome is that there isn’t always consent, the less powerful being is frequently being manipulated into actions and reactions that they haven’t actually consented to, and might not even be capable of realizing why they should consent to — but ones that, if they were as intellectually capable as the more powerful being, they would in fact consent to.
I also think that,. for situations where the less powerful being is able to understand the alternatives and make an rational and informed decision, and wants to, the more powerful should give them the option and let them do so.. That’s the polite, respectful way to do things But often that isn’t going to be practical, or desirable. and the baby sitter should just distract the baby before they get into the dangerous situation.
Consent is a concept that fundamentally assumes that I am the best person available to make decisions about my own well-being. Outside parental situations, for interactions between evolved intelligence like humans, that’s almost invariably true. But if I had a superintelligence aligned to me, then yes, I would want it to keep me away from dangers so complex that I’m not capable of making an informed decision about them.
Relevant post by Richard Ngo: “Moral Strategies at different capability levels”. Crucial excerpt:
Thanks for this! I think the categories of morality is a useful framework. I am very wary of the judgement that care-morality is appropriate for less capable subjects—basically because of paternalism.
I think at some level, maybe a crux is that I believe that the harder version of the problem is more useful to solve, where we cannot remove the power differential, or at best cannot remove it totally, or at least do better than society does under such power differentials.
Also, maybe I view paternalism in a more positive context, especially as it relates to parenting, especially for legal guardians, as well as raising animals, where I’d argue that the power differential shouldn’t be removed.