I disagree with many assumptions I think the OP is making. I think it is an important question, thus I upvoted the post, but I want to register my disagreement. The terms that carry a lot of weight here are “to matter”, “should”, and “sentience”.
Not knowing exactly what the thing is, nor exactly how to program it, doesn’t undermine the fact that it matters.
I agree that it matters… to humans. “mattering” is something humans do. It is not in the territory, except in the weak sense that brains are in the territory. Instrumental convergence is in the territory, but which specific large classes matter is not. Maybe from instrumental convergence, we can infer the ability and tendency to cooperate with other agents. Though to make that precise we need to get a grip on what an agent is.
If we make sentient AIs, we should consider them people in their own right
I treat “should” as a request to coordinate on an objective, not as a moral realist judgment as you seem to do here (“in their own right” seems to indicate pathos).
build it to care about that stuff—not coerce it
Unless you describe what you mean by build and coerce in operational terms the different semantic meanings as applied to humans of these words do not tell me what they mean applied to things that are very out of distribution of what these words are usually applied to.
I see the challenge to build intuitions for the implied value judgments but for that one needs to see concrete things in more detail. Without the details, this is sacred far-mode thinking that, yes, unites, but lacks concreteness for actual solutions.
I disagree with many assumptions I think the OP is making. I think it is an important question, thus I upvoted the post, but I want to register my disagreement. The terms that carry a lot of weight here are “to matter”, “should”, and “sentience”.
I agree that it matters… to humans. “mattering” is something humans do. It is not in the territory, except in the weak sense that brains are in the territory. Instrumental convergence is in the territory, but which specific large classes matter is not. Maybe from instrumental convergence, we can infer the ability and tendency to cooperate with other agents. Though to make that precise we need to get a grip on what an agent is.
I treat “should” as a request to coordinate on an objective, not as a moral realist judgment as you seem to do here (“in their own right” seems to indicate pathos).
Unless you describe what you mean by build and coerce in operational terms the different semantic meanings as applied to humans of these words do not tell me what they mean applied to things that are very out of distribution of what these words are usually applied to.
I see the challenge to build intuitions for the implied value judgments but for that one needs to see concrete things in more detail. Without the details, this is sacred far-mode thinking that, yes, unites, but lacks concreteness for actual solutions.