Next, John suggests that “human values” may be such a “natural abstraction”, such that “human values” may wind up a “prominent” member of an AI’s latent space, so to speak.
I’m fairly confident that the inputs to human values are natural abstractions—i.e. the “things we care about” are things like trees, cars, other humans, etc, not low-level quantum fields or “head or thumb but not any other body part”. (The “head or thumb” thing is a great example, by the way). I’m much less confident that human values themselves are a natural abstraction, for exactly the same reasons you gave.
I’m fairly confident that the inputs to human values are natural abstractions—i.e. the “things we care about” are things like trees, cars, other humans, etc, not low-level quantum fields or “head or thumb but not any other body part”. (The “head or thumb” thing is a great example, by the way). I’m much less confident that human values themselves are a natural abstraction, for exactly the same reasons you gave.