What distinguishes a person from any other computation
A person is reflectively self aware.
and why does that particular distinction carry so much moral weight?
Evolution built me to care about humans, and upon reflection, the values I have include non-humans who have features like being reflectively self aware.
Is that what you would want to want, given the option, or is that a lizard-brain instinct that gets in the way of your ability to evaluate what’s really the right thing to do?
I can still interpret that either way. Do you mean that on reflection you realize that you emotionally desire that, or that on reflection you *decide” that that’s what’s important?
There’s also Hayekian arguments—self-aware agents are apt to accumulate information about their own desires and activities. Systems which allow that information to have an effect seem to be more capable.
A person is reflectively self aware.
Evolution built me to care about humans, and upon reflection, the values I have include non-humans who have features like being reflectively self aware.
Is that what you would want to want, given the option, or is that a lizard-brain instinct that gets in the way of your ability to evaluate what’s really the right thing to do?
I can still interpret that either way. Do you mean that on reflection you realize that you emotionally desire that, or that on reflection you *decide” that that’s what’s important?
There’s also Hayekian arguments—self-aware agents are apt to accumulate information about their own desires and activities. Systems which allow that information to have an effect seem to be more capable.