Just as a data-point about intuition frequency, I found your intuitions about “a search algorithm which found the motor output solutions which maximized the original human’s preference” to be very surprising
Do you mean that the idea itself is weird and surprising to consider?
Or do you mean that my intuition that this search algorithm fits the definition of a “person” and is imbued with moral weight is surprising and does not match your moral intuition?
Just as a data-point about intuition frequency, I found your intuitions about “a search algorithm which found the motor output solutions which maximized the original human’s preference” to be very surprising
Do you mean that the idea itself is weird and surprising to consider?
Or do you mean that my intuition that this search algorithm fits the definition of a “person” and is imbued with moral weight is surprising and does not match your moral intuition?