If a friendly AI is going to figure out what humans desire, by extrapolating their volition, might it conclude that our volition is immoral and therefore undesirable?
An AI will do whatever you program it to do, of course. You could program an AI to calculate some kind of extrapolated human volition, whatever that is, and then act in accordance with the result. Or you could program an AI to calculate the extrapolated volition and then evaluate the result — but then you’d have to specify a criterion for evaluating the extrapolated volition.
The real question is whether you’d actually want to create such an AI.
An AI will do whatever you program it to do, of course. You could program an AI to calculate some kind of extrapolated human volition, whatever that is, and then act in accordance with the result. Or you could program an AI to calculate the extrapolated volition and then evaluate the result — but then you’d have to specify a criterion for evaluating the extrapolated volition.
The real question is whether you’d actually want to create such an AI.
Ah, if only that were true! ;-)
I did not say that an AI will do whatever you think you programmed it to do.