“The temptation is to ask what «AIs» will «want», forgetting that the space of minds-in-general is much wider than the tiny human dot”
Optional paragraph form:
“The critical challenge is not to predict that «AIs» will attack humanity with marching robot armies, or alternatively invent a cure for cancer. The task is not even to make the prediction for an arbitrary individual AI design. Rather, the task [for humanity to accomplish] is choosing into existence some particular powerful optimization process whose beneficial effects can legitimately be asserted.
[It’s best to avoid] thinking up reasons why a fully generic optimization process would be friendly. Natural selection isn’t friendly, nor does it hate you, nor will it leave you alone. Evolution cannot be so anthropomorphized, it does not work like you do”
Eliezer Yudkowsky, AI as a pos and neg factor, around 2006
“The temptation is to ask what «AIs» will «want», forgetting that the space of minds-in-general is much wider than the tiny human dot”
Optional paragraph form:
“The critical challenge is not to predict that «AIs» will attack humanity with marching robot armies, or alternatively invent a cure for cancer. The task is not even to make the prediction for an arbitrary individual AI design. Rather, the task [for humanity to accomplish] is choosing into existence some particular powerful optimization process whose beneficial effects can legitimately be asserted.
[It’s best to avoid] thinking up reasons why a fully generic optimization process would be friendly. Natural selection isn’t friendly, nor does it hate you, nor will it leave you alone. Evolution cannot be so anthropomorphized, it does not work like you do”
Eliezer Yudkowsky, AI as a pos and neg factor, around 2006