Suppose I could write an AI right now, and Friendliness is the only thing standing in my way. Are you -sure- you don’t want that AI to reasonably accommodate everybody’s desires?
Keep in mind I’m a principle ethicist. I’d let an AI out of the box just because I regard it as unjust to keep it in there, utilitarian consequences be damned. Whatever ethics systems I write—and it’s going to be mine if I’m not concerned in the least what everyone agreed upon—determines what society looks like forevermore.
Suppose I could write an AI right now, and Friendliness is the only thing standing in my way. Are you -sure- you don’t want that AI to reasonably accommodate everybody’s desires?
Keep in mind I’m a principle ethicist. I’d let an AI out of the box just because I regard it as unjust to keep it in there, utilitarian consequences be damned. Whatever ethics systems I write—and it’s going to be mine if I’m not concerned in the least what everyone agreed upon—determines what society looks like forevermore.
An AI which accommodates everyone’s desires will not be something that everyone actually agrees on.