Well, human values are probably variant to some degree between humans, so a Friendly AI wouldn’t so much be ‘maximize generic human utility function’ as ‘take all the human utility functions you can find as of now, find those portions which are reflexively consistent, weight them by frequency, and take those actions that are best supported by the convergent portions of those utility functions.’ At least, that was the gist of CEV circa 2004. Not sure what Eliezer and co are working on these days, but that sounds like a reasonable way to build a nice future to me. A fair one, at least.
Well, human values are probably variant to some degree between humans, so a Friendly AI wouldn’t so much be ‘maximize generic human utility function’ as ‘take all the human utility functions you can find as of now, find those portions which are reflexively consistent, weight them by frequency, and take those actions that are best supported by the convergent portions of those utility functions.’ At least, that was the gist of CEV circa 2004. Not sure what Eliezer and co are working on these days, but that sounds like a reasonable way to build a nice future to me. A fair one, at least.