One crucial concern related to “what people want” is this seems underdefined, un-stable in interactions with wildly superintelligent systems, and prone to problems with scaling of values within systems where intelligence increases.
This is what I was referring to with
by assumption the superintelligence will be able to answer any question you’re able to operationalize about human values
The superintelligence can answer any operationalizable question about human values, but as you say, it’s not clear how to elicit the right operationalization.
This is what I was referring to with
The superintelligence can answer any operationalizable question about human values, but as you say, it’s not clear how to elicit the right operationalization.