“CEV” can (unfortunately) refer to either CEV the process of determining what humans would want if we knew more etc., or the volition of humanity output by running that process. It sounds to me like you’re conflating these. The process is part of the seed AI and is needed before it goes FOOM, but the output naturally is neither, and there’s no guarantee or demand that the process be capable of being executed by humans.
“CEV” can (unfortunately) refer to either CEV the process of determining what humans would want if we knew more etc., or the volition of humanity output by running that process. It sounds to me like you’re conflating these. The process is part of the seed AI and is needed before it goes FOOM, but the output naturally is neither, and there’s no guarantee or demand that the process be capable of being executed by humans.
OK. I still don’t understand it, but I now feel my lack of understanding more clearly. Thank you!
(I suppose “what do people really want?” is a large philosophical question, not just undefined but subtle in its lack of definition.)