The intelligence to calculate the CEV needs to be pre-FOOM.
No, a complete question to which CEV is an answer needs to be pre-FOOM. All an AI needs to know about morality before it is superintelligent is (1) how to arrive at a CEV-answer by looking at things and doing calculations and (2) how to look at things without breaking them and do calculations without breaking everything else.
Yes, but saying “we get the FAI to do it” is just moving the hard bit.
The intelligence to calculate the CEV needs to be pre-FOOM. We have general intelligences of pre-FOOM level already.
So: what would a pre-FOOM general intelligence actually do?
No, a complete question to which CEV is an answer needs to be pre-FOOM. All an AI needs to know about morality before it is superintelligent is (1) how to arrive at a CEV-answer by looking at things and doing calculations and (2) how to look at things without breaking them and do calculations without breaking everything else.
Ah, OK. So do we have any leads on how to ask the question?
I believe the idea is the have the pre-FOOM AI commit to doing the calculation first thing post-FOOM.
Eliezer beat me to the punch. My answer was approximately the same.