wedrifid comments on Two questions about CEV that worry me

wedrifid 23 Dec 2010 17:08 UTC
0 points

I’m still wondering how you’d calculate a CEV. I’m still wondering how you’d calculate one human’s volition.

The implication is that you let the FAI do it. (The task in building the FAI so that it will do this has somewhat different challenges.)
- David_Gerard 23 Dec 2010 17:43 UTC
  5 points
  Parent
  Yes, but saying “we get the FAI to do it” is just moving the hard bit.
  
  The intelligence to calculate the CEV needs to be pre-FOOM. We have general intelligences of pre-FOOM level already.
  
  So: what would a pre-FOOM general intelligence actually do?
  - Eliezer Yudkowsky 23 Dec 2010 21:38 UTC
    18 points
    Parent
    
    The intelligence to calculate the CEV needs to be pre-FOOM.
    
    No, a complete question to which CEV is an answer needs to be pre-FOOM. All an AI needs to know about morality before it is superintelligent is (1) how to arrive at a CEV-answer by looking at things and doing calculations and (2) how to look at things without breaking them and do calculations without breaking everything else.
    - David_Gerard 24 Dec 2010 10:53 UTC
      3 points
      Parent
      Ah, OK. So do we have any leads on how to ask the question?
  - Eugine_Nier 23 Dec 2010 21:17 UTC
    1 point
    Parent
    I believe the idea is the have the pre-FOOM AI commit to doing the calculation first thing post-FOOM.
  - wedrifid 24 Dec 2010 2:31 UTC
    0 points
    Parent
    Eliezer beat me to the punch. My answer was approximately the same.