Stuart_Armstrong comments on An overall schema for the friendly AI problems: self-referential convergence criteria

Stuart_Armstrong 15 Jul 2015 9:53 UTC
0 points
Because FAI’s can change themselves very effectively in ways that we can’t.

It might be that human brain in computer software would have the same issues.
- Kaj_Sotala 15 Jul 2015 13:02 UTC
  4 points
  Parent
  
  Because FAI’s can change themselves very effectively in ways that we can’t.
  
  Doesn’t mean the FAI couldn’t remain genuinely uncertain about some value question, or consider it not worth solving at this time, or run into new value questions due to changed circumstances, etc.
  
  All of those could prevent reflective equilibria, while still being compatible with the ability for extensive self-modification.
  - Stuart_Armstrong 15 Jul 2015 15:34 UTC
    0 points
    Parent
    
    All of those could prevent reflective equilibria, while still being compatible with the ability for extensive self-modification.
    
    It’s possible. They feel very unstable, though.