My comment wasn’t directed towards CEV at all—CEV sounds like a sensible working definition of “friendly enough”, and I agree that it’s probably computationally hard.
I was suggesting that any program, AI or no, that is coded to rewrite critical parts of itself in substantial ways is likely to go “splat”, not “FOOM”—to degenerate into something that doesn’t work at all.
My comment wasn’t directed towards CEV at all—CEV sounds like a sensible working definition of “friendly enough”, and I agree that it’s probably computationally hard.
I was suggesting that any program, AI or no, that is coded to rewrite critical parts of itself in substantial ways is likely to go “splat”, not “FOOM”—to degenerate into something that doesn’t work at all.