Stuart_Armstrong comments on An overall schema for the friendly AI problems: self-referential convergence criteria

[Stuart_Armstrong]

See the point on Paul Christiano’s design. The problem I discussed applies not only to UFAIs but to other designs that seek to get round it, but use potentially unrestricted search.