The scariest kind of dream, perhaps, is exemplified by someone with merely human intelligence who wants to hastily rewrite their own values to conform to their favorite ideology. We’d want an implementation of CEV to recognize this as a bad step in extrapolation. The question is, how do we define what is a “bad step”?
That “best self” thing makes me nervous. People have a wide range of what they think they ought to be, and some of those dreams are ill-conceived.
The scariest kind of dream, perhaps, is exemplified by someone with merely human intelligence who wants to hastily rewrite their own values to conform to their favorite ideology. We’d want an implementation of CEV to recognize this as a bad step in extrapolation. The question is, how do we define what is a “bad step”?