Stuart_Armstrong comments on AI caught by a module that counterfactually doesn’t exist

[Stuart_Armstrong]

Isn’t that essentially a false beliefs about one’s own preferences?

No. It’s an adjusted preference that functions in practice just like a false belief.