As far as I can tell, people typically use the orthogonality thesis to argue that smart agents could have any motivations. But the orthogonality thesis is stronger than that, and its extra content is false—there are some goals that are too complicated for a dumb agent to have, because the agent couldn’t understand those goals. I think people should instead directly defend the claim that smart agents could have arbitrary goals.
As far as I can tell, people typically use the orthogonality thesis to argue that smart agents could have any motivations. But the orthogonality thesis is stronger than that, and its extra content is false—there are some goals that are too complicated for a dumb agent to have, because the agent couldn’t understand those goals. I think people should instead directly defend the claim that smart agents could have arbitrary goals.
I no longer endorse this claim about what the orthogonality thesis says.