The real issue with the orthogonality thesis, at least for the defensible version, is that it makes a claim that I think is true if we quantify over all possible intelligences, but the problem is that this has basically ~no implications for AI safety, even if rationalists are right, because it is far too weak to support AI risk claims.
It only says that an AI can have arbitrary goals, not that it is likely or will have arbitrary goals. You can justify any probability of AI having bad goals from probability 0 to probability 1, and everything in between.
Outside of history, it is completely useless, since it doesn’t constrain your anticipation very much.
Agree specifically on that the argument is bad, but someone might have other reasons for accelerating AI progress that do not depend on the orthogonality thesis being false.
The real issue with the orthogonality thesis, at least for the defensible version, is that it makes a claim that I think is true if we quantify over all possible intelligences, but the problem is that this has basically ~no implications for AI safety, even if rationalists are right, because it is far too weak to support AI risk claims.
It only says that an AI can have arbitrary goals, not that it is likely or will have arbitrary goals. You can justify any probability of AI having bad goals from probability 0 to probability 1, and everything in between.
Outside of history, it is completely useless, since it doesn’t constrain your anticipation very much.
I feel like you are not really addressing my argument. Don’t you agree that the orthogonality thesis shoots down E. Yudkowsky’s argument?
Agree specifically on that the argument is bad, but someone might have other reasons for accelerating AI progress that do not depend on the orthogonality thesis being false.
I agree that the orthogonality thesis doesn’t by itself prove most things related to AI safety.