By creating an “orthogonality thesis” that we defend as part of our arguments, we make it sound like we have a separate burden of proof to meet, whereas in fact it’s the assertion that superoptimization tells us something about the goal system that needs defending.
So: evolution tends to produce large-scale cooperative systems. Kropotkin, Nowak, Wilson, and many others have argued this. Cooperative systems are favoured by game theory—which is why they currently dominate the biosphere. “Arbitrary” goal systems tend not to evolve.
I’m glad to see that you implicitly accept my point, which is that in the absence of specific arguments such as the one you advance here we have no reason to believe any particular non-orthogonality thesis.
So: evolution tends to produce large-scale cooperative systems. Kropotkin, Nowak, Wilson, and many others have argued this. Cooperative systems are favoured by game theory—which is why they currently dominate the biosphere. “Arbitrary” goal systems tend not to evolve.
I’m glad to see that you implicitly accept my point, which is that in the absence of specific arguments such as the one you advance here we have no reason to believe any particular non-orthogonality thesis.