It’s a good point, and I think enough to call the paper’s findings seriously into question, but I don’t think fixing it would be enough to salvage the methodology. Ideological bias tends to be transparent from the inside: I’d expect any academic with a strong commitment to academic neutrality to punish perceived ideological bias in proportion to its magnitude, but I’d also expect the same academics to perceive viewpoints leaning toward their own ideology as less biased than the alternatives. Probably much less.
You’d need to do something a lot more clever to filter that out: maybe something like asking academics about the perceived rates of each type of ideological bias in papers and grants they evaluate, and normalizing based on that.
It’s a good point, and I think enough to call the paper’s findings seriously into question, but I don’t think fixing it would be enough to salvage the methodology. Ideological bias tends to be transparent from the inside: I’d expect any academic with a strong commitment to academic neutrality to punish perceived ideological bias in proportion to its magnitude, but I’d also expect the same academics to perceive viewpoints leaning toward their own ideology as less biased than the alternatives. Probably much less.
You’d need to do something a lot more clever to filter that out: maybe something like asking academics about the perceived rates of each type of ideological bias in papers and grants they evaluate, and normalizing based on that.