I’m not sure whether this is important to the main thrust of the post, but I disagree with most of this paragraph:
Again, they’re an expert in the field—and this is the sort of claim that would be fairly easy to check even if you’re not an expert yourself, just by Googling around and skimming recent papers. It’s also not the sort of claim where there’s any obvious incentive for deception. It’s hard to think of a plausible scenario in which this person writes this sentence, and yet the sentence is false or even controversial.
In my experience, it’s quite hard to check what “the gold standard” of something is, particularly in cutting-edge research fields. There are lots of different metrics on which methods compete, and it’s hard to know their importance as an outsider.
And the obvious incentive for deception is that the physics prof works on NPsM, and so is talking it up (or has developed a method that beats NPsM on some benchmark, and so is talking it up to impress people with their new method …)
Yeah, you’re totally right—actually, I was reading over that section just now and thinking of adding a caveat about just this point, but I worried it would be distracting.
But this is just a flaw in my example, not in the point it’s meant to illustrate (?), which is hopefully clear enough despite the flaw.
I’m not sure whether this is important to the main thrust of the post, but I disagree with most of this paragraph:
In my experience, it’s quite hard to check what “the gold standard” of something is, particularly in cutting-edge research fields. There are lots of different metrics on which methods compete, and it’s hard to know their importance as an outsider.
And the obvious incentive for deception is that the physics prof works on NPsM, and so is talking it up (or has developed a method that beats NPsM on some benchmark, and so is talking it up to impress people with their new method …)
Yeah, you’re totally right—actually, I was reading over that section just now and thinking of adding a caveat about just this point, but I worried it would be distracting.
But this is just a flaw in my example, not in the point it’s meant to illustrate (?), which is hopefully clear enough despite the flaw.