There’s another toy example that might help too. Suppose Statistician 2 is willing to flip the coin 3 times, but gets heads on the first flip and stops there. Surely you can’t accept this data, or else you’re practically guaranteed to let Statistician 2 manipulate you, right?
Well, P(H | coin A) = 2⁄3, and P(H| | coin B) = 1⁄3, so clearly “first flip heads” is an event that happens twice as often when it’s coin A. What kind of scientist would you be if you couldn’t derive evidence from an event that happens twice as often under some conditions?
The weird thing is that even though you can see the event “first flip heads,” you’ll never see the event “first flip tails.” How come these individual data points are still “good bets,” even though you’ll never see the event of first flip tails? It seems like Statistician 2 has a sure-fire system for “beating the house” and convincing you no matter what.
Why am I suddenly making gambling analogies? Because Statistician 2 is trying to use a Martingale betting system. And at the end of the day, the house always wins—Statistician 2 has a large chance to submit a “biased towards heads” sample, but only at the cost of having their other samples be even more biased towards tails. On average, they are still accurate, just like how on average, you can’t win money with a Martingale betting strategy.
In this analogy, publication bias is like running away without paying your gambling debts.
There’s another toy example that might help too. Suppose Statistician 2 is willing to flip the coin 3 times, but gets heads on the first flip and stops there. Surely you can’t accept this data, or else you’re practically guaranteed to let Statistician 2 manipulate you, right?
Well, P(H | coin A) = 2⁄3, and P(H| | coin B) = 1⁄3, so clearly “first flip heads” is an event that happens twice as often when it’s coin A. What kind of scientist would you be if you couldn’t derive evidence from an event that happens twice as often under some conditions?
The weird thing is that even though you can see the event “first flip heads,” you’ll never see the event “first flip tails.” How come these individual data points are still “good bets,” even though you’ll never see the event of first flip tails? It seems like Statistician 2 has a sure-fire system for “beating the house” and convincing you no matter what.
Why am I suddenly making gambling analogies? Because Statistician 2 is trying to use a Martingale betting system. And at the end of the day, the house always wins—Statistician 2 has a large chance to submit a “biased towards heads” sample, but only at the cost of having their other samples be even more biased towards tails. On average, they are still accurate, just like how on average, you can’t win money with a Martingale betting strategy.
In this analogy, publication bias is like running away without paying your gambling debts.