My heuristics say that this study is likely bunk. It has the unholy trinity of being counter-intuitive, politically useful, and sounding cool.
I’m going to pre-register my predictions here before I do an analysis.
Predictions:
50% chance there is no attempt at correcting for multiplicity (I’ll set this as unresolved if they only do this for a data table but not their multiple hypotheses, which is depressingly common in genomics). 90% chance they didn’t do it well. 20% chance they’re intentionally testing large numbers (10+) of hypotheses with no attempt at correction.
80% chance this study won’t replicate. 10% I will think the main conclusions of this paper are true 5 years from now.
40% chance of a significant hole in the authors’ logic (not taking into account an alternative hypothesis that better explains the data).
I should make it clear that these practices are very much not common in any field and greatly exceeded my expectations. I applaud the authors for making the extra effort and strongly encourage other researchers to follow in their footsteps.
My yell-at-people-on-the-internet-for-doing-statistics-wrong senses are still tingling, though, for reasons I don’t understand. It’s probably nothing, but maybe it’s foreshadowing.
These may be reasonable heuristics, given how much research doesn’t replicate. But why do you consider this finding “politically useful”? The study says that this behavior happens regardless of political affiliation, so it’s not like those studies that say “<my political opponents> are <dumb / naive / racist>” and which then serve as ammunition against the other side.
I meant more like it slides neatly into someone’s political theory, and “increased political polarization” is a pretty common topic nowadays. I should probably come up with a better description for this.
Does it slide neatly into the political theory of increased political polarization, though? I feel like I could’ve told stories consistent with that theory for all conceivable study outcomes:
“As expected, people mostly choose to support the other sider rather than withholding money from their own side, probably because they think the latter is more effective at using the money.”
“As expected, given such an unpalatable choice, people essentially flip a coin.”
My heuristics say that this study is likely bunk. It has the unholy trinity of being counter-intuitive, politically useful, and sounding cool.
I’m going to pre-register my predictions here before I do an analysis.
Predictions:
50% chance there is no attempt at correcting for multiplicity (I’ll set this as unresolved if they only do this for a data table but not their multiple hypotheses, which is depressingly common in genomics). 90% chance they didn’t do it well. 20% chance they’re intentionally testing large numbers (10+) of hypotheses with no attempt at correction.
80% chance this study won’t replicate. 10% I will think the main conclusions of this paper are true 5 years from now.
40% chance of a significant hole in the authors’ logic (not taking into account an alternative hypothesis that better explains the data).
I was wrong. This study actually looks solid, with pre-registration and good sample-sizes.
Also, they made all the code and datasets available!
https://osf.io/gzxke/files/osfstorage
I should make it clear that these practices are very much not common in any field and greatly exceeded my expectations. I applaud the authors for making the extra effort and strongly encourage other researchers to follow in their footsteps.
My yell-at-people-on-the-internet-for-doing-statistics-wrong senses are still tingling, though, for reasons I don’t understand. It’s probably nothing, but maybe it’s foreshadowing.
A full analysis will follow. Eventually.
Also their respondants are not just undergraduates—the first study at least was a representative sample of about 1,000 US and UK residents.
Upvoted for preregistration.
These may be reasonable heuristics, given how much research doesn’t replicate. But why do you consider this finding “politically useful”? The study says that this behavior happens regardless of political affiliation, so it’s not like those studies that say “<my political opponents> are <dumb / naive / racist>” and which then serve as ammunition against the other side.
Also, kudos to pre-registering your predictions!
I meant more like it slides neatly into someone’s political theory, and “increased political polarization” is a pretty common topic nowadays. I should probably come up with a better description for this.
Does it slide neatly into the political theory of increased political polarization, though? I feel like I could’ve told stories consistent with that theory for all conceivable study outcomes:
“As expected, people mostly choose to support the other sider rather than withholding money from their own side, probably because they think the latter is more effective at using the money.”
“As expected, given such an unpalatable choice, people essentially flip a coin.”
“As expected, <actual study result>.”