Survey and other data indicate that in these fields most people were doing p-hacking/QRPs (running tests selected ex post, optional stopping, reporting and publication bias, etc), but a substantial minority weren’t, with individual, subfield, and field variation. Some people produced ~100% bogus work while others were ~0%. So it was possible to have a career without the bad practices Yarkoni criticizes, aggregating across many practices to look at overall reproducibility of research.
And he is now talking about people who have been informed about the severe effects of the QRPs (that they result in largely bogus research at large cost to science compared to reproducible alternatives that many of their colleagues are now using and working to reward) but choose to continue the bad practices. That group is also disproportionately tenured, so it’s not a question of not getting a place in academia now, but of giving up on false claims they built their reputation around and reduced grants and speaking fees.
I think the core issue is that even though the QRPs that lead to mostly bogus research in fields such as social psych and neuroimaging often started off without intentional bad conduct, their bad effects have now become public knowledge, and Yarkoni is right to call out those people on continuing them and defending continuing them.
Some people produced ~100% bogus work while others were ~0%. So it was possible to have a career without the bad practices Yarkoni criticizes, aggregating across many practices to look at overall reproducibility of research.
I’m curious how many were able to hit 0%? Based on my 10x estimate below I’d estimate 9%, but that was definitely a number I pulled out of nowhere.
That group is also disproportionately tenured, so it’s not a question of not getting a place in academia now, but of giving up on false claims they built their reputation around and reduced grants and speaking fees.
I personally feel the most pressure to publish because the undergrads I work with need a paper to get into grad school. I wonder if it’s similar for tenured professors with their grad students.
Also, the article seems to be condemning academics who are not tenured, e.g.
“I would publish in open access journals,” your friendly neighborhood scientist will say. “But those have a lower impact factor, and I’m up for tenure in three years.”
I think the core issue is that even though the QRPs that lead to mostly bogus research in fields such as social psych and neuroimaging often started off without intentional bad conduct, their bad effects have now become public knowledge, and Yarkoni is right to call out those people on continuing them and defending continuing them.
Thought experiment (that I acknowledge is not reality): Suppose that it were actually the case that in order to stay in academia you had to engage in QRPs. Do you still think it is right to call out / punish such people? It seems like this ends up with you always punishing everyone in academia, with no gains to actually published research, or you abolish academia outright.
Survey and other data indicate that in these fields most people were doing p-hacking/QRPs (running tests selected ex post, optional stopping, reporting and publication bias, etc), but a substantial minority weren’t, with individual, subfield, and field variation. Some people produced ~100% bogus work while others were ~0%. So it was possible to have a career without the bad practices Yarkoni criticizes, aggregating across many practices to look at overall reproducibility of research.
And he is now talking about people who have been informed about the severe effects of the QRPs (that they result in largely bogus research at large cost to science compared to reproducible alternatives that many of their colleagues are now using and working to reward) but choose to continue the bad practices. That group is also disproportionately tenured, so it’s not a question of not getting a place in academia now, but of giving up on false claims they built their reputation around and reduced grants and speaking fees.
I think the core issue is that even though the QRPs that lead to mostly bogus research in fields such as social psych and neuroimaging often started off without intentional bad conduct, their bad effects have now become public knowledge, and Yarkoni is right to call out those people on continuing them and defending continuing them.
I’m curious how many were able to hit 0%? Based on my 10x estimate below I’d estimate 9%, but that was definitely a number I pulled out of nowhere.
I personally feel the most pressure to publish because the undergrads I work with need a paper to get into grad school. I wonder if it’s similar for tenured professors with their grad students.
Also, the article seems to be condemning academics who are not tenured, e.g.
Thought experiment (that I acknowledge is not reality): Suppose that it were actually the case that in order to stay in academia you had to engage in QRPs. Do you still think it is right to call out / punish such people? It seems like this ends up with you always punishing everyone in academia, with no gains to actually published research, or you abolish academia outright.