“Is Science Broken?” is underspecified
http://fivethirtyeight.com/features/science-isnt-broken/
This is an interesting article—it’s got an overview of what’s currently seen as the problems with replicability and fraud, and some material I haven’t seen before about handing the same question to a bunch of scientists, and looking at how they come up with their divergent answers.
However, while I think it’s fair to say that science is really hard, the article gets into claiming that scientists aren’t especially awful people (probably true), but doesnn’t address the hard question of “Given that there’s a lot of inaccurate science, how much should we trust specific scientific claims?”
I think this article suffers from aggregating all science into one big bin. In reality, different disciplines have a radically different level of problems with replicability and fraud. Classical hard sciences like physics and chemistry don’t have much of a problem. Very soft sciences like psychology or anthropology have a huge problem.
You’re right, though I’m not sure what the best way to phrase it better is.
My question still stands, since the parts of science which are most fucked seems to be the parts that have the most immediate impact on people’s choices.
Sure, but the problem here is that the causality probably goes in the opposite direction. That is, the more a scientific endeavor will affect people’s choices, the more pressure there is to corrupt that scientific endeavor.
What do you mean by immediate impact on choices? Very few people make choices based on what the psychological theory of the day says they must do.
The most impactful branches are probably medicine and economics. They are medium-fucked, I think, because at the psych/anthro levels of dysfunction your patients just die or your economy implodes and people tend to dislike such things :-/
Now that I’m thinking about it, psychological papers probably have more effect in the LW-sphere than in the world generally. Are you counting nutrition as part of medicine?
Parenting might be even worse, with plenty of contradictions between self-proclaimed experts, one claiming something is very important to do, the other claiming you must never do it under any circumstances.
Nutrition is particularly bad and sits at the edge of medicine, I’d say.
The infographic on that page is amazing.
Given the OP and this, I thought that you might like this.
That’s not too closely related to the OP in one sense, but I’ve been collecting what I might call ‘stories of broken science,’ and thought you might be doing the same thing for different reasons.
Has anyone heard about the book “The egg-laying dog” from Beck-Bornholdt? I don’t know about an English translation, I freely translated the title from German. It is a book about fallacies in statistics, research, especially in medicine, written in a style to be comprehensible by the layman.
It discusses at great length the problems plaguing modern research (well, the research of the 1990′s when the book was written, but I doubt that very much has changed). For example, the required statistical significance for a publication is much more relaxed than it was a long time ago. Often a p-value of 5% is enough for a publication, so even with perfectly unbiased researchers, without p-fishing or other unethical tricks, there is a huge number of accepted publications around which are utterly rubbish. This is all made much worse by the fact that everyone wants new results, so few researchers can get funding by repeating and verifying already published results (unless the publication in question is on every headline), and also few researchers are inclined (or supported by the system) to publish negative results.
I dream of a future where published papers will no more need to be about a 0-1 effect (wether an hypothesis is true or not), but will be like that amazing infographic; a future where a scientist’s job will be no more to interpret data, but to construct interactive models and allow anyone to explore them and draw the conclusions they like.