Yes, there is A LOT of garbage. This is why I am recommending using heuristics such as numbers of citations—to maximize the accuracy of the information. And, yes, peer review is not perfect but compare journals/fields that rely on peer-review to those that do not...
Furthermore, Systematic Reviews have a pretty good track record as far as I know and this is why I recommend them.
So, if there is any controversy on the issue, remain agnostic
This post is not so much about academically controversial issues but even in those cases if you don’t have any reasons not to, then siding with the majority will bring you to the truth more often than the alternative.
Don’t become one of these “reserch sez...” people who just regurgitate abstracts. You’ll wind up worried about the dangers of red meat, getting too much sunlight, doing 45 minutes of cardio every day, etc.
This is the type of thing that you see if you do a normal google search instead of a scholarly search. I have not checked but I bet that the most cited recent review articles on those issues can provide you with some pretty good information.
Yes, there is A LOT of garbage. This is why I am recommending using heuristics such as numbers of citations—to maximize the accuracy of the information. And, yes, peer review is not perfect but compare journals/fields that rely on peer-review to those that do not...
My argument really boils down to 2 things. Researchers being systematically biased (ex: red meat), and researchers having a very low probability of actually knowing the right answer but publishing something that fits some narrow set of data (ex: “advanced” simulation). To be sure, I’ve used research to make a lot of informed decisions over my lifetime, but it’s always been straightforward, pretty much unanimous, and with lots of testimonials from online groups to give it statistical mass.
This post is not so much about academically controversial issues but even in those cases if you don’t have any reasons not to, then siding with the majority will bring you to the truth more often than the alternative.
Would you adopt this heuristic in any other scenario where the “right answer” isn’t obvious? Music, books, diet, politics, etc? Even when you restrict your sampling pool to “experts only”, the results are still pretty bad. These people are self-selecting to do research. It’s not like you’re picking a random disinterested intelligent person and asking them to study the problem. No one becomes a nutritionist because they have no opinion on food.
This is the type of thing that you see if you do a normal google search instead of a scholarly search. I have not checked but I bet that the most cited recent review articles on those issues can provide you with some pretty good information.
The overwhelming trend is fear mongering coming out of epidemiological studies.
My argument really boils down to 2 things. Researchers being systematically biased (ex: red meat), and researchers having a very low probability of actually knowing the right answer but publishing something that fits some narrow set of data (ex: “advanced” simulation). To be sure, I’ve used research to make a lot of informed decisions over my lifetime, but it’s always been straightforward, pretty much unanimous, and with lots of testimonials from online groups to give it statistical mass.
I acknowledged that there are problems, nothing is perfect. But I don’t know what you want from me. To convince you that science as a whole works!? Or that information in studies is more accurate than made-up information?
All I am advocating is to look for ‘respected’ studies and look at them. If you don’t think that looking at studies ‘approved’ by the field gives you more accurate information than not doing it I can’t really do much.
Would you adopt this heuristic in any other scenario where the “right answer” isn’t obvious? Music, books, diet, politics, etc? Even when you restrict your sampling pool to “experts only”, the results are still pretty bad.
Yes, I believe in science no matter what scenario I am in. You don’t need to blindly trust it or anything, I put different weights on different claims etc. but I would still take into account information from recent, well-cited meta-analyses or whatever I can get my hands on.
These people are self-selecting to do research. It’s not like you’re picking a random disinterested intelligent person and asking them to study the problem. No one becomes a nutritionist because they have no opinion on food.
So I should worry that researchers are interested in the topic that they are researching. What douchbags, eh?
The overwhelming trend is fear mongering coming out of epidemiological studies.
Okay. Citation? And remember we are not talking about ‘most studies’ or anything. The studies that we are talking about are well cited, by known researchers if possible and systematic reviews if possible.
Or that information in studies is more accurate than made-up information?
This is exactly my point. Studies on many many subjects may not contain information more useful than coin flip, let alone an educated guess.
All I am advocating is to look for ‘respected’ studies and look at them. If you don’t think that looking at studies ‘approved’ by the field gives you more accurate information than not doing it I can’t really do much.
This is question begging. You have to have a theory about why a “respected” study is likely to be correct. I’ve already provided theories explaining why they’re likely to be incorrect a large portion of the time.
Yes, I believe in science no matter what scenario I am in. You don’t need to blindly trust it or anything, I put different weights on different claims etc. but I would still take into account information from recent, well-cited meta-analyses or whatever I can get my hands on.
I believe in science too. But “science” and “science articles” are different things. But you didn’t answer my question, and I really want to drive home that almost no one thinks it’s a good idea to trust “majority expert opinion” in all sorts of areas.
So I should worry that researchers are interested in the topic that they are researching. What douchbags, eh?
Don’t be dense. You know exactly what I mean. A vegetarian goes to grad school and does research on nutrition. What do you think is going to happen?
Okay. Citation? And remember we are not talking about ‘most studies’ or anything. The studies that we are talking about are well cited, by known researchers if possible and systematic reviews if possible.
Citations above where you commented. You can also just punch “red meat” into google scholar and it’s all about how you can die from it.
This is exactly my point. Studies on many many subjects may not contain information more useful than coin flip, let alone an educated guess.
Wow. This is a pretty far-fetched claim..
You have to have a theory about why a “respected” study is likely to be correct. I’ve already provided theories explaining why they’re likely to be incorrect a large portion of the time.
My theory is that respected papers are done in a method more resembling the scientific method than coin flip on average and thus they get more accurate results than a coin flip. There, happy?
I believe in science too. But “science” and “science articles” are different things. But you didn’t answer my question
I did answer your question—the answer was yes.
I really want to drive home that almost no one thinks it’s a good idea to trust “majority expert opinion” in all sorts of areas.
Except, you know, the majority.
Don’t be dense. You know exactly what I mean. A vegetarian goes to grad school and does research on nutrition. What do you think is going to happen?
He is biased. So is the guy that went into grad school with anti-vegetarian views. If those guys are not changing their opinion based on the evidence then the chance is smaller (not nil though) that their papers will be highly cited.
Citations above where you commented.
You call studies that find correlations between things fear mongering? Oh my.
You can also just punch “red meat” into google scholar and it’s all about how you can die from it.
Oh my. Okay, first of all you can die of pretty much anything and pretty much anything has some dangers. Or at least that’s what does fear mongering scientists claim. The studies show you some numbers to guide you in how much danger X (in this case red meat) poses to specific individuals.
Do you have any specific reason to think that those studies are fabricated and that in fact red meat has none of the effects that they claim?
Furthermore, if I tell you that drinking a large amount of water can kill you and do a study to prove it then am I a fear mongering scientist?
My theory is that respected papers are done in a method more resembling the scientific method than coin flip on average and thus they get more accurate results than a coin flip. There, happy?
Thanks for clarifying. I disagree. See the systematic bias/complexity arguments.
I did answer your question—the answer was yes.
Do you really choose your music based on the average opinion of “experts”? Give me a break. Look, if you could randomly draft 20 people who had demonstrated independent rationality and objectivity, assign them to a problem, and take the majority opinion, I would be fine with that. But that’s not what we have at all. Anyone with an IQ above 110 can get any degree they want.
He is biased. So is the guy that went into grad school with anti-vegetarian views. If those guys are not changing their opinion based on the evidence then the chance is smaller (not nil though) that their papers will be highly cited.
Why would the best research win out? Why not the most fashionable research that confirms everyones’ worldviews? Why not the research that has the punchier abstract title? Why not the research that was fudged to show more impressive results?
You call studies that find correlations between things fear mongering? Oh my.
They could probably find a correlation between eating red meat and watching action movies, but that’s not exactly publishable.
Oh my. Okay, first of all you can die of pretty much anything and pretty much anything has some dangers. Or at least that’s what does fear mongering scientists claim. The studies show you some numbers to guide you in how much danger X (in this case red meat) poses to specific individuals.
I mean sure, if you consumed more red meat than was physiologically possible to scarf down without choking, you’d die. But that’s not unique to red meat. They’re claiming that there is a unique property of red meat which causes all these health problems, so not it doesn’t fall under the same category as “pretty much anything can kill you”.
And no, they technically don’t even show danger. All they do is show correlations. Would you also conclude that wearing XXL t-shirts makes you fat?
Do you have any specific reason to think that those studies are fabricated and that in fact red meat has none of the effects that they claim?
Confounding variables mentioned above. Lack of replication/opposite findings in controlled studies. Testimonies from thousands of people on the paleo diet who have reversed their blood chemistry. Fat doctors/nutritionists, etc.
Furthermore, if I tell you that drinking a large amount of water can kill you and do a study to prove it then am I a fear mongering scientist?
If you try to publish dozens of studies on it in the year 2012, yes you are.
“Hey guys I just did ANOTHER study showing that drinking 82 gallons of water in one sitting will kill you (p<0.05)”
That would be fear mongering, although people probably wouldn’t take it seriously.
Do you really choose your music based on the average opinion of “experts”?
Yes, except that I am the only expert on what music I like.
Anyone with an IQ above 110 can get any degree they want.
Are we talking about degrees here. I am pretty sure Ive been talking about top level articles. Or can anyone with an IQ above 110 publish one of those?
Why would the best research win out?
No winning out here. The research will be closer to the truth than a random answer because the accuracy of the theories gets compared to reality buy doing experiments for example. Or because not every single person is completely biased and blind to the results that they get.
And no, they technically don’t even show danger. All they do is show correlations. Would you also conclude that wearing XXL t-shirts makes you fat?
Hey, that’s why they are correlations. I am not stopping you from believing that being predisposed to diabetes and cancer or whatever makes you more likely to eat red meat for example.
As I said in the other thread, I am not participating in this conversation any more.
Yes, except that I am the only expert on what music I like.
Oh, so you agree there are can be good reasons to discount the “expert” establishment, no matter how much “peer review” or citations they have.
Are we talking about degrees here. I am pretty sure Ive been talking about top level articles. Or can anyone with an IQ above 110 publish one of those?
Yes. But getting a degree is normally a prereq for publishing, and everyone who gets a degree publishes something. And yes, you can publish in the “top” journal articles in grad school.
No winning out here. The research will be closer to the truth than a random answer because the accuracy of the theories gets compared to reality buy doing experiments for example. Or because not every single person is completely biased and blind to the results that they get.
Not every single person has to be biased. Just enough of them.
Hey, that’s why they are correlations. I am not stopping you from believing that being predisposed to diabetes and cancer or whatever makes you more likely to eat red meat for example.
But the researchers conclude that red meat increases your risk of heart disease simply because it is associated with heart disease. That is dishonest. If they can get away with blatantly unsubstantiated statements like that in epidemiological papers, what can’t they get away with buried in their SAS databases and algorithms?
Yes, there is A LOT of garbage. This is why I am recommending using heuristics such as numbers of citations—to maximize the accuracy of the information. And, yes, peer review is not perfect but compare journals/fields that rely on peer-review to those that do not...
Furthermore, Systematic Reviews have a pretty good track record as far as I know and this is why I recommend them.
This post is not so much about academically controversial issues but even in those cases if you don’t have any reasons not to, then siding with the majority will bring you to the truth more often than the alternative.
This is the type of thing that you see if you do a normal google search instead of a scholarly search. I have not checked but I bet that the most cited recent review articles on those issues can provide you with some pretty good information.
My argument really boils down to 2 things. Researchers being systematically biased (ex: red meat), and researchers having a very low probability of actually knowing the right answer but publishing something that fits some narrow set of data (ex: “advanced” simulation). To be sure, I’ve used research to make a lot of informed decisions over my lifetime, but it’s always been straightforward, pretty much unanimous, and with lots of testimonials from online groups to give it statistical mass.
Would you adopt this heuristic in any other scenario where the “right answer” isn’t obvious? Music, books, diet, politics, etc? Even when you restrict your sampling pool to “experts only”, the results are still pretty bad. These people are self-selecting to do research. It’s not like you’re picking a random disinterested intelligent person and asking them to study the problem. No one becomes a nutritionist because they have no opinion on food.
The overwhelming trend is fear mongering coming out of epidemiological studies.
I acknowledged that there are problems, nothing is perfect. But I don’t know what you want from me. To convince you that science as a whole works!? Or that information in studies is more accurate than made-up information?
All I am advocating is to look for ‘respected’ studies and look at them. If you don’t think that looking at studies ‘approved’ by the field gives you more accurate information than not doing it I can’t really do much.
Yes, I believe in science no matter what scenario I am in. You don’t need to blindly trust it or anything, I put different weights on different claims etc. but I would still take into account information from recent, well-cited meta-analyses or whatever I can get my hands on.
So I should worry that researchers are interested in the topic that they are researching. What douchbags, eh?
Okay. Citation? And remember we are not talking about ‘most studies’ or anything. The studies that we are talking about are well cited, by known researchers if possible and systematic reviews if possible.
This is exactly my point. Studies on many many subjects may not contain information more useful than coin flip, let alone an educated guess.
This is question begging. You have to have a theory about why a “respected” study is likely to be correct. I’ve already provided theories explaining why they’re likely to be incorrect a large portion of the time.
I believe in science too. But “science” and “science articles” are different things. But you didn’t answer my question, and I really want to drive home that almost no one thinks it’s a good idea to trust “majority expert opinion” in all sorts of areas.
Don’t be dense. You know exactly what I mean. A vegetarian goes to grad school and does research on nutrition. What do you think is going to happen?
Citations above where you commented. You can also just punch “red meat” into google scholar and it’s all about how you can die from it.
Wow. This is a pretty far-fetched claim..
My theory is that respected papers are done in a method more resembling the scientific method than coin flip on average and thus they get more accurate results than a coin flip. There, happy?
I did answer your question—the answer was yes.
Except, you know, the majority.
He is biased. So is the guy that went into grad school with anti-vegetarian views. If those guys are not changing their opinion based on the evidence then the chance is smaller (not nil though) that their papers will be highly cited.
You call studies that find correlations between things fear mongering? Oh my.
Oh my. Okay, first of all you can die of pretty much anything and pretty much anything has some dangers. Or at least that’s what does fear mongering scientists claim. The studies show you some numbers to guide you in how much danger X (in this case red meat) poses to specific individuals. Do you have any specific reason to think that those studies are fabricated and that in fact red meat has none of the effects that they claim?
Furthermore, if I tell you that drinking a large amount of water can kill you and do a study to prove it then am I a fear mongering scientist?
This is a pretty solid argument.
Thanks for clarifying. I disagree. See the systematic bias/complexity arguments.
Do you really choose your music based on the average opinion of “experts”? Give me a break. Look, if you could randomly draft 20 people who had demonstrated independent rationality and objectivity, assign them to a problem, and take the majority opinion, I would be fine with that. But that’s not what we have at all. Anyone with an IQ above 110 can get any degree they want.
Why would the best research win out? Why not the most fashionable research that confirms everyones’ worldviews? Why not the research that has the punchier abstract title? Why not the research that was fudged to show more impressive results?
They could probably find a correlation between eating red meat and watching action movies, but that’s not exactly publishable.
I mean sure, if you consumed more red meat than was physiologically possible to scarf down without choking, you’d die. But that’s not unique to red meat. They’re claiming that there is a unique property of red meat which causes all these health problems, so not it doesn’t fall under the same category as “pretty much anything can kill you”.
And no, they technically don’t even show danger. All they do is show correlations. Would you also conclude that wearing XXL t-shirts makes you fat?
Confounding variables mentioned above. Lack of replication/opposite findings in controlled studies. Testimonies from thousands of people on the paleo diet who have reversed their blood chemistry. Fat doctors/nutritionists, etc.
If you try to publish dozens of studies on it in the year 2012, yes you are.
“Hey guys I just did ANOTHER study showing that drinking 82 gallons of water in one sitting will kill you (p<0.05)”
That would be fear mongering, although people probably wouldn’t take it seriously.
Yes, except that I am the only expert on what music I like.
Are we talking about degrees here. I am pretty sure Ive been talking about top level articles. Or can anyone with an IQ above 110 publish one of those?
No winning out here. The research will be closer to the truth than a random answer because the accuracy of the theories gets compared to reality buy doing experiments for example. Or because not every single person is completely biased and blind to the results that they get.
Hey, that’s why they are correlations. I am not stopping you from believing that being predisposed to diabetes and cancer or whatever makes you more likely to eat red meat for example.
As I said in the other thread, I am not participating in this conversation any more.
Oh, so you agree there are can be good reasons to discount the “expert” establishment, no matter how much “peer review” or citations they have.
Yes. But getting a degree is normally a prereq for publishing, and everyone who gets a degree publishes something. And yes, you can publish in the “top” journal articles in grad school.
Not every single person has to be biased. Just enough of them.
But the researchers conclude that red meat increases your risk of heart disease simply because it is associated with heart disease. That is dishonest. If they can get away with blatantly unsubstantiated statements like that in epidemiological papers, what can’t they get away with buried in their SAS databases and algorithms?