I agree with your reading of my points 1,2,4 and 5 but think we are not seeing eye to eye on points 3 and 6. It also saddens me that you condensed the paragraph on how I would like to view the how-much-should-we-trust-science landscape to its least important sentence (point 4), at least from my point of view.
As for point 3, I do not want to make a general point about the reliability of science at all. I want to discuss what tools we have to evaluate the accuracy of any particular paper or claim, so that we can have more appropriate confidence across the board. I think this is the most important discussion regardless of whether it increases or decreases general confidence. In my opinion, attempting to give a 0th-order summary by discussing the average change in confidence from this approach is doing more harm than good. The sentence “You just want to make the general point that you can’t trust everything you read, with the background understanding that sometimes this is more important, and sometimes less.” is exactly backwards from what I am trying to say.
For point 6, I think it might be very relevant to point out that I’m European, and the anti-vax and global warming denialism really is not that popular around where I live. They are more considered stereotypes of being untrustworthy than properly held beliefs, thankfully. But ignoring that, I think that most of the people influencing social policy and making important decisions are leaning heavily on science, and unfortunately particularly on the types of science I have the lowest confidence in. I was hoping to avoid going into great detail on this, but as short summary I think it is reasonable to be less concerned with the accuracy of papers that have low (societal) impact and more concerned with papers that have high impact. If you randomly sample a published paper on Google Scholar or whatever I’ll happily agree that you are likely to find an accurate piece of research. But this is not an accurate representation of how people encounter scientific studies in reality. I see people break the fourth virtue all the way from coffeehouse discussions to national policy debates, which is so effective precisely because the link between data and conclusion is murky. So a lot of policy proposals can be backed by some amount of references. Over the past few years my attempts to be more even have led me to strongly decrease my confidence in a large number of scientific studies, if only to account for the selection effect that these, and not others, were brought to my attention.
Also I think psychology and nutrition are doing a lot better than they were a decade or two ago, which I consider a great sign. But that’s more of an aside than a real point.
This makes a lot of sense, actually. You’re focused on mechanisms that a good thinker could use to determine whether or not a particular scientific finding is true or not. I’m worried about the ways that the conversation around skepticism can and does go astray.
Perhaps I read some of the quotes from the papers uncharitably. Silberzahn asks “What if scientific results are highly contingent on subjective decisions at the analysis stage?” I interpreted this question, in conjunction with the paper’s conclusion, as pointing to a line of thinking that goes something like this:
What if scientific results are highly contingent on subjective decisions at the analysis stage?
Some scientific results are highly contingent on subjective decisions at the analysis stage.
What if ALL scientific results are highly contingent on subjective decisions at the analysis stage across the board???!!!
But a more charitable version for the third step is:
3. This method helped us uncover one such case, and might help us uncover more. Also, it’s a reminder to avoid overconfidence in published research, especially in politically charged and important issues where good evidence is hard to come by.
I spent the last ten years teaching children, and so my default mode is one of “educating the young and naive to be a little more sophisticated.” Part of my role was to sequence and present ideas with care in order to increase the chance that an impressionable and naive young mind absorbed the healthy version of an idea, rather than a damaging misinterpretation. Maybe that informs the way I perceive this debate.
I agree with your reading of my points 1,2,4 and 5 but think we are not seeing eye to eye on points 3 and 6. It also saddens me that you condensed the paragraph on how I would like to view the how-much-should-we-trust-science landscape to its least important sentence (point 4), at least from my point of view.
As for point 3, I do not want to make a general point about the reliability of science at all. I want to discuss what tools we have to evaluate the accuracy of any particular paper or claim, so that we can have more appropriate confidence across the board. I think this is the most important discussion regardless of whether it increases or decreases general confidence. In my opinion, attempting to give a 0th-order summary by discussing the average change in confidence from this approach is doing more harm than good. The sentence “You just want to make the general point that you can’t trust everything you read, with the background understanding that sometimes this is more important, and sometimes less.” is exactly backwards from what I am trying to say.
For point 6, I think it might be very relevant to point out that I’m European, and the anti-vax and global warming denialism really is not that popular around where I live. They are more considered stereotypes of being untrustworthy than properly held beliefs, thankfully. But ignoring that, I think that most of the people influencing social policy and making important decisions are leaning heavily on science, and unfortunately particularly on the types of science I have the lowest confidence in. I was hoping to avoid going into great detail on this, but as short summary I think it is reasonable to be less concerned with the accuracy of papers that have low (societal) impact and more concerned with papers that have high impact. If you randomly sample a published paper on Google Scholar or whatever I’ll happily agree that you are likely to find an accurate piece of research. But this is not an accurate representation of how people encounter scientific studies in reality. I see people break the fourth virtue all the way from coffeehouse discussions to national policy debates, which is so effective precisely because the link between data and conclusion is murky. So a lot of policy proposals can be backed by some amount of references. Over the past few years my attempts to be more even have led me to strongly decrease my confidence in a large number of scientific studies, if only to account for the selection effect that these, and not others, were brought to my attention.
Also I think psychology and nutrition are doing a lot better than they were a decade or two ago, which I consider a great sign. But that’s more of an aside than a real point.
This makes a lot of sense, actually. You’re focused on mechanisms that a good thinker could use to determine whether or not a particular scientific finding is true or not. I’m worried about the ways that the conversation around skepticism can and does go astray.
Perhaps I read some of the quotes from the papers uncharitably. Silberzahn asks “What if scientific results are highly contingent on subjective decisions at the analysis stage?” I interpreted this question, in conjunction with the paper’s conclusion, as pointing to a line of thinking that goes something like this:
What if scientific results are highly contingent on subjective decisions at the analysis stage?
Some scientific results are highly contingent on subjective decisions at the analysis stage.
What if ALL scientific results are highly contingent on subjective decisions at the analysis stage across the board???!!!
But a more charitable version for the third step is:
3. This method helped us uncover one such case, and might help us uncover more. Also, it’s a reminder to avoid overconfidence in published research, especially in politically charged and important issues where good evidence is hard to come by.
I spent the last ten years teaching children, and so my default mode is one of “educating the young and naive to be a little more sophisticated.” Part of my role was to sequence and present ideas with care in order to increase the chance that an impressionable and naive young mind absorbed the healthy version of an idea, rather than a damaging misinterpretation. Maybe that informs the way I perceive this debate.
Just wanted to confirm you have accurately described my thoughts, and I feel I have a better understanding of your position as well now.