I mean, in practice most of the people making use of the work scientists have done aren’t really testing the scientists’ work for themselves (they’re kinda doing it implicitly by making use of that work, but the whole point is that they are confident it’s not going to fail).
First, I think the “implicitly” part is very important. That glowing gizmo with melted-sand innards in front of me works. By working it verifies, very directly, a whole lot of science.
And “working in practice” is what leads to confidence, not vice versa. When a sailor took the first GPS unit on a cruise, he didn’t say “Oh, science says it’s going to work, so that’s all going to be fine”. He took it as a secondary or, probably, a tertiary navigation device. Now, after years of working in practice sailors take the GPS as a primary device and most often, a second GPS as a secondary.
Note, by the way, that we want useful science and useful science leads to practical technologies that we test and use all the time.
Calibration (in the sense we’re talking about here) isn’t of much relevance to Alice when she’s doing the primary research.
Oh, good, we agree.
But Bob’s opinion … is an altogether slipperier thing; and the opinion to which he and Beth and the others converge is slipperier still.
Sure, that’s fine. Bob and Beth are not scientists and are not doing science. Allow me to quote myself: “Calibration is good for guesstimates, it’s not particularly valuable for actual research.” Bob and Bill and Beth and Bert are not doing actual research. They are trying to use published results to form some opinions, some guesstimates and, as I agree, their calibration matters for the quality of their guesstimates. But, again, that’s not science.
Bob and Beth are not scientists and are not doing science.
Bob and Beth are scientists (didn’t I make it clear enough in my gedankenexperiment that they are intended to be journo-oncologists just as much as Alice et al, it’s just that we’re considering them in a different role here?). And they are forming their opinions in the course of their professional activities. Doing science is not only about doing experiments and working out knotty theoretical problems; when two scientists discuss their work, they are doing science; when a scientist attends a conference presentation given by another, they are doing science; when a scientist sits and thinks about what might be a good problem to attack next, they are doing science.
Doing actual research is a more “central” scientific activity than those other things. But the other things are real, they are things scientists actually do, they are things scientists need to do, and I don’t see any reason to deny that doing them is part of how science (the whole collective enterprise) functions.
when a scientist sits and thinks about what might be a good problem to attack next, they are doing science.
Sure, and you’ve expanded the definition of “doing science” into uselessness. “Doodling on paper napkins is doing science!”—well, yeah, if you want it so, what next?
I’m not talking about what large variety of things scientists do in the course of their professional lives. I’m talking about the core concept of science and whether it, as MattG believes, “moves forward through something called scientific consensus”.
In particular, I would like to distinguish between “doing science” (discovering how the world works) and “applying science” (changing the world based on your beliefs about how it works).
Let’s distinguish two things. (1) The core activities of science are, for sure, things like doing carefully designed experiments and applying mathematics to make quantitative predictions based on precisely formulated theories. These activities, indeed, don’t proceed by consensus, but no one claimed otherwise; even to ask whether they do is a type error. (2) How scientific knowledge actually advances. This is not only a matter of #1; if we had nothing but #1 then science wouldn’t advance at all, because in order for science to advance each scientist’s work needs to be based in, or at least aware of, the work of their predecessors. And #2, as it happens, does involve something like consensus, and it’s reasonable to wonder whether being more explicitly and carefully rational about #2 would help science to advance more effectively. And that is what (AIUI) MattG is proposing.
I do believe MattG claimed otherwise. At least that was the most straightforward reading of what he said.
in order for science to advance each scientist’s work needs to be based in, or at least aware of, the work of their predecessors.
That is true, the scientists do trust what’s considered “solved”, but that trust is conditional. One little ugly fact can blow up a lot of consensus sky-high.
I think one of the core issues here is resistance to cargo cult science. Consensus is dangerous because it is enables cargo cults, but the sceptical “show me” attitude is invaluable here.
more explicitly and carefully rational about #2 would help science to advance more effectively
What do you mean by “carefully rational”? How is that better than the baseline “show me”?
I think you can only reach that conclusion by applying your preferred definition of “science” to MattG’s statement about science. That’s a mistake unless you know he’s not using a substantially different definition.
that trust is conditional
Yes, of course. (Did anyone suggest it’s not?)
For the avoidance of doubt, I am not for a minute suggesting blind or unquestioning trust of scientific consensus; at least, not for scientists. (It is possible that below some threshold of scientific competence blind trust is in fact the best available strategy.)
What do you mean by “carefully rational”?
I mean what happens if the Bobs in my thought experiment, rather than arriving at their opinions informally and qualitatively, think explicitly about what they’ve heard and read and about how much evidence each thing they’ve heard or read provides, and determine their own opinions by deliberate reflection on that (not necessarily by actual calculation, but with that always available in cases of doubt).
This might well not be an improvement (e.g., because System 1 has hardware support that System 2 doesn’t) but it’s not obvious that it isn’t.
How is that better than the baseline “show me”?
“Carefully rational” isn’t a proposed replacement for “show me”, it’s a proposed replacement for things like “I’ve read about this in a few papers so I’ll assume it’s true” (which probably doesn’t get said explicitly very often, of course).
“Show me” is always there (usually in the background) as an option. Most scientists, most of the time, don’t go banging on other scientists’ lab doors demanding further evidence for what’s in their papers. Most scientists, most of the time, don’t attempt to replicate other scientists’ results before (at least provisionally) accepting them.
(One reason is that replication and door-banging take effort. This is also an argument against the more explicit “carefully rational” approach I think MattG is advocating.)
In the absence of any more information than that you “fail to discern [my] point”, I don’t know what I can usefully say to help. In ascending order of cynicism:
If nothing in my previous comment conveyed any meaning to you at all, then it seems like we have a big impedance mismatch and fixing the problem (whatever it is) seems likely to be more trouble than it’s worth.
If you just can’t be bothered to say with any specificity what the problem is, then I suppose that indicates that you think your time is much more valuable than mine, a position I cordially decline to share.
If you’re just being generally dismissive because that’s rhetorically more effective than engagement, I’m not interested in discussion on those terms.
(I’m sorry if you find my style uncongenially cautious. This deep into a tangential discussion like this one, I’d expect much of what’s said to be clarifications and edge-nibbling, and in particular it seems peculiar to (1) ask questions of the form “what do you mean by X and why is it better than Y?” and then (2) complain that you’re getting clarification and edge-nibbling in response.)
I mean it literally. I can’t see a coherent position behind your criticisms, there is no overarching framework which backs them up. I don’t understand what is the core of your disagreement amongst all the clarifications.
I don’t know that my disagreement has a single core. It looks to me as if you are making a number of separate (but related) mistakes.
I think you are defining “science” narrowly, to include only actual experimentation and analysis, then interpreting MattG’s comments as if he is using a similarly narrow definition of “science” (which he has said he isn’t). This is a mistake because of course what someone says is liable to come out wrong when you give its words different meanings from the one they had in mind.
I think you are defining “science” narrowly, to include only actual experimentation and analysis, in a discussion of whether knowledge would advance more effectively if scientists explicitly represented their beliefs about scientific theories in probabilistic terms, did something like Bayes-rule updates on learning new things, and attempted to monitor the reliability of other scientists using notions like “calibration”. This is a mistake because the question at issue is not about actual experimentation and analysis.
I think you are writing as if the only important things scientists do in their capacity as scientists are actual experimentation and analysis. This is a mistake because science is in fact a collective endeavour whose success in advancing knowledge depends on scientists’ communication with other scientists, and evaluation of their work.
Perhaps this is the core: I do not think that, in this discussion, it is helpful for you to insist on a narrow definition of what counts as “science”. I think your suggestion upthread that the only alternative is to say that absolutely anything is “science” is ridiculous. I don’t have any objection to a narrow definition of “science” as such; there are surely contexts in which it’s better than a broad one; but I don’t think this discussion is such a context.
Perhaps this is the core: I do not think that, in this discussion, it is helpful for you to insist on a narrow definition of what counts as “science”
Interesting. I don’t perceive this subthread as mostly about definitions, I think of it as being about the balance between two approaches to claims about reality: the hard one (“show me”, see also this) and the soft one (“let’s construct as subjective probability assessment on the basis of opinions of experts”).
Notably, this subthread started with MattG saying “Science moves forward through something called scientific consensus” and me going “Whaaaa...?”
I also don’t think the discussion is about definitions, but I think it’s being made needlessly more difficult by differences in definitions.
It is (I think) a simple matter of empirical fact that most of the time scientists get information from one another without saying “show me!”. That doesn’t mean that “show me!” isn’t always there in the background—it is—but only that the actual practice of science-broadly-conceived (by which I don’t mean “science-narrowly-conceived plus fake science”, I mean “science-narrowly-conceived plus the other things scientists do without which science as a whole would make much less progress”) does in fact involve subjective probability assessments on the basis of experts’ opinions.
It is (I think) a simple matter of empirical fact that most of the time scientists get information from one another without saying “show me!”.
Actually, I will disagree with that. There is a reason published papers consist mostly of detailed descriptions of what was done and what happened. If what you are saying were true, executive summaries would suffice: We have discovered that frobnicating frotzed blivets leads to emission of magic smoke. The End.
Certainly, large parts of scientific knowledge have passed into the “just accept it’s true” realm, but any new claims are required to be supported by fairly large amounts of “show me”.
If what you are saying were true, executive summaries would suffice
I don’t see why. The details are there for the following reasons, none of which appears to me to be invalidated by anything I’ve said. (1) They are interesting for their own sake (to those immersed in the field, at least). (2) They clarify what useful opportunities there may be for followup work (“Hmm, all their blivets were frotzed with titanium chloride. What happens if we use uranium nitride instead?”). (3) They provide a way to do “show me!”-like checks for those relatively few who want to without needing to interrogate the authors (replicating the analysis is easier than replicating the experiment). (4) They provide, in principle, the information needed for a more thorough “show me!” check (outright replication) for those even fewer who want to do that.
If you’ve got the impression that I don’t agree that independent experimental test is the nearest thing we have to an ultimate arbiter of scientific truth, then I’ve been unclear or you’ve been obtuse or both; I do agree with that. Most of the time, though, scientists don’t go all the way to the ultimate arbiter.
But this thread has drifted far from reality. It began with Lumifer’s comment about estimates of historical poverty:
The charts posted claim to reflect the entire world and they go back to early XIX century. Whole-world data at that point is nothing but a collection of guesstimates.
To which MattG replied:
My understanding is you basically get a bunch of economists in the room to break down the problem into relevant parts, then get a bunch of historians in the room, calibrate them, get them to give credible intervals for the relevant data, and plug it all in to the model.
Lumifer:
Is this how you think it works or is this how you think it should work?
MattG:
It’s how I think it works.
And the conversation drifted into the stratosphere with no further discussion of where those numbers actually came from.
First, I think the “implicitly” part is very important. That glowing gizmo with melted-sand innards in front of me works. By working it verifies, very directly, a whole lot of science.
And “working in practice” is what leads to confidence, not vice versa. When a sailor took the first GPS unit on a cruise, he didn’t say “Oh, science says it’s going to work, so that’s all going to be fine”. He took it as a secondary or, probably, a tertiary navigation device. Now, after years of working in practice sailors take the GPS as a primary device and most often, a second GPS as a secondary.
Note, by the way, that we want useful science and useful science leads to practical technologies that we test and use all the time.
Oh, good, we agree.
Sure, that’s fine. Bob and Beth are not scientists and are not doing science. Allow me to quote myself: “Calibration is good for guesstimates, it’s not particularly valuable for actual research.” Bob and Bill and Beth and Bert are not doing actual research. They are trying to use published results to form some opinions, some guesstimates and, as I agree, their calibration matters for the quality of their guesstimates. But, again, that’s not science.
Bob and Beth are scientists (didn’t I make it clear enough in my gedankenexperiment that they are intended to be journo-oncologists just as much as Alice et al, it’s just that we’re considering them in a different role here?). And they are forming their opinions in the course of their professional activities. Doing science is not only about doing experiments and working out knotty theoretical problems; when two scientists discuss their work, they are doing science; when a scientist attends a conference presentation given by another, they are doing science; when a scientist sits and thinks about what might be a good problem to attack next, they are doing science.
Doing actual research is a more “central” scientific activity than those other things. But the other things are real, they are things scientists actually do, they are things scientists need to do, and I don’t see any reason to deny that doing them is part of how science (the whole collective enterprise) functions.
Sure, and you’ve expanded the definition of “doing science” into uselessness. “Doodling on paper napkins is doing science!”—well, yeah, if you want it so, what next?
I’m not talking about what large variety of things scientists do in the course of their professional lives. I’m talking about the core concept of science and whether it, as MattG believes, “moves forward through something called scientific consensus”.
In particular, I would like to distinguish between “doing science” (discovering how the world works) and “applying science” (changing the world based on your beliefs about how it works).
Let’s distinguish two things. (1) The core activities of science are, for sure, things like doing carefully designed experiments and applying mathematics to make quantitative predictions based on precisely formulated theories. These activities, indeed, don’t proceed by consensus, but no one claimed otherwise; even to ask whether they do is a type error. (2) How scientific knowledge actually advances. This is not only a matter of #1; if we had nothing but #1 then science wouldn’t advance at all, because in order for science to advance each scientist’s work needs to be based in, or at least aware of, the work of their predecessors. And #2, as it happens, does involve something like consensus, and it’s reasonable to wonder whether being more explicitly and carefully rational about #2 would help science to advance more effectively. And that is what (AIUI) MattG is proposing.
I do believe MattG claimed otherwise. At least that was the most straightforward reading of what he said.
That is true, the scientists do trust what’s considered “solved”, but that trust is conditional. One little ugly fact can blow up a lot of consensus sky-high.
I think one of the core issues here is resistance to cargo cult science. Consensus is dangerous because it is enables cargo cults, but the sceptical “show me” attitude is invaluable here.
What do you mean by “carefully rational”? How is that better than the baseline “show me”?
I think you can only reach that conclusion by applying your preferred definition of “science” to MattG’s statement about science. That’s a mistake unless you know he’s not using a substantially different definition.
Yes, of course. (Did anyone suggest it’s not?)
For the avoidance of doubt, I am not for a minute suggesting blind or unquestioning trust of scientific consensus; at least, not for scientists. (It is possible that below some threshold of scientific competence blind trust is in fact the best available strategy.)
I mean what happens if the Bobs in my thought experiment, rather than arriving at their opinions informally and qualitatively, think explicitly about what they’ve heard and read and about how much evidence each thing they’ve heard or read provides, and determine their own opinions by deliberate reflection on that (not necessarily by actual calculation, but with that always available in cases of doubt).
This might well not be an improvement (e.g., because System 1 has hardware support that System 2 doesn’t) but it’s not obvious that it isn’t.
“Carefully rational” isn’t a proposed replacement for “show me”, it’s a proposed replacement for things like “I’ve read about this in a few papers so I’ll assume it’s true” (which probably doesn’t get said explicitly very often, of course).
“Show me” is always there (usually in the background) as an option. Most scientists, most of the time, don’t go banging on other scientists’ lab doors demanding further evidence for what’s in their papers. Most scientists, most of the time, don’t attempt to replicate other scientists’ results before (at least provisionally) accepting them.
(One reason is that replication and door-banging take effort. This is also an argument against the more explicit “carefully rational” approach I think MattG is advocating.)
I fail to discern your point. There is a lot of clarifications, adjustments, and edge-nibbling, but what is it that you want to say?
In the absence of any more information than that you “fail to discern [my] point”, I don’t know what I can usefully say to help. In ascending order of cynicism:
If nothing in my previous comment conveyed any meaning to you at all, then it seems like we have a big impedance mismatch and fixing the problem (whatever it is) seems likely to be more trouble than it’s worth.
If you just can’t be bothered to say with any specificity what the problem is, then I suppose that indicates that you think your time is much more valuable than mine, a position I cordially decline to share.
If you’re just being generally dismissive because that’s rhetorically more effective than engagement, I’m not interested in discussion on those terms.
(I’m sorry if you find my style uncongenially cautious. This deep into a tangential discussion like this one, I’d expect much of what’s said to be clarifications and edge-nibbling, and in particular it seems peculiar to (1) ask questions of the form “what do you mean by X and why is it better than Y?” and then (2) complain that you’re getting clarification and edge-nibbling in response.)
I mean it literally. I can’t see a coherent position behind your criticisms, there is no overarching framework which backs them up. I don’t understand what is the core of your disagreement amongst all the clarifications.
I don’t know that my disagreement has a single core. It looks to me as if you are making a number of separate (but related) mistakes.
I think you are defining “science” narrowly, to include only actual experimentation and analysis, then interpreting MattG’s comments as if he is using a similarly narrow definition of “science” (which he has said he isn’t). This is a mistake because of course what someone says is liable to come out wrong when you give its words different meanings from the one they had in mind.
I think you are defining “science” narrowly, to include only actual experimentation and analysis, in a discussion of whether knowledge would advance more effectively if scientists explicitly represented their beliefs about scientific theories in probabilistic terms, did something like Bayes-rule updates on learning new things, and attempted to monitor the reliability of other scientists using notions like “calibration”. This is a mistake because the question at issue is not about actual experimentation and analysis.
I think you are writing as if the only important things scientists do in their capacity as scientists are actual experimentation and analysis. This is a mistake because science is in fact a collective endeavour whose success in advancing knowledge depends on scientists’ communication with other scientists, and evaluation of their work.
Perhaps this is the core: I do not think that, in this discussion, it is helpful for you to insist on a narrow definition of what counts as “science”. I think your suggestion upthread that the only alternative is to say that absolutely anything is “science” is ridiculous. I don’t have any objection to a narrow definition of “science” as such; there are surely contexts in which it’s better than a broad one; but I don’t think this discussion is such a context.
Interesting. I don’t perceive this subthread as mostly about definitions, I think of it as being about the balance between two approaches to claims about reality: the hard one (“show me”, see also this) and the soft one (“let’s construct as subjective probability assessment on the basis of opinions of experts”).
Notably, this subthread started with MattG saying “Science moves forward through something called scientific consensus” and me going “Whaaaa...?”
I also don’t think the discussion is about definitions, but I think it’s being made needlessly more difficult by differences in definitions.
It is (I think) a simple matter of empirical fact that most of the time scientists get information from one another without saying “show me!”. That doesn’t mean that “show me!” isn’t always there in the background—it is—but only that the actual practice of science-broadly-conceived (by which I don’t mean “science-narrowly-conceived plus fake science”, I mean “science-narrowly-conceived plus the other things scientists do without which science as a whole would make much less progress”) does in fact involve subjective probability assessments on the basis of experts’ opinions.
Actually, I will disagree with that. There is a reason published papers consist mostly of detailed descriptions of what was done and what happened. If what you are saying were true, executive summaries would suffice: We have discovered that frobnicating frotzed blivets leads to emission of magic smoke. The End.
Certainly, large parts of scientific knowledge have passed into the “just accept it’s true” realm, but any new claims are required to be supported by fairly large amounts of “show me”.
I don’t see why. The details are there for the following reasons, none of which appears to me to be invalidated by anything I’ve said. (1) They are interesting for their own sake (to those immersed in the field, at least). (2) They clarify what useful opportunities there may be for followup work (“Hmm, all their blivets were frotzed with titanium chloride. What happens if we use uranium nitride instead?”). (3) They provide a way to do “show me!”-like checks for those relatively few who want to without needing to interrogate the authors (replicating the analysis is easier than replicating the experiment). (4) They provide, in principle, the information needed for a more thorough “show me!” check (outright replication) for those even fewer who want to do that.
If you’ve got the impression that I don’t agree that independent experimental test is the nearest thing we have to an ultimate arbiter of scientific truth, then I’ve been unclear or you’ve been obtuse or both; I do agree with that. Most of the time, though, scientists don’t go all the way to the ultimate arbiter.
Consensus is the result, not the means.
But this thread has drifted far from reality. It began with Lumifer’s comment about estimates of historical poverty:
To which MattG replied:
Lumifer:
MattG:
And the conversation drifted into the stratosphere with no further discussion of where those numbers actually came from.
Consensus is the result, not the means.