Perhaps. I appreciate the prompt to think more about this.
Here’s a picture that underpins our perspective and might provide a crux:
The world is full of hard questions. Under some suitable measure over questions, we might find that almost all questions are too hard to answer. Many large scale systems exhibit chaotic behavior, most parts of the universe are unreachable to us, most nonlinear systems of differential equations have no analytic solution. Some prediction tasks are theoretically unsolvable (e.g., an analytic solution to the three-body problem), some are just practically unsolvable given current technology (e.g., knowledge of sufficiently far off domains in our forward light cone).
Are there any actual predictions past physics was trying to make which we still can’t make and don’t even care about? None that I can think of.
Here are a few prediction tasks regarding the physical world that we have not answered. What is the exact direction of movement of a single atom in the next instant? How do we achieve nuclear fusion at room temperature? What constitutes a measurement in QM? Why do we observe electric charges but not magnetic monopoles? Why did the Pioneer 10 and 11 spacecraft experience an unexplained deceleration? What is the long-run behavior of any real-world chaotic system? Is the sound of knuckles cracking caused by the formation or collapse of air bubbles? How many grains of sand are there on a given beach on third planet on the fourth star of Alpha Centauri?
But this list is deceptive in that it hides the vast number of problems we have not answered but don’t remember because we were not pre-committed to their solution—we simply picked them up only to realize they were too hard or unpromising and we put them back down again on the search for more tractable and promising problems.
As you say, you can’t think of any problems in physics we couldn’t solve and no longer care about! In labs across the globe, PIs, post-docs, grad students, and researchers of all stripes are formulating vast numbers of questions, the majority of which are abandoned.
Scientists in the hard sciences routinely dismiss equivocal test results as unsatisfactory and opt to pursue alternative lines of inquiry that promise unequivocal findings. In many cases, the strength of typical inferences even makes statistical analysis unnecessary. The biologist Pamela Reinagel captures this common attitude concisely in this talk: “If you needed to do a statistical test, you just did a bad experiment,” and “If you needed statistics, you are studying something so trifling it doesn’t matter.” What is happening is that we are redefining our problems until we find regularities that are sufficiently strong so as to be worth pursuing.
In contrast, the education researcher and the development economist are stuck with only a few outcome variables they’re allowed to care about, and so have to dig through the crud of tiny R2 values to find anything of interest. When I give you such a problem—explain the cause of market crashes or effective interventions to improve educational outcomes—you might just be out of luck for how much of it you can explain with a few crisp variables and elegant relations. The social scientist in such cases doesn’t get to scour the expanse of the space of regularities in physical systems until they find a promising vein of inquiry heretofore unimagined.
Of course, no scientist is fully unrestricted in her capacity to redefine her problems, but we suspect that there are differences in degree here between the sciences that make a difference.
Tell me if the following vignette is elucidating.
You and your colleague are given the following tasks: you have 10 years in which to work on cancer come back with your greatest success, your colleague also has 10 years in which to work on cancer but they are also allowed to try to make progress on any other disease. Which do you expect will achieve the greatest success?
This case is simple in the sense that one domain of inquiry is a strict superset of another, which makes the value of larger search space more clear, but the difference in sizes of search space will be there in cases that are less obvious as well.
Our hypotheses suggest that the social scientists may be like the first researcher. They were pre-committed to smaller domain with less flexibility to go find promising veins of tractability in problem space. This, and variations on the general cardinality reasoning mentioned in footnote 7, are the core of our results.
I understand the argument, I think I buy a limited version of it (and also want to acknowledge that it is very clever and I do like it), but I also don’t think this can explain the magnitude of the difference between the different fields. If we go back and ask “what was physics’ original goal?” we end up with “to explain how the heavens move, and the path that objects travel”, and this has basically been solved. Physicists didn’t substitute this for something easier. The next big problem was to explain heat & electricity, and that was solved. Then the internals of the atom, and the paradox of a fixed speed of light. And those were solved.
I think maybe your argument holds for individual researchers. Individual education researchers are perhaps more constrained in what their colleagues will be interested in than individual physicists (though even that I’m somewhat doubtful of, maybe less doubtful on the scale of labs). But it seems to definitely break down when comparing the two fields against each other. Then, physics clearly has a very good track record of asking questions and then solving them extraordinarily well.
I understand the argument, I think I buy a limited version of it (and also want to acknowledge that it is very clever and I do like it)…
Thanks! I’ve so appreciated your comments and the chance to think about this with you!
…but I also don’t think this can explain the magnitude of the difference between the different fields.
I think that’s right—and we agree. As we note in the post, we only expect our hypotheses to explain a fairly modest fraction of the differences between fields. We see our contribution as showing how certain structural features—e.g., the cardinality of the set of tasks in a field’s search space—should influence our expectations about perceived difference in difficulty; not claiming they explain all or even most of the difference.
Then, physics clearly has a very good track record of asking questions and then solving them extraordinarily well.
I agree that the greatest hits of physics are truly great! That said, if by “track record” we mean something like the ratio of successes to failures (rather than greatest successes), then I think it’s genuinely tricky to assess—largely for structural reasons akin to those we highlight in the paper. We tend to preserve extraordinary successes while forgetting the countless unremarkable failures.
Perhaps. I appreciate the prompt to think more about this.
Here’s a picture that underpins our perspective and might provide a crux:
The world is full of hard questions. Under some suitable measure over questions, we might find that almost all questions are too hard to answer. Many large scale systems exhibit chaotic behavior, most parts of the universe are unreachable to us, most nonlinear systems of differential equations have no analytic solution. Some prediction tasks are theoretically unsolvable (e.g., an analytic solution to the three-body problem), some are just practically unsolvable given current technology (e.g., knowledge of sufficiently far off domains in our forward light cone).
Here are a few prediction tasks regarding the physical world that we have not answered. What is the exact direction of movement of a single atom in the next instant? How do we achieve nuclear fusion at room temperature? What constitutes a measurement in QM? Why do we observe electric charges but not magnetic monopoles? Why did the Pioneer 10 and 11 spacecraft experience an unexplained deceleration? What is the long-run behavior of any real-world chaotic system? Is the sound of knuckles cracking caused by the formation or collapse of air bubbles? How many grains of sand are there on a given beach on third planet on the fourth star of Alpha Centauri?
But this list is deceptive in that it hides the vast number of problems we have not answered but don’t remember because we were not pre-committed to their solution—we simply picked them up only to realize they were too hard or unpromising and we put them back down again on the search for more tractable and promising problems.
As you say, you can’t think of any problems in physics we couldn’t solve and no longer care about! In labs across the globe, PIs, post-docs, grad students, and researchers of all stripes are formulating vast numbers of questions, the majority of which are abandoned.
Scientists in the hard sciences routinely dismiss equivocal test results as unsatisfactory and opt to pursue alternative lines of inquiry that promise unequivocal findings. In many cases, the strength of typical inferences even makes statistical analysis unnecessary. The biologist Pamela Reinagel captures this common attitude concisely in this talk: “If you needed to do a statistical test, you just did a bad experiment,” and “If you needed statistics, you are studying something so trifling it doesn’t matter.” What is happening is that we are redefining our problems until we find regularities that are sufficiently strong so as to be worth pursuing.
In contrast, the education researcher and the development economist are stuck with only a few outcome variables they’re allowed to care about, and so have to dig through the crud of tiny R2 values to find anything of interest. When I give you such a problem—explain the cause of market crashes or effective interventions to improve educational outcomes—you might just be out of luck for how much of it you can explain with a few crisp variables and elegant relations. The social scientist in such cases doesn’t get to scour the expanse of the space of regularities in physical systems until they find a promising vein of inquiry heretofore unimagined.
Of course, no scientist is fully unrestricted in her capacity to redefine her problems, but we suspect that there are differences in degree here between the sciences that make a difference.
Tell me if the following vignette is elucidating.
You and your colleague are given the following tasks: you have 10 years in which to work on cancer come back with your greatest success, your colleague also has 10 years in which to work on cancer but they are also allowed to try to make progress on any other disease. Which do you expect will achieve the greatest success?
This case is simple in the sense that one domain of inquiry is a strict superset of another, which makes the value of larger search space more clear, but the difference in sizes of search space will be there in cases that are less obvious as well.
Our hypotheses suggest that the social scientists may be like the first researcher. They were pre-committed to smaller domain with less flexibility to go find promising veins of tractability in problem space. This, and variations on the general cardinality reasoning mentioned in footnote 7, are the core of our results.
I understand the argument, I think I buy a limited version of it (and also want to acknowledge that it is very clever and I do like it), but I also don’t think this can explain the magnitude of the difference between the different fields. If we go back and ask “what was physics’ original goal?” we end up with “to explain how the heavens move, and the path that objects travel”, and this has basically been solved. Physicists didn’t substitute this for something easier. The next big problem was to explain heat & electricity, and that was solved. Then the internals of the atom, and the paradox of a fixed speed of light. And those were solved.
I think maybe your argument holds for individual researchers. Individual education researchers are perhaps more constrained in what their colleagues will be interested in than individual physicists (though even that I’m somewhat doubtful of, maybe less doubtful on the scale of labs). But it seems to definitely break down when comparing the two fields against each other. Then, physics clearly has a very good track record of asking questions and then solving them extraordinarily well.
Thanks! I’ve so appreciated your comments and the chance to think about this with you!
I think that’s right—and we agree. As we note in the post, we only expect our hypotheses to explain a fairly modest fraction of the differences between fields. We see our contribution as showing how certain structural features—e.g., the cardinality of the set of tasks in a field’s search space—should influence our expectations about perceived difference in difficulty; not claiming they explain all or even most of the difference.
I agree that the greatest hits of physics are truly great! That said, if by “track record” we mean something like the ratio of successes to failures (rather than greatest successes), then I think it’s genuinely tricky to assess—largely for structural reasons akin to those we highlight in the paper. We tend to preserve extraordinary successes while forgetting the countless unremarkable failures.