Any chance of turning this (and some of your other comments) into a top-level post? (perhaps something like, “When You Can (And Can’t) Do Better Than Science”?)
I think the first section should ignore the philosophy of science and cover the science of science, the sociology of it, and concede the sharpshooter’s fallacy, assuming that whatever science does it is trying to do. The task of improving upon the method is then not too normative, since one can simply achieve the same results with fewer resources/better results with the same resources. Also, that way science can’t blame perceived deficiencies on the methods of philosophy, as it could were one to evaluate science according to philosophy’s methods and standards. This section would be the biggest added piece of value that isn’t tying together things already on this site.
A section should look for edges with only one labeled node in the scientific methods where science requires input from a mystery method, such as how scientists generate hypotheses or how scientific revolutions occur. These show the incompleteness of the scientific method as a means to acquire knowledge, even if it is perfect at what it does. Formalization and improvement of the mystery methods would contribute to the scientific method, even if nothing formal within the model changes.
A section should discuss how science isn’t a single method (according to just about everybody), but instead a family of similar methods varying especially among fields. This weakens any claim idealizing science in general, as at most one could claim that a particular field’s method is ideal for human thought and discovery. Assuming each (or most) fields’ methods are ideal (this is the least convenient possible world for the critic of the scientific method as practiced), the costs and benefits of using that method rather than a related scientific method can be speculated upon. I expect to find, as policy debates should not be one sided, that were a field to use other fields’ methods it would have advantages and disadvantages; the simple case is choice of stricter p-value modulating wrong things believed at the expense of true things not believed.
Sections should discuss abuses of statistics, one covering violations of the law (failing to actually test P(B|~A) and instead testing P((B + (some random stuff) - (some other random stuff)|~A) and another covering systemic failures such as publication bias and failure to publish replications. This would be a good place to introduce intra-scientific debates about such things to show both that science isn’t a monolithic outlook that can be supported and how one side in the civil war is aligned with Bayesian critiques. To the extent science is not settled on what the sociology of science is, that is a mark of weakness—it may be perfectly calibrated, but it isn’t too discriminatory here.
A concession I imagine pro-science people might make is to concede the weakness of soft science, such as sociology. Nonetheless, sociology’s scientific method is deeply related to hard sciences’, and its shortcomings somewhat implicate them. What’s more, if sociology is so weak, one wonders whence the pro-science person gets their strong pro-science view. One possibility is that they get it purely from philosophy of science, (a school of which) they wholly endorse, but if that is the case they don’t have an objection in kind to those who also predict science as is works decently but have severe criticisms of it and ideas on how to improve upon it, i.e. Bayesians.
I think it’s fair to contrast the scientific view of science with a philosophical view of Bayesianism to see if they are of the same scope. If science has no position on whether or not science is an approximation of Bayesian reasoning, and Bayesianism does, that is at least one question addressed by the one and not the other. It would be easy to invent a method that’s not useful for finding truth that has a broader scope than science, e.g. answering “yes” to every yes or no question unless it would contradict a previous response. This alone would show they are not synonymous.
A problem with the title “When You Can (And Can’t) Do Better Than Science” is that it is binary, but I really want three things explicitly expressed: 1) When you can do better than science by being stricter than science, 2) when you can do better than science by being more lenient than science, 3) when you can’t do better than science. The equivocation and slipperiness surrounding what it is reasonable to do is a significant part of the last category, e.g. one doesn’t drive where the Tappan Zee Bridge should have been built. The other part is near-perfect ways science operates now according to a reasonable use of “can’t”; I wouldn’t expect science to be absolutely and exactly perfect anywhere, any more than I can be absolutely sure with a probability of 1 that the Flying Spaghetti Monster doesn’t exist.
Second order Bayesianism deserves mention as the thing being advocated. A “good Bayesian” may use heuristics to counteract bias other than just Bayes’ rule, such as the principle of charity, or pretending things are magic to counteract the effort heuristic, or reciting a large number of variably sized numbers to counteract the anchoring effect, etc.
Is there a better analogy than the driving to the airport one for why Bayes’ Rule being part of the scientific toolbox doesn’t show the scientific toolbox isn’t a rough approximation of how to apply Bayes’ Rule? The other one I thought of is light’s exhibiting quantum behavior directly, it being a subset of all that is physical, but all that is physical actually embodying quantum behavior.
A significant confusion is discussing beliefs as if they weren’t probabilistic and actions in some domains as if they ought not be influenced by anything not in a category of true belief “scientifically established”. Bayesianism explains why this is a useful approximation of how one should actually act and thereby permits one to deviate from it without having to claim something like “science doesn’t work”.
Any chance of turning this (and some of your other comments) into a top-level post? (perhaps something like, “When You Can (And Can’t) Do Better Than Science”?)
Yes.
I think the first section should ignore the philosophy of science and cover the science of science, the sociology of it, and concede the sharpshooter’s fallacy, assuming that whatever science does it is trying to do. The task of improving upon the method is then not too normative, since one can simply achieve the same results with fewer resources/better results with the same resources. Also, that way science can’t blame perceived deficiencies on the methods of philosophy, as it could were one to evaluate science according to philosophy’s methods and standards. This section would be the biggest added piece of value that isn’t tying together things already on this site.
A section should look for edges with only one labeled node in the scientific methods where science requires input from a mystery method, such as how scientists generate hypotheses or how scientific revolutions occur. These show the incompleteness of the scientific method as a means to acquire knowledge, even if it is perfect at what it does. Formalization and improvement of the mystery methods would contribute to the scientific method, even if nothing formal within the model changes.
A section should discuss how science isn’t a single method (according to just about everybody), but instead a family of similar methods varying especially among fields. This weakens any claim idealizing science in general, as at most one could claim that a particular field’s method is ideal for human thought and discovery. Assuming each (or most) fields’ methods are ideal (this is the least convenient possible world for the critic of the scientific method as practiced), the costs and benefits of using that method rather than a related scientific method can be speculated upon. I expect to find, as policy debates should not be one sided, that were a field to use other fields’ methods it would have advantages and disadvantages; the simple case is choice of stricter p-value modulating wrong things believed at the expense of true things not believed.
Sections should discuss abuses of statistics, one covering violations of the law (failing to actually test P(B|~A) and instead testing P((B + (some random stuff) - (some other random stuff)|~A) and another covering systemic failures such as publication bias and failure to publish replications. This would be a good place to introduce intra-scientific debates about such things to show both that science isn’t a monolithic outlook that can be supported and how one side in the civil war is aligned with Bayesian critiques. To the extent science is not settled on what the sociology of science is, that is a mark of weakness—it may be perfectly calibrated, but it isn’t too discriminatory here.
A concession I imagine pro-science people might make is to concede the weakness of soft science, such as sociology. Nonetheless, sociology’s scientific method is deeply related to hard sciences’, and its shortcomings somewhat implicate them. What’s more, if sociology is so weak, one wonders whence the pro-science person gets their strong pro-science view. One possibility is that they get it purely from philosophy of science, (a school of which) they wholly endorse, but if that is the case they don’t have an objection in kind to those who also predict science as is works decently but have severe criticisms of it and ideas on how to improve upon it, i.e. Bayesians.
I think it’s fair to contrast the scientific view of science with a philosophical view of Bayesianism to see if they are of the same scope. If science has no position on whether or not science is an approximation of Bayesian reasoning, and Bayesianism does, that is at least one question addressed by the one and not the other. It would be easy to invent a method that’s not useful for finding truth that has a broader scope than science, e.g. answering “yes” to every yes or no question unless it would contradict a previous response. This alone would show they are not synonymous.
A problem with the title “When You Can (And Can’t) Do Better Than Science” is that it is binary, but I really want three things explicitly expressed: 1) When you can do better than science by being stricter than science, 2) when you can do better than science by being more lenient than science, 3) when you can’t do better than science. The equivocation and slipperiness surrounding what it is reasonable to do is a significant part of the last category, e.g. one doesn’t drive where the Tappan Zee Bridge should have been built. The other part is near-perfect ways science operates now according to a reasonable use of “can’t”; I wouldn’t expect science to be absolutely and exactly perfect anywhere, any more than I can be absolutely sure with a probability of 1 that the Flying Spaghetti Monster doesn’t exist.
Second order Bayesianism deserves mention as the thing being advocated. A “good Bayesian” may use heuristics to counteract bias other than just Bayes’ rule, such as the principle of charity, or pretending things are magic to counteract the effort heuristic, or reciting a large number of variably sized numbers to counteract the anchoring effect, etc.
Is there a better analogy than the driving to the airport one for why Bayes’ Rule being part of the scientific toolbox doesn’t show the scientific toolbox isn’t a rough approximation of how to apply Bayes’ Rule? The other one I thought of is light’s exhibiting quantum behavior directly, it being a subset of all that is physical, but all that is physical actually embodying quantum behavior.
A significant confusion is discussing beliefs as if they weren’t probabilistic and actions in some domains as if they ought not be influenced by anything not in a category of true belief “scientifically established”. Bayesianism explains why this is a useful approximation of how one should actually act and thereby permits one to deviate from it without having to claim something like “science doesn’t work”.
Thoughts?