Similarly, if the Bayesian answer is difficult to compute, that doesn’t mean that Bayes is inapplicable; it means you don’t know what the Bayesian answer is.
The author is far too free with the notion of the Bayesian answer. At the level of common practice there is meta-analysis, which is fraught with problems. There’s subjective Bayesianism, which is fine in principle, but in practice has the same limitations: why should that be my prior? what underlying mechanism can explain all these inconsistently measured results and how do I formulate all those complicating possibilities into a likelihood function? Objective priors are a perennial subject of research in statistics which help somewhat in simple parametric problems. Non-parametric priors (e.g. Dirichlet, Gaussian, Levy, … processes) can be made to work in some cases, but aren’t easy to formulate in statistically efficient, computationally efficient, or even sensible (e.g. statistically consistent) ways, in general. AIXI and Solomonoff priors hold out tantalizing theoretical possibilities, but these are not yet practicable.
The best practice of smart scientists and statisticians in my experience is a process of iterative refinement. (This is true both in their own head and in the conduct of “Science.” Call it a computational shortcut, if you will, designed to accomodate our limited human brains. Importantly, it also let’s us stand on the shoulders of the giants who came before.)
One conducts experiments, build models and hypotheses, tests predictions, and conducts new experiments. Bayesian inference is only a consistent way to procede to the truth if the prior contains mass on that truth. Often reality turns out to be more complicated than we had any business imagining it to be before conducting the experiments, and practicable priors would have missed out on it.
The main point here then is that iteration and testing of models happens as a part of the process of understanding an experiment / analyzing data, not just at the level of designing a sequence of experiments.
The author is far too free with the notion of the Bayesian answer. At the level of common practice there is meta-analysis, which is fraught with problems. There’s subjective Bayesianism, which is fine in principle, but in practice has the same limitations: why should that be my prior? what underlying mechanism can explain all these inconsistently measured results and how do I formulate all those complicating possibilities into a likelihood function? Objective priors are a perennial subject of research in statistics which help somewhat in simple parametric problems. Non-parametric priors (e.g. Dirichlet, Gaussian, Levy, … processes) can be made to work in some cases, but aren’t easy to formulate in statistically efficient, computationally efficient, or even sensible (e.g. statistically consistent) ways, in general. AIXI and Solomonoff priors hold out tantalizing theoretical possibilities, but these are not yet practicable.
The best practice of smart scientists and statisticians in my experience is a process of iterative refinement. (This is true both in their own head and in the conduct of “Science.” Call it a computational shortcut, if you will, designed to accomodate our limited human brains. Importantly, it also let’s us stand on the shoulders of the giants who came before.)
One conducts experiments, build models and hypotheses, tests predictions, and conducts new experiments. Bayesian inference is only a consistent way to procede to the truth if the prior contains mass on that truth. Often reality turns out to be more complicated than we had any business imagining it to be before conducting the experiments, and practicable priors would have missed out on it.
The main point here then is that iteration and testing of models happens as a part of the process of understanding an experiment / analyzing data, not just at the level of designing a sequence of experiments.
Box’s 1980 paper contains much wisdom still relevant today to the aspiring rationalist: http://www.cs.princeton.edu/courses/archive/fall11/cos597C/reading/Box1980.pdf