Model-checking must (he says) be undertaken by other means because the truth may not be in the support of the prior, a situation in which the strict Bayesian is lost.
Loath as I am to disagree with Gelman & Shalizi, I’m not convinced that the sort of model-checking they advocate such as posterior p-values are fundamentally and in principle non-Bayesian, rather than practical problems. I mostly agree with “Posterior predictive checks can and should be Bayesian: Comment on Gelman and Shalizi,‘Philosophy and the practice of Bayesian statistics’”, Kruschke 2013 - I don’t see why that sort of procedure cannot be subsumed with more flexible and general models in an ensemble approach, and poor fits of particular parametric models found automatically and posterior shifted to more complex but better fitting models. If we fit one model and find that it is a bad model, then the root problem was that we were only looking at one model when we knew that there were many other models but out of laziness or limited computations we discarded them all. You might say that when we do an informal posterior predictive check, what we are doing is a Bayesian model comparison of one or two explicit models with the models generated by a large multi-layer network of sigmoids (specifically <80 billion of them)… If you’re running into problems because your model-space is too narrow—expand it! Models should be able to grow (this is a common feature of Bayesian nonparametrics).
This may be hard in practice, but then it’s just another example of how we must compromise our ideals because of our limits, not a fundamental limitation on a theory or paradigm.
Loath as I am to disagree with Gelman & Shalizi, I’m not convinced that the sort of model-checking they advocate such as posterior p-values are fundamentally and in principle non-Bayesian, rather than practical problems. I mostly agree with “Posterior predictive checks can and should be Bayesian: Comment on Gelman and Shalizi,‘Philosophy and the practice of Bayesian statistics’”, Kruschke 2013 - I don’t see why that sort of procedure cannot be subsumed with more flexible and general models in an ensemble approach, and poor fits of particular parametric models found automatically and posterior shifted to more complex but better fitting models. If we fit one model and find that it is a bad model, then the root problem was that we were only looking at one model when we knew that there were many other models but out of laziness or limited computations we discarded them all. You might say that when we do an informal posterior predictive check, what we are doing is a Bayesian model comparison of one or two explicit models with the models generated by a large multi-layer network of sigmoids (specifically <80 billion of them)… If you’re running into problems because your model-space is too narrow—expand it! Models should be able to grow (this is a common feature of Bayesian nonparametrics).
This may be hard in practice, but then it’s just another example of how we must compromise our ideals because of our limits, not a fundamental limitation on a theory or paradigm.