The way you begin to grasp the Quest for the Holy Bayes is that you learn about cognitive phenomenon XYZ, which seems really useful—and there’s this bunch of philosophers who’ve been arguing about its true nature for centuries, and they are still arguing—and there’s a bunch of AI scientists trying to make a computer do it, but they can’t agree on the philosophy either -
And—Huh, that’s odd! - this cognitive phenomenon didn’t look anything like Bayesian on the surface, but there’s this non-obvious underlying structure that has a Bayesian interpretation—but wait, there’s still some useful work getting done that can’t be explained in Bayesian terms—no wait, that’s Bayesian too—OH MY GOD this completely different cognitive process, that also didn’t look Bayesian on the surface, ALSO HAS BAYESIAN STRUCTURE—hold on, are these non-Bayesian parts even doing anything?
Yes: Wow, those are Bayesian too!
No: Dear heavens, what a stupid design. I could eat a bucket of amino acids and puke a better brain architecture than that.
As Eliezer said in Searching for Bayes-Structure: