That seems like a very valid point. Humans keep all sorts of reference classes and prior experience in the back of their minds. Much of it probably is so far back that making it conscious or even realizing which prior data is informing the conclusions could be quite difficult. At some level when people speak of intuition about things like math problems they are talking about precisely this sort of thing.
In my opinion, that humans aren’t Bayesians is much less of a problem than that humans can’t share all their information.
That seems like a very valid point. Humans keep all sorts of reference classes and prior experience in the back of their minds. Much of it probably is so far back that making it conscious or even realizing which prior data is informing the conclusions could be quite difficult. At some level when people speak of intuition about things like math problems they are talking about precisely this sort of thing.