To clarify/rephrase/expand on this, i think Alexandros is suggesting that questions “how do humans think”, and “what is a rational way to think” are separate questions, and if we are discussing the first of these two questions then perhaps we have been sidetracked.
In fact, this is nicely highlighted by your very first sentence:
People think A&B is more likely than A alone, if you ask the right question. That’s not very Bayesian; as far as you Bayesians can tell it’s really quite stupid.
That is a quite stupid way to think, and if we want to think rationally we should desire to not think that way, regardless of whether it is in fact a common way of thinking.
To clarify/rephrase/expand on this, i think Alexandros is suggesting that questions “how do humans think”, and “what is a rational way to think” are separate questions, and if we are discussing the first of these two questions then perhaps we have been sidetracked.
In fact, this is nicely highlighted by your very first sentence:
That is a quite stupid way to think, and if we want to think rationally we should desire to not think that way, regardless of whether it is in fact a common way of thinking.