One option would be to have another percentage — a meta-percentage. e.g. “What credence do i give to “this is an accurate model of the world””? For coin flips, you’re 99.999% that 50% is a good model. For bloxors, you’re ~0% that 50% is a good model.
This is a model that I always tend to fall back on but I can never find a name for it so find it hard to look into. I have always figured I am misunderstanding Bayesian statistics and somehow credence is all factored in somehow. That doesn’t really seem like the case though.
Does the Scott Alexander post lay this out? I am having difficulty finding it.
The closest term I have been able to find is Kelly constants, which is a measure of how much “wealth” you should rationally put into a probabilistic outcome. Replace “wealth” with credence and maybe it could be useful for decisions but even this misses the point!
Agreed, great post. But I think you are trying to push Bayesian Statistics past what it SHOULD be used for.
Bayesian Statistics are only useful because we approach the correct answer as we gain all the information possible. Only in this limit (of infinite information) is Bayesian useful. Priors based off no information are, well, useless.
Scenario 1: You flip a fair coin and have a 50⁄50 chance of it landing heads
Scenario 2: (to steal xepo’s example) are bloxors greeblic? You have NO IDEA, so your priors are 50⁄50
Even though in both scenarios the chances are 50⁄50, I would feel much more confident betting money on scenario 1 than scenario 2. Therefore my model of choices contains something MORE than probabilities. As far as I know Bayesian statistics just doesn’t convey this NEEDED information. You cant use Bayesian probabilities here in a useful way. It’s the wrong tool for the job.
Even frequentest statistics is useless here.
A lot of day-to-day decisions are based off very limited information. I am not able to lay out a TRUE model of how we intuitively make those decisions but “how much information I have to work with” is definitely an aspect in my mental model that is not entirely captured by Bayes Theorem.