I have some answers (for some guesses about what your question is, based on your comments) below.
Suppose I estimate the probability for event X at 50%. It’s possible that this is just my prior and if you give me any amount of evidence, I’ll update dramatically. Or it’s possible that this number is the result of a huge amount of investigation and very strong reasoning, such that even if you give me a bunch more evidence, I’ll barely shift the probability at all. In what way can I quantify the difference between these two things?
This sounds like Bayes’ Theorem, but the actual question about how you generate numbers given a hypothesis...I don’t know. There’s stuff around here about a good scoring rule I could dig up. Personally, I just make up numbers to give me an idea.
specify the function that you’ll shift to if a randomly chosen domain expert told you that yours was a certain amount too high/low.
I found this on higher order probabilities. (It notes the rule “for any x, x = PR[E given that Pr(E) = x]”.) Google also turned up some papers on the subject I haven’t read yet.
That makes sense to me, and what I’d do in practice too, but it still feels odd that there’s no theoretical solution to this question.
What’s your question?
I have some answers (for some guesses about what your question is, based on your comments) below.
This sounds like Bayes’ Theorem, but the actual question about how you generate numbers given a hypothesis...I don’t know. There’s stuff around here about a good scoring rule I could dig up. Personally, I just make up numbers to give me an idea.
This sounds like Inadequate Equilibria.
I found this on higher order probabilities. (It notes the rule “for any x, x = PR[E given that Pr(E) = x]”.) Google also turned up some papers on the subject I haven’t read yet.