I think the bigger question is what exactly it means for the probability estimate to be wrong. The best I can figure is that it’s not whatever it’s called where exactly x% of the predictions that you’re x% sure of are correct. In that case, Romney winning is evidence that it’s more extreme than it should be, and Obama winning is evidence that it’s less extreme. Whether or not it’s evidence that it’s wrong depends on whether you thought it was more likely to be more or less extreme beforehand.
If you buy the Bayesian argument (e.g. in Jaynes) that there is a single correct Pr(A|I) where I is your state of information and A is any proposition in question, then p, an estimate of Pr(A|I), is wrong if and only if p != Pr(A|I). In practice, we virtually never know Pr(A|I), so we can’t make this check. But as far as a conceptual understanding goes, that’s it—if, as I said, you buy the argument.
I think the bigger question is what exactly it means for the probability estimate to be wrong. The best I can figure is that it’s not whatever it’s called where exactly x% of the predictions that you’re x% sure of are correct. In that case, Romney winning is evidence that it’s more extreme than it should be, and Obama winning is evidence that it’s less extreme. Whether or not it’s evidence that it’s wrong depends on whether you thought it was more likely to be more or less extreme beforehand.
Usually probability prediction quality are broken into calibration and discrimination.
If you buy the Bayesian argument (e.g. in Jaynes) that there is a single correct Pr(A|I) where I is your state of information and A is any proposition in question, then p, an estimate of Pr(A|I), is wrong if and only if p != Pr(A|I). In practice, we virtually never know Pr(A|I), so we can’t make this check. But as far as a conceptual understanding goes, that’s it—if, as I said, you buy the argument.
In practice, we check the things gwern mentioned.