A related issue I seem to be having on Prediction Book. Judging from my past predictions I’m generally underconfident by quite a bit. According to the outcome graph no matter what percentage I estimate for something to happen, as long as I am giving it more than 50%, it seems to happen around 88% of the time. Slightly more for the >90% section, and similarly for the rounding to 100%, so my probability are at least weakly correlated in the right direction. Any thoughts on what I can do to better calibrate myself?
You don’t have very many predictions judged, so I’m not sure how reliable your worry is—only 8-11 predictions for each decile seems quite possible to just get lucky. Assuming that’s not the case, you could try mechanically bumping up every prediction by 10% and see what happens.
Also I notice there are many sets of predictions of the the form X will happen in 1 month/2 months/1 year/… with a separate prediction for each time period. How are these scored? Since these types of predictions are highly correlated scoring them individually can cause people to appear over or under confident.
Yes, that seems like a good point. I’m especially worried that my long-term predictions will turn out to be much too overconfident. And presumably I won’t see that until at least a while has gone by.
A related issue I seem to be having on Prediction Book. Judging from my past predictions I’m generally underconfident by quite a bit. According to the outcome graph no matter what percentage I estimate for something to happen, as long as I am giving it more than 50%, it seems to happen around 88% of the time. Slightly more for the >90% section, and similarly for the rounding to 100%, so my probability are at least weakly correlated in the right direction. Any thoughts on what I can do to better calibrate myself?
You don’t have very many predictions judged, so I’m not sure how reliable your worry is—only 8-11 predictions for each decile seems quite possible to just get lucky. Assuming that’s not the case, you could try mechanically bumping up every prediction by 10% and see what happens.
Also I notice there are many sets of predictions of the the form X will happen in 1 month/2 months/1 year/… with a separate prediction for each time period. How are these scored? Since these types of predictions are highly correlated scoring them individually can cause people to appear over or under confident.
Is there any reason to think the sets won’t balance out eventually?
The will, for very large values of eventually.
Yes, that seems like a good point. I’m especially worried that my long-term predictions will turn out to be much too overconfident. And presumably I won’t see that until at least a while has gone by.