I’ve encountered similar things insofar as I’m better calibrated for some tasks than others. And I agree with you that defining the right reference classes for when to trust my estimations vs. when to trust the outside view (and which outside views to trust) is important.
I’m curious: if you re-express your data set in terms of standard deviations… e.g., the percentage of your estimated test grades that are within a std dev of the correct answer… rather than absolute percentages, do you still get very different results in the two cases?
I meant within the set of your 50 test scores, assuming they’re normalized to a common range.
To pick an extreme example: if all your test scores fall between 92% and 98%, it becomes less remarkable that your estimations of your test scores all fall within 3% of your actual test scores… anyone else could do about as well, given that fact about the data set. So it seems that knowing something about the distribution is helpful in reasoning about the causes of the differences in the accuracy of your judgments.
I’ve encountered similar things insofar as I’m better calibrated for some tasks than others. And I agree with you that defining the right reference classes for when to trust my estimations vs. when to trust the outside view (and which outside views to trust) is important.
I’m curious: if you re-express your data set in terms of standard deviations… e.g., the percentage of your estimated test grades that are within a std dev of the correct answer… rather than absolute percentages, do you still get very different results in the two cases?
Maybe I’m being really stupid, but how exactly would I define a standard deviation of the correct answer? Using the distribution for the whole class?
I meant within the set of your 50 test scores, assuming they’re normalized to a common range.
To pick an extreme example: if all your test scores fall between 92% and 98%, it becomes less remarkable that your estimations of your test scores all fall within 3% of your actual test scores… anyone else could do about as well, given that fact about the data set. So it seems that knowing something about the distribution is helpful in reasoning about the causes of the differences in the accuracy of your judgments.
Oh, that makes sense.
Nope, still a big difference. For example, here are my scores from the last few weeks:
Predicted/Actual: 98/100 72/72.5 94/94 85/86 82.5/87.5 90/92
Interesting that there were no too-high predictions.