I’ve noticed in the past month that I’m really bottlenecked on my lack-of-calibration-training. Over the past couple years I’ve gotten into the habit of trying to operationalize predictions, but I haven’t actually tracked them in any comprehensive way.
This is supposed to be among the more trainable rationality skills, and nowadays it suddenly feels really essential. How long are lockdowns going to last? What’s going to happen with coronavirus cases? What’s going to happen with various political things going on that might affect me? Will the protests turn out to cause a covid wave? What am I even going to _want_ in a month? In six months?In a year?
I don’t have a justified sense of how likely things are, and how confident I am in that.
I looked around for good tools for making predictions, and found that PredictionBook.com is just pretty great. I’ve created a bookmark for the “new prediction” page so that it’s easy to get to. Some neat things that make it nice are:
1. You get to enter the “prediction resolution date” however you want (“in a week”, “6/31/2020”, “10 days from now” all work), so whatever my intuitive sense of when to resolve things is easy to enter.
2. It stores the last few prediction resolution dates, so if you want to keep using “N days from now” for similar reference class predictions, you can do so.
3. It emails you when the prediction should resolve, so you don’t forget.
Buy Wits & Wagers, use their cards for bite-sized numeric predictions you can state ranges for and check immediately. Best source of deliberate practice I know of.
I’ve played Wits and Wagers for this reason. But the issue is it doesn’t actually map that well to the skills I actually want (which is “calibrate estimate of how likely and event is to happen”, where the type of event is filtered for ‘the sorts of events I actually care about.’)
has made me pretty decent at calibration. By calibration I mean translating my feeling of uncertainty into a quantitative guess at uncertainty where that guess tracks with reality. I do not mean estimating accurately, I mean these two things:
1. Thinking about a sort of event I actually care about, coming up with a point estimate, then guessing the range around that point estimate such that the true answer is in that range roughly 50% of the time or roughly 90% of the time depending on what I’m going for.
2. Thinking about a sort of event I actually care about, coming up with a lower bound on a point estimate, coming up with an upper bound on a point estimate, shifting those bounds until my feelings of uncertainty that they’re actually lower/upper bounds are approximately equal for both of them, then taking the appropriate mean as my point estimate and having that point estimate be basically as good as I would have come up with in a more analytical way and also way faster to come up with.
I’ve noticed in the past month that I’m really bottlenecked on my lack-of-calibration-training. Over the past couple years I’ve gotten into the habit of trying to operationalize predictions, but I haven’t actually tracked them in any comprehensive way.
This is supposed to be among the more trainable rationality skills, and nowadays it suddenly feels really essential. How long are lockdowns going to last? What’s going to happen with coronavirus cases? What’s going to happen with various political things going on that might affect me? Will the protests turn out to cause a covid wave? What am I even going to _want_ in a month? In six months?In a year?
I don’t have a justified sense of how likely things are, and how confident I am in that.
I looked around for good tools for making predictions, and found that PredictionBook.com is just pretty great. I’ve created a bookmark for the “new prediction” page so that it’s easy to get to. Some neat things that make it nice are:
1. You get to enter the “prediction resolution date” however you want (“in a week”, “6/31/2020”, “10 days from now” all work), so whatever my intuitive sense of when to resolve things is easy to enter.
2. It stores the last few prediction resolution dates, so if you want to keep using “N days from now” for similar reference class predictions, you can do so.
3. It emails you when the prediction should resolve, so you don’t forget.
Buy Wits & Wagers, use their cards for bite-sized numeric predictions you can state ranges for and check immediately. Best source of deliberate practice I know of.
I’ve played Wits and Wagers for this reason. But the issue is it doesn’t actually map that well to the skills I actually want (which is “calibrate estimate of how likely and event is to happen”, where the type of event is filtered for ‘the sorts of events I actually care about.’)
Interesting. I believe some combination of
Wits & Wagers (not playing, practicing)
Poker
Software development
Ambient practice
has made me pretty decent at calibration. By calibration I mean translating my feeling of uncertainty into a quantitative guess at uncertainty where that guess tracks with reality. I do not mean estimating accurately, I mean these two things:
1. Thinking about a sort of event I actually care about, coming up with a point estimate, then guessing the range around that point estimate such that the true answer is in that range roughly 50% of the time or roughly 90% of the time depending on what I’m going for.
2. Thinking about a sort of event I actually care about, coming up with a lower bound on a point estimate, coming up with an upper bound on a point estimate, shifting those bounds until my feelings of uncertainty that they’re actually lower/upper bounds are approximately equal for both of them, then taking the appropriate mean as my point estimate and having that point estimate be basically as good as I would have come up with in a more analytical way and also way faster to come up with.