Hey, thanks for mentioning this. I hadn’t heard about it.
I’ve tried my hand at this app (50 questions or so), and it seems like the correct strategy, for me, is to go 50% for anything I have a little doubt on, and 99% for that I’m sure about. Maybe 5% of the questions fall into the 60%-90% range.
I’m still working to understand the tutorial and how to interpret my results.
It’s not particularly hard to “perfect” your calibration in that game—if you’re over/under on a certain percentile, you can throw questions where you’re confident into percentiles where you’re “poorly calibrated” in order to spoof a good calibration curve.
The trick to that game, if you actually want to asses your calibration, is to play for points rather than for a good curve. Being well-calibrated means that when you play for points, you have a good curve automatically.
(I wish that they’d show you your curve less often, perhaps only when you leave the game. It’s hard to resist cheating the curve. Then again, I’m not sure of a better way to provide the necessary feedback.)
I’m not strong enough in math to figure out how the scoring actually works without spending some time with it, and I wouldn’t “throw” questions anyway. But I do like seeing that, say, on my 60%s I’m actually right 70% of the time. So when I’m feeling “60%” I should actually go with 70% more often. I think I’m afraid of getting questions wrong because the score penalty appears so high relative to the score bonus (I know that’s likely appropriate, even though I don’t understand the actual log bits, etc of scoring ).
The scoring is done so that if you have 70% of your answers right, then you get the best average score by guessing 70%, not 60%. The increased penalty you get for getting 30% of those answers wrong is smaller than the increased gain for getting 70% of them right.
But that’s true only as long as you really get 70% of them right; so changing your answer e.g. to 80% while being only 70% correct would decrease the average score, because then the increased penalty for getting 30% of those answers wrong would be greater than the increased gain for getting the 70% right.
Without understanding the log bits, you can easily verify this in a spreadsheet calculator. Make a formula saying how many points you get if you report probability R and if you really get P answers right. Playing with numbers, you will find out that for a given P, you get the highest average score for R = P.
Hey, thanks for mentioning this. I hadn’t heard about it.
I’ve tried my hand at this app (50 questions or so), and it seems like the correct strategy, for me, is to go 50% for anything I have a little doubt on, and 99% for that I’m sure about. Maybe 5% of the questions fall into the 60%-90% range.
I’m still working to understand the tutorial and how to interpret my results.
:-)
It’s not particularly hard to “perfect” your calibration in that game—if you’re over/under on a certain percentile, you can throw questions where you’re confident into percentiles where you’re “poorly calibrated” in order to spoof a good calibration curve.
The trick to that game, if you actually want to asses your calibration, is to play for points rather than for a good curve. Being well-calibrated means that when you play for points, you have a good curve automatically.
(I wish that they’d show you your curve less often, perhaps only when you leave the game. It’s hard to resist cheating the curve. Then again, I’m not sure of a better way to provide the necessary feedback.)
I’m not strong enough in math to figure out how the scoring actually works without spending some time with it, and I wouldn’t “throw” questions anyway. But I do like seeing that, say, on my 60%s I’m actually right 70% of the time. So when I’m feeling “60%” I should actually go with 70% more often. I think I’m afraid of getting questions wrong because the score penalty appears so high relative to the score bonus (I know that’s likely appropriate, even though I don’t understand the actual log bits, etc of scoring ).
The scoring is done so that if you have 70% of your answers right, then you get the best average score by guessing 70%, not 60%. The increased penalty you get for getting 30% of those answers wrong is smaller than the increased gain for getting 70% of them right.
But that’s true only as long as you really get 70% of them right; so changing your answer e.g. to 80% while being only 70% correct would decrease the average score, because then the increased penalty for getting 30% of those answers wrong would be greater than the increased gain for getting the 70% right.
Without understanding the log bits, you can easily verify this in a spreadsheet calculator. Make a formula saying how many points you get if you report probability R and if you really get P answers right. Playing with numbers, you will find out that for a given P, you get the highest average score for R = P.