This is somewhat solved if you have a forecaster that you trust that can make a prediction based on Sophia’s seeming ability and honesty. The naive thing would be for that forecaster to predict their own distribution of the log-loss of Sophia, but there’s perhaps a simpler solution. If Sophia’s provided loss distribution is correct, that would mean that she’s calibrated in this dimension (basically, this is very similar to general forecast calibration). The trusted forecaster could forecast the adjustment made to her term, instead of forecasting the same distribution. Generally this would be in the direction of adding expected loss, as Sophia probably had more of an incentive to be overconfident ( which would result in a low expected score from her) than underconfident. This could perhaps make sense as a percentage modifier (-30% points), a mean modifier (-3 to −8 points), or something else. Is it actually true that forecasters would find it easier to forecast the adjustment?> This is somewhat solved if you have a forecaster that you trust that can make a prediction based on Sophia’s seeming ability and honesty. The naive thing would be for that forecaster to predict their own distribution of the log-loss of Sophia, but there’s perhaps a simpler solution. If Sophia’s provided loss distribution is correct, that would mean that she’s calibrated in this dimension (basically, this is very similar to general forecast calibration). The trusted forecaster could forecast the adjustment made to her term, instead of forecasting the same distribution. Generally this would be in the direction of adding expected loss, as Sophia probably had more of an incentive to be overconfident ( which would result in a low expected score from her) than underconfident. This could perhaps make sense as a percentage modifier (-30% points), a mean modifier (-3 to −8 points), or something else.
Is it actually true that forecasters would find it easier to forecast the adjustment?
One nice thing about adjustments is that they can be applied to many forecasts. Like, I can estimate the adjustment for someone’s [list of 500 forecasts] without having to look at each one.
Over time, I assume that there would be heuristics for adjustments, like, “Oh, people of this reference class typically get a +20% adjustment”, similar to margins of error in engineering.
That said, these are my assumptions, I’m not sure what forecasters will find to be the best in practice.
Is it actually true that forecasters would find it easier to forecast the adjustment?
One nice thing about adjustments is that they can be applied to many forecasts. Like, I can estimate the adjustment for someone’s [list of 500 forecasts] without having to look at each one.
Over time, I assume that there would be heuristics for adjustments, like, “Oh, people of this reference class typically get a +20% adjustment”, similar to margins of error in engineering.
That said, these are my assumptions, I’m not sure what forecasters will find to be the best in practice.