Does it make sense to calculate the score like this for events that aren’t independent? You no longer have the cool property that it doesn’t matter how you chop up your observations.
I think the correct thing to do would be to score the single probability that each model gave to this exact outcome. Equivalently you could add the scores for each state, but for each use the probabilities conditional on the states you’ve already scored. For 538 these probabilities are available via their interactive forecast.
Otherwise you’re counting the correlated part of the outcomes multiple times. So it’s not surprising that The Economist does best overall, because they had the highest probability for a Biden win and that did in fact occur.
EDIT: My suggested method has the nice property that if you score two perfectly correlated events then the second one always gives exactly 0 points.
Does it make sense to calculate the score like this for events that aren’t independent? You no longer have the cool property that it doesn’t matter how you chop up your observations.
I think the correct thing to do would be to score the single probability that each model gave to this exact outcome. Equivalently you could add the scores for each state, but for each use the probabilities conditional on the states you’ve already scored. For 538 these probabilities are available via their interactive forecast.
Otherwise you’re counting the correlated part of the outcomes multiple times. So it’s not surprising that The Economist does best overall, because they had the highest probability for a Biden win and that did in fact occur.
EDIT: My suggested method has the nice property that if you score two perfectly correlated events then the second one always gives exactly 0 points.
I think this comment would be better placed as a reply to the post that I’m linking. Perhaps you should put it there?
Done.