Does it make sense to calculate the score like this for events that aren’t independent? You no longer have the cool property that it doesn’t matter how you chop up your observations.
I think the correct thing to do would be to score the single probability that each model gave to this exact outcome. Equivalently you could add the scores for each state, but for each use the probability conditional on the states you’ve already scored. For 538 these probabilities are available via their interactive forecast.
Otherwise you’re counting the correlated part of the outcomes multiple times. So it’s not surprising that The Economist does best overall, because they had the highest probability for a Biden win and that did in fact occur.
My suggested method has the nice property that if you score two perfectly correlated events then the second one always gives exactly 0 points.
I just did the calculations. Using the interactive forecast from 538 gives them a score of −9.027; using the electoral_college_simulations.csv data from The Economist gives them a score of −7.841. So The Economist still wins!
Thanks; you are right and the thing I originally wrote was wrong. For posterity, I added a disclaimer (sixth paragraph) that credits you and the other person who pointed this out.
Does it make sense to calculate the score like this for events that aren’t independent? You no longer have the cool property that it doesn’t matter how you chop up your observations.
I think the correct thing to do would be to score the single probability that each model gave to this exact outcome. Equivalently you could add the scores for each state, but for each use the probability conditional on the states you’ve already scored. For 538 these probabilities are available via their interactive forecast.
Otherwise you’re counting the correlated part of the outcomes multiple times. So it’s not surprising that The Economist does best overall, because they had the highest probability for a Biden win and that did in fact occur.
My suggested method has the nice property that if you score two perfectly correlated events then the second one always gives exactly 0 points.
I just did the calculations. Using the interactive forecast from 538 gives them a score of −9.027; using the electoral_college_simulations.csv data from The Economist gives them a score of −7.841. So The Economist still wins!
Thanks; you are right and the thing I originally wrote was wrong. For posterity, I added a disclaimer (sixth paragraph) that credits you and the other person who pointed this out.