Thanks for a great post! I have a concern about your sample sizes however.
I am looking into similar questions myself, and while reading your post I was surprised to see your Metaculus sample claimed as 45k predictions. These are not actually individual predictions, but rather the time series of community predictions, which are much less information dense, as this is just the median of the recent community predictions at that time and typically a new prediction will have a small effect on this value. I think claiming the sample size is 45k is therefore a bit misleading.
It also has the effect of linearly weighting Metaculus questions by community interest, which is not obviously a desirable method (this is mitigated by the cap on the time series length to 101, which means the effect will ultimately be small as you have 557 Metaculus questions, implying your average question must have more than 80 predictions)
Yep, I share your concerns! I wanted to include them in the post, but then I got busy. Perhaps I’ll update it in the forseeable future (no promises however, I’m pretty busy with other things). Maybe I’ll just put a warning at the top of the article.
And, in case you publish your stuff, I’d love to read it.
Thanks for a great post! I have a concern about your sample sizes however.
I am looking into similar questions myself, and while reading your post I was surprised to see your Metaculus sample claimed as 45k predictions. These are not actually individual predictions, but rather the time series of community predictions, which are much less information dense, as this is just the median of the recent community predictions at that time and typically a new prediction will have a small effect on this value. I think claiming the sample size is 45k is therefore a bit misleading.
It also has the effect of linearly weighting Metaculus questions by community interest, which is not obviously a desirable method (this is mitigated by the cap on the time series length to 101, which means the effect will ultimately be small as you have 557 Metaculus questions, implying your average question must have more than 80 predictions)
Yep, I share your concerns! I wanted to include them in the post, but then I got busy. Perhaps I’ll update it in the forseeable future (no promises however, I’m pretty busy with other things). Maybe I’ll just put a warning at the top of the article.
And, in case you publish your stuff, I’d love to read it.