On behalf of the Metaculus team here: If @Dan H (or anyone) wants to actually easily, rigorously demonstrate a single-prompt forecasting bot that can outcompete Metaculus forecasters, I invite them to do so. We’ll even cover your API credits. Just enter your bot in the benchmarking series. We’ve completely solved the data leakage issue—which does appear to be a fatal flaw in Dan’s approach. I myself am skeptical that a bot using a single prompt can really compete given everyone’s attempts so far, but I would love to be proven wrong!
Ah, sure, I guess forecasting things in the future is of course a way to “solve the data leakage issues”, but that sure feels like a weird way of phrasing that. Like, there isn’t really a data leakage issue for future predictions.
On behalf of the Metaculus team here: If @Dan H (or anyone) wants to actually easily, rigorously demonstrate a single-prompt forecasting bot that can outcompete Metaculus forecasters, I invite them to do so. We’ll even cover your API credits. Just enter your bot in the benchmarking series. We’ve completely solved the data leakage issue—which does appear to be a fatal flaw in Dan’s approach. I myself am skeptical that a bot using a single prompt can really compete given everyone’s attempts so far, but I would love to be proven wrong!
How did you solve the data leakage issue? It seems quite gnarly, though not impossible, so am curious what you did.
I believe the approach is to forecast questions that resolve in the future and allow arbitrary internet access. I’m not totally sure though.
Ah, sure, I guess forecasting things in the future is of course a way to “solve the data leakage issues”, but that sure feels like a weird way of phrasing that. Like, there isn’t really a data leakage issue for future predictions.
Maybe “sidestep the data leakage issue” then. The series was designed with these issues in mind. (I work at Metaculus.)
Yeah, I mean, I think it’s fine and am glad about this work, I just definitely misunderstood (but it’s all a very harmless misunderstanding).