Introducing Pastcasting: A tool for forecasting practice

[Sage Future]

TL;DR: Visit pastcasting.com to forecast on already resolved questions you don’t have prior knowledge about to get quick feedback on how you’re doing.

We may try to coordinate multiplayer sessions in our Discord

Motivation

We want to make it easy for people to get better at forecasting. Existing tools are not well optimized for practicing relevant skills.

• Forecasting platforms and betting markets^[1] have slow feedback loops between predictions and question resolution. The questions with shorter time horizons often feel less important and may be systematically different than those with longer ones. Their scoring systems also incentivize constantly keeping predictions up to date and often heavily reward being the first to react to news

• Calibration training^[2] isolates the skill of intuiting probabilities and confidence intervals but doesn’t help with other aspects of forecasting (choosing reference classes & base rates, trend extrapolation, coming up with considerations, investigating different views, and determining the trustworthiness of news sources). Additionally, there is usually no reference point to compare your accuracy against.

Pastcasting

With pastcasting, you can:

• Forecast on already resolved questions from a vantage point further in the past

• Use our filtered search engine (“Vantage Search”) to look up relevant information without accidentally revealing the answer

• Receive immediate feedback on your forecasts and get scored against the crowd

• Host friendly multiplayer competitions where you and your friends can simultaneously pastcast on the same questions and see who does best

How it works

Question sources

Our questions are currently pulled from resolved Metaculus and GJOpen questions.

Scoring

We use relative log scoring against the original crowd forecast at that time. This means that you will receive zero points if you submit the same value as the crowd. The scoring rule is also strictly proper, meaning that your expected score is maximized if you report your true beliefs.

Vantage Search

Our preliminary results (in yellow) come from a search api with a restricted date range up to the vantage date. To further reduce information leakage (from the website changing its contents), we then pass the results through the internet archive api (in blue). This tends to be much slower, so some pastcasters opt to use the preliminary results directly.

Prior Knowledge

In many cases, users will have specific knowledge of how a particular question was resolved (from reading about it in the news, participating in forecasting that question, etc). We provide a button to skip these questions, and over time we will show questions that many users know the answer to less often.

Give it a try!

If this sounds useful, give it a try and give us feedback on what features would be the most useful and any aspects of the site you find confusing.

1. ^

   Such as Metaculus, Manifold Markets, etc…

2. ^

   Such as https://​​www.openphilanthropy.org/​​calibration, https://​​acritch.com/​​credence-game/​​, Wits & Wagers.

[Nathan Young]

I’ve used this a bit, but not loads. I prefer fatebook, metaculus and manifold and betting. I don’t quite know why I don’t use it more, here are some guesses.

• I found the tool kind of hard to use

• It was hard to search for the kind of information that I use to forecast

• Often I would generate priors based on my current state, but those were wrong in strange ways (I knew something happened but after the deadline)

• It wasn’t clear that it was helping me to get better versus doing lots of forecasting on other platforms.

[habryka]

I cannot claim to have used this tool very much since it was announced, but this sure seems like the way to go if one wanted to quickly get better at any kind of medium or long-term forecasting. I would also really love a review from someone who has actually used it a bunch, or a self-review by the people who built it.

[Raemon]

Oh man this is great. I was just wishing for something like this.

[Raemon]

Minor bug/​sadness report: I did the first question, then it prompted me to log in before revealing the answer, and AFAICT the question-answer didn’t get recorded and now I can’t find that question again and am curious and sad.

[Sage Future]

Thanks for the feedback! We’ve fixed the issue and you should be able to see our default question again after login.

[Thomas Kwa]

This is great. It seems to directly hit at making me more “genre-savvy”. If I were spending lots of time forecasting, I would probably do this for an hour a day.

The only improvement I can think of is to have forecasters study in hindsight what their ex-ante probability should have been, and show you that aggregate number too. But that seems like a lot of work plus potentially biased.

[Ben Snyder]

I really like this idea and I’ve been enjoying using the site!

I have noticed, however, that some of the questions include how they were resolved in the description. This is not a huge issue given that the system depends on self-regulation and there is already a mechanism for skipping questions for which users have prior knowledge. That said, it was confusing when I saw the question below because I thought “if the resolution of the forecast is embedded in the description, it would be silly to include it in the question set. Therefore it must not actually be giving me information about how the question resolved.”

This only happened for 2 out of the 30 questions I looked at, but it seems like something that people should be aware of.

[MP]

This is the most lesswrongian thing I’ve seen in a long time. And boy, count me in.

[MSRayne]

What’s to stop someone from pretending not to know the answer to a question they actually do know the answer to so they can miraculously “predict” it better and get more points?

[Sage Future]

Unfortunately, there’s no fully general solution to prevent cheating. We rely on people’s honor to skip questions they already know and intend it to be used mainly as a tool for training and competing against people you trust.