Can you elaborate more on whether there have been noticeable results in either A) taking successful actions based on the most recent predictions or B) improving the forecasting skills of the players? And if so- how were these things measured? How would you prefer to measure them?
We don’t have well-defined stats on how well people’s prediction skills have improved over time. From my anecdotal observations, pretty much everyone (myself included) starts out vastly overconfident, and then after losing a lot of points in their first few predictions, reaches an appropriate level of confidence. I’m not sure if anyone goes from ok to great though.
There have been few times that we’ve been able to take actions based on the predictions, because that requires the following combination of factors that tends not to occur together:
The answers are sufficiently clear-cut to make the prediction scorable
There are specific actions that depend on those well-defined answers
Enough people in our local community have enough insight to get some sort of wisdom of crowds.
The examples where the predictions led to decisions are:
Scott Alexander visited Boston and we hosted a meetup that he came to, and we wanted to run a survey at the meetup. We took predictions on how many people would attend, along with a conditional prediction market on the survey response rate for paper vs for tablet. Based on this, we went with paper. Attendance was much higher than anticipated, and we ended up running out of forms (but were able to make copies, so it worked out ok).
When our apartment had some maintenance issues and the landlord was giving us mixed signals about whether we’d be able to renew the lease, we took predictions on when/whether the issues would be resolved and whether the landlord would offer a renewal. Based on these, we decided not to look for a new place. The issue was in fact resolved and we were able to renew the lease.
One of our housemates has moved away, with some ambiguity around whether it’s temporary permanent, and we have predictions on whether they will return. TBD how this one will go.
Can you elaborate more on whether there have been noticeable results in either A) taking successful actions based on the most recent predictions or B) improving the forecasting skills of the players? And if so- how were these things measured? How would you prefer to measure them?
We don’t have well-defined stats on how well people’s prediction skills have improved over time. From my anecdotal observations, pretty much everyone (myself included) starts out vastly overconfident, and then after losing a lot of points in their first few predictions, reaches an appropriate level of confidence. I’m not sure if anyone goes from ok to great though.
There have been few times that we’ve been able to take actions based on the predictions, because that requires the following combination of factors that tends not to occur together:
The answers are sufficiently clear-cut to make the prediction scorable
There are specific actions that depend on those well-defined answers
Enough people in our local community have enough insight to get some sort of wisdom of crowds.
The examples where the predictions led to decisions are:
Scott Alexander visited Boston and we hosted a meetup that he came to, and we wanted to run a survey at the meetup. We took predictions on how many people would attend, along with a conditional prediction market on the survey response rate for paper vs for tablet. Based on this, we went with paper. Attendance was much higher than anticipated, and we ended up running out of forms (but were able to make copies, so it worked out ok).
When our apartment had some maintenance issues and the landlord was giving us mixed signals about whether we’d be able to renew the lease, we took predictions on when/whether the issues would be resolved and whether the landlord would offer a renewal. Based on these, we decided not to look for a new place. The issue was in fact resolved and we were able to renew the lease.
One of our housemates has moved away, with some ambiguity around whether it’s temporary permanent, and we have predictions on whether they will return. TBD how this one will go.