I think this is a great project! Have you considered adding a donation button or using Patreon to allow readers to support the project?
I do have one big issue with the current way the information is presented: one of the most important things to take into account when making and interpreting predictions is the timeframe of the question. For example, if your are asking about the probability that Putin losses power, they the probability would likely be twice as high if you consider a 2 year timeframe compared to a 1 year time frame, assuming the probability per month does not change much.
Currently, the first 5 top-level headlines all ignores the timeframe, making the headlines meaningless:
“Putin >90% likely to remain in power, no change”
“Odds of ceasefire ~15%, recovering from ~12% low”
“Russia ~85% likely to gain territory, up from ~78%”
“Crimea land bridge ~30% chance of being cut, no change”
“Escalation: ~5% risk of NATO clash with Russia, down from ~9%”
The last one is particularly misleading. It compares the probabilities from start April to the probabilities now (start June). But one of the markets have deadline on June 12th and the other prediction have deadline July 1, so it is not surprising that the probability is down!
In order to conclude that the risk is decreasing, the question should have a moving deadline. I’m not aware of any prediction sites that allows questions with a moving timeframe, although it would be a great feature to have.
One thing that I have seen on manifold is markets that will resolve at a random time, with a distribution such that at any time, their expected duration (from the current day, conditional on not having already resolved) is 6 months. They do not seem particularly common, and are not quite equivalent to a market with a deadline exactly 6 months in the future. (I can’t seem to find the market.)
I’ve fixed the last headline now. I agree that it was being driven by the resolution date and therefore misleading.
I’m curious what you think of e.g. the Putin headline. Of the 8 Putin markets, some resolve by July, some by October, and most by 2024 -- but all of them show ~90% or better odds (well, Hypermind is at 88%). So even if technically correct, would including the resolution dates really add value?
Especially on an data-ink ratio basis?
Most prediction markets used on the site resolve by 2024 (unless specified otherwise on the chart). It would feel redundant to include “by 2024” in every single headline?
My subjective impression is that odds start collapsing as the resolution date gets near, more so than some linear decline as the market window slowly closes. Some markets do trend down (peace odds), but others sideways (Putin, Crimea), and others trend up (Russian territory gain). So my current sense is that as long as headlines don’t overweight markets closing within say ~2 months, it’s probably okay?
Anyway am still very open to feedback on this. Just trying to navigate the trade-off of keeping things simple on an already cluttered site...
Thanks again, it was a stimulating post and I think this is going to be an important issue for me to get right. Will keep thinking about it.
Oh, I’ve also added a Patreon buttonnow. Warning: no benefits yet! Purely to support the mission
Sorry my last comments wasn’t very constructive. I was also confusing two different critisisms:
that some changes in predicted probabilities are due to the deadline getting closer and you need to make sure not to claim that as news, and
that deadlines are not the in headlines and not always in the graphs either.
About 2): I don’t actually think this is much of a problem, if you ensure that the headline is not misleading and that the information about deadlines is easily available. However if the headline does not contain a deadline, and the deadline is relevant, I would not write any percentages in it. Putin has a 100% chance of dying, just like the rest of us, so it doesn’t make sense to say he has 90% chance of staying in power. In that case, I would prefer the headline to just state the direction the probability is moving in, e.g. “Putin hold on power in Russia is as stable as 3 month ago” or something like that.
To avoid writing “by 2024” in all headlines, maybe you could create subsections of the site by deadline. It would be a good user experince if you could feel like you are scrolling further into the future, starting with predictions for start of 2024, then 2025, then 2030. Of course this requires that there are several predictions for each deadline.
About 1), I think you should only include predictions if they cannot be explained by the nearing deadline.
For some questions this is not a problem at all, e.g. who is going to win an election.
For questions about whether something happens within a given timeframe, the best solution would be if prediction sites started making predictions with constant timeframe (e.g. 1 year) instead of constants deadline. I made a feature request about this to metaculous. They said they liked the idea, but they did not provide any prediction of the probability it would be implemented!
An alternative is to ask for a probability distribution for when something is going to happen. Such questions already exists on metaculous. Then you can see of expected remaining time or if time until median prediction is increasing, or something similar.
For questions with a fixed deadline, if the predicted probability of something happening is increasing, you can still conclude that the event is getting more likely.
For questions with fixed deadline and declining probabilities, it is harder to conclude anything. The very naive prediction would be linear decline, so p(t)/t is constant, where t denote time left and p(t) the probability at that time. E.g. with one month left t=1/12. A slightly less naive solution would be to model the event having constant probability at any time given that is hasn’t already happened. In this case, the constant would be log(1-p(t))/t.
In this model, if the probability is declining faster, meaning that |log(1-p(t))/t| is decreasing, I would stay that is a signal that the probability of the event is getting lower.
If p(t) is declining, but slow enough that |log(1-p(t))/t| is increasing, I would not conclude anything based on that, at least not on metaculus, because people forget to update their predictions sufficiently. I’m not familiar with other prediction sites, maybe it works better when there is money at stakes.
However, this model does not apply for all questions. It would be a useful model for e.g. “will there be a big cyber attack of a given type by a given date” that happens without warning, but for other questions like “Will Putin loose power be a given date”, we might expect some further indication of it happening before it happens, so we would expect the probability to go to 0 faster. For such questions questions, I don’t think you should ever conclude that the underlying event is getting less likely for a single fixed-deadline question.
So to conclude: if predicted probability of some event happing before the deadline is going up, it is fair to report it as the probability going up. If the prediction is going down you should only in rare cases conclude on the. The rare case is if you think the event would happen without much notice and |log(1-p(t))/t| is decreasing.
I think this is a great project! Have you considered adding a donation button or using Patreon to allow readers to support the project?
I do have one big issue with the current way the information is presented: one of the most important things to take into account when making and interpreting predictions is the timeframe of the question. For example, if your are asking about the probability that Putin losses power, they the probability would likely be twice as high if you consider a 2 year timeframe compared to a 1 year time frame, assuming the probability per month does not change much.
Currently, the first 5 top-level headlines all ignores the timeframe, making the headlines meaningless: “Putin >90% likely to remain in power, no change” “Odds of ceasefire ~15%, recovering from ~12% low” “Russia ~85% likely to gain territory, up from ~78%” “Crimea land bridge ~30% chance of being cut, no change” “Escalation: ~5% risk of NATO clash with Russia, down from ~9%”
The last one is particularly misleading. It compares the probabilities from start April to the probabilities now (start June). But one of the markets have deadline on June 12th and the other prediction have deadline July 1, so it is not surprising that the probability is down!
In order to conclude that the risk is decreasing, the question should have a moving deadline. I’m not aware of any prediction sites that allows questions with a moving timeframe, although it would be a great feature to have.
+1 was looking for a patreon etc.
In case you didn’t see the author’s comment below: there is now a patreon button!
One thing that I have seen on manifold is markets that will resolve at a random time, with a distribution such that at any time, their expected duration (from the current day, conditional on not having already resolved) is 6 months. They do not seem particularly common, and are not quite equivalent to a market with a deadline exactly 6 months in the future. (I can’t seem to find the market.)
Thank you!
I’ve fixed the last headline now. I agree that it was being driven by the resolution date and therefore misleading.
I’m curious what you think of e.g. the Putin headline. Of the 8 Putin markets, some resolve by July, some by October, and most by 2024 -- but all of them show ~90% or better odds (well, Hypermind is at 88%). So even if technically correct, would including the resolution dates really add value?
Especially on an data-ink ratio basis?
Most prediction markets used on the site resolve by 2024 (unless specified otherwise on the chart). It would feel redundant to include “by 2024” in every single headline?
My subjective impression is that odds start collapsing as the resolution date gets near, more so than some linear decline as the market window slowly closes. Some markets do trend down (peace odds), but others sideways (Putin, Crimea), and others trend up (Russian territory gain). So my current sense is that as long as headlines don’t overweight markets closing within say ~2 months, it’s probably okay?
Anyway am still very open to feedback on this. Just trying to navigate the trade-off of keeping things simple on an already cluttered site...
Thanks again, it was a stimulating post and I think this is going to be an important issue for me to get right. Will keep thinking about it.
Oh, I’ve also added a Patreon button now. Warning: no benefits yet! Purely to support the mission
I joined the Patreon!
Sorry my last comments wasn’t very constructive. I was also confusing two different critisisms:
that some changes in predicted probabilities are due to the deadline getting closer and you need to make sure not to claim that as news, and
that deadlines are not the in headlines and not always in the graphs either.
About 2): I don’t actually think this is much of a problem, if you ensure that the headline is not misleading and that the information about deadlines is easily available. However if the headline does not contain a deadline, and the deadline is relevant, I would not write any percentages in it. Putin has a 100% chance of dying, just like the rest of us, so it doesn’t make sense to say he has 90% chance of staying in power. In that case, I would prefer the headline to just state the direction the probability is moving in, e.g. “Putin hold on power in Russia is as stable as 3 month ago” or something like that.
To avoid writing “by 2024” in all headlines, maybe you could create subsections of the site by deadline. It would be a good user experince if you could feel like you are scrolling further into the future, starting with predictions for start of 2024, then 2025, then 2030. Of course this requires that there are several predictions for each deadline.
About 1), I think you should only include predictions if they cannot be explained by the nearing deadline.
For some questions this is not a problem at all, e.g. who is going to win an election.
For questions about whether something happens within a given timeframe, the best solution would be if prediction sites started making predictions with constant timeframe (e.g. 1 year) instead of constants deadline. I made a feature request about this to metaculous. They said they liked the idea, but they did not provide any prediction of the probability it would be implemented!
An alternative is to ask for a probability distribution for when something is going to happen. Such questions already exists on metaculous. Then you can see of expected remaining time or if time until median prediction is increasing, or something similar.
For questions with a fixed deadline, if the predicted probability of something happening is increasing, you can still conclude that the event is getting more likely.
For questions with fixed deadline and declining probabilities, it is harder to conclude anything. The very naive prediction would be linear decline, so p(t)/t is constant, where t denote time left and p(t) the probability at that time. E.g. with one month left t=1/12. A slightly less naive solution would be to model the event having constant probability at any time given that is hasn’t already happened. In this case, the constant would be log(1-p(t))/t.
In this model, if the probability is declining faster, meaning that |log(1-p(t))/t| is decreasing, I would stay that is a signal that the probability of the event is getting lower.
If p(t) is declining, but slow enough that |log(1-p(t))/t| is increasing, I would not conclude anything based on that, at least not on metaculus, because people forget to update their predictions sufficiently. I’m not familiar with other prediction sites, maybe it works better when there is money at stakes.
However, this model does not apply for all questions. It would be a useful model for e.g. “will there be a big cyber attack of a given type by a given date” that happens without warning, but for other questions like “Will Putin loose power be a given date”, we might expect some further indication of it happening before it happens, so we would expect the probability to go to 0 faster. For such questions questions, I don’t think you should ever conclude that the underlying event is getting less likely for a single fixed-deadline question.
So to conclude: if predicted probability of some event happing before the deadline is going up, it is fair to report it as the probability going up. If the prediction is going down you should only in rare cases conclude on the. The rare case is if you think the event would happen without much notice and |log(1-p(t))/t| is decreasing.