I find the datapoint interesting, but I don’t really why it’s much evidence of this being “a huge issue”. Most markets resolve neatly. It seems fine for 5% of markets or so to end up in dispute. People can price in a bunch of the resolution uncertainty. Do we have any evidence this is seriously hurting prediction market adoption or performance?
I agree that most markets resolve successfully, but think we might not be on the same page on how big a deal it is for 5% of markets to end up ambiguous.
If someone offered you a security with 95% odds to track Google stock performance and 5% odds to instead track how many hairs were on Sundar Pichai’s head, this would not be a great security! A stock market that worked like that would not be a great stock market!
In particular:
I think this ambiguity is a massive blow to arbitrage strategies (which are a big part of the financial infrastructure we’re hoping will make prediction markets accurate). There are already a lot of barriers in the way of profiting from a situation where e.g. one market says 70% and one market says 80%: if there’s a chance that the two will resolve some ambiguity differently, that adds a very big risk to anyone attempting to arbitrage that difference.[1]
I think this ambiguity is very dangerous to the hopes of prediction markets as a trustworthy canonical source on controversial events. If Maduro says “See, everyone, I won the election fair and square, the prediction markets agree!”[2], I think it’s very important that the prediction markets be perfectly clear on what they are tracking and what they are not tracking, and I don’t think that’s currently the case.
Related to #2, I think that this ambiguity is vastly more likely to occur in the case of the controversial events where it’s most valuable to have a trustworthy and canonical source. The cases where markets resolve cleanly are exactly the cases where non-prediction-market mechanisms already reach a canonical consensus on their own.
This also makes Manifold’s preferred strategy of dealing with ambiguity by N/A-ing a market less valuable: that’s an acceptable resolution for someone who just did buy-and-hold on that one market, but can be very bad for someone who was trading actively across multiple markets some of which N/A-ed and some of which did not.
This imagines a world where prediction markets are major enough and mainstream enough for people to be looking at them and talking about them: but that’s exactly what prediction market advocates want!
If someone offered you a security with 95% odds to track Google stock performance and 5% odds to instead track how many hairs were on Sundar Pichai’s head, this would not be a great security!
My sense is this security would be fine? Is there a big issue with this being a security?
In most domains except the most hardened part of the stock market counterparty risk is generally >5%. The key issues come when failure is correlated, but it seems to me indeed that in prediction markets it’s pretty random which way ambiguity resolves, and so you get pretty uncorrelated failures (like, if you are invested in 10,000 markets, while it might be the case that 500 of them resolve in a surprising and ambiguous way, you will pretty randomly be on the losing or winning side of it, so it mostly just cancels out).
In most domains except the most hardened part of the stock market counterparty risk is generally >5%.
This seems quite wrong to me:
High Yield Corporate Bond OAS spreads are <5% according to bloomberg, and most of that is economic risk, not “you will get screwed by a change of rules” risk.
Trades on US stock exchanges almost always succeed, many more 9s than just one.
If I buy a product in a box in a supermarket the contents of the box match the label >>95% of the time.
Banks make errors with depositor balances <<5% of the time.
Most employers manage to pay fortnightly wages on time without missing one or more paycheques per year.
Once you’re seated in an Uber or Taxi they take you to your destination almost all the time.
Your utility company fulfills its obligations to supply your house >>95% of the time under all but the most extreme circumstances.
Most employees turn up >95% of non-holiday days, and most students maintain >95% attendance.
Sorry, I wanted to say “except the most hardened parts of the world (like the stock market)”. I agree with you that basically anything in the stock market has much less counterparty risk than that. I disagree with basically all non-trading examples you give.
Once you’re seated in an Uber or Taxi they take you to your destination almost all the time.
My sense is around 1⁄20 Ubers don’t show up, or if they show up, fail to do their job in some pretty obvious and clear way.
If I buy a product in a box in a supermarket the contents of the box match the label >>95% of the time.
True for the most commoditized products. For anything else error rates seem to me to be around 5%. My guess is my overall Amazon error rate has been around 2%, which is lower, but not much lower (usually Amazon sent me something broken that previously was returned where they couldn’t spot the error).
Most employers manage to pay fortnightly wages on time without missing one or more paycheques per year.
I think that’s false, at least the statistics on wage theft seemed quite substantial to me. I am kind of confused how to interpret these, but various different studies on Wikipedia suggest wage theft on-average to be around 5%-15% (higher among lower-income workers).
Your utility company fulfills its obligations to supply your house >>95% of the time under all but the most extreme circumstances.
I agree this is true for gas and water (and mostly true for electricity, though PG&E is terrible and Berkeley really has a lot of outages).
Overall, I think 5% counterparty risk seems about right for most contracts I sign or business relationship I have. I agree that trading infrastructure is quite robust and in highly commoditized environments you get below that, but that’s not the majority of my economic transactions.
I agree with you that basically anything in the stock market has much less counterparty risk than that. I disagree with basically all non-trading examples you give.
It’s not just the stock market, it’s true for the bond market, the derivatives market, the commodities market… financial markets, a category which includes prediction markets, cannot function effectively with counterparty risk anything like 5%.
My sense is around 1⁄20 Ubers don’t show up, or if they show up, fail to do their job in some pretty obvious and clear way.
If the Uber doesn’t show up I’m not sure that’s counterparty risk: you haven’t paid anything, so it seems more like them declining the contract. The equivalent for a prediction market would be if you hit ‘buy’ and the button didn’t work, not for when you have paid the money and then don’t get the result taken from you. That’s much less bad than if the trade went through and then was settled incorrectly.
I think that’s false, at least the statistics on wage theft seemed quite substantial to me. I am kind of confused how to interpret these, but various different studies on Wikipedia suggest wage theft on-average to be around 5%-15% (higher among lower-income workers).
I think those studies have significant methodological flaws, though unfortunately I can’t remember the specific issues off the top off my head, so this may not be very convincing to you.
I agree this is true for gas and water (and mostly true for electricity, though PG&E is terrible and Berkeley really has a lot of outages).
According to the first google hit, PG&E said the average customer suffered 255.9 minutes of outage in 2013, which is a lot higher than I expected, but is still only 100*255.9/(60*24*365) = 0.05%
It’s not just the stock market, it’s true for the bond market, the derivatives market, the commodities market… financial markets, a category which includes prediction markets, cannot function effectively with counterparty risk anything like 5%.
Hmm, maybe I am just failing to model something here. Isn’t really the only thing that happens when you have 5% randomly-distributed counterparty risk that you end up with like 5% spreads? That seems fine to me.
To be clear, I don’t feel very confident here, I just don’t really understand why you can’t just price in counterparty risk and then maybe end up with some bigger spreads (which I do agree is sad for prediction markets, but for most markets I don’t mind the spread that much).
My sense is this security would be fine? Is there a big issue with this being a security?
In the sense that it would find a market-clearing price, it’s fine. But in the sense of its price movements being informative...well. Say the price of that security has just dropped by 10%.
Is the market reflecting that bad news about Google’s new AI model is likely to reflect poor long-term prospects? Is it indicating that increased regulatory scrutiny is likely to be bad for Google’s profitability?
I mean, I feel like random things affect the price of securities all the time. During early COVID random fiscal policy decisions had a much bigger effect on the stock price of companies than their actual competence. Similarly, COVID itself of course had huge effects.
I feel like it’s normal that when the stock price of a company moves, this often has little to do with the company, but can be chased back to kind of “random” other things. In this case, the stock price would go down, and it would be pretty easy to check whether that was because something related to the resolution criteria changed, or whether something “core” to the company changed.
Most Polymarket markets resolve neatly, I’d also estimate <5% contentious.
For myself, and I’d guess many LW users, the AI-related questions on Manifold and Metaculus are of particular interest though, and these are a lot worse. My guesses as to the state of affairs there:
33% of AI-related questions on Metaculus having significant ambiguity (shifting my credence by >10%).
66% of AI-related questions on Manifold having significant ambiguity
For example, most AI benchmarking questions do not specify whether or not they allow things like N-trajectory majority vote or web search. And, most of the ambiguities I’m thinking of are worse than this.
On AI, I expect bringing down the ambiguity rate by a factor of 2 would be quite easy, but getting to 5% sounds hard. I wrote up my suggestions for Manifold here a few days ago. For Metaculus, I think they’d benefit from having a dedicated AI-benchmarking mod who is familiar with common ambiguities in that area (they might already have one, but they should be assigned by default).
I find the datapoint interesting, but I don’t really why it’s much evidence of this being “a huge issue”. Most markets resolve neatly. It seems fine for 5% of markets or so to end up in dispute. People can price in a bunch of the resolution uncertainty. Do we have any evidence this is seriously hurting prediction market adoption or performance?
I agree that most markets resolve successfully, but think we might not be on the same page on how big a deal it is for 5% of markets to end up ambiguous.
If someone offered you a security with 95% odds to track Google stock performance and 5% odds to instead track how many hairs were on Sundar Pichai’s head, this would not be a great security! A stock market that worked like that would not be a great stock market!
In particular:
I think this ambiguity is a massive blow to arbitrage strategies (which are a big part of the financial infrastructure we’re hoping will make prediction markets accurate). There are already a lot of barriers in the way of profiting from a situation where e.g. one market says 70% and one market says 80%: if there’s a chance that the two will resolve some ambiguity differently, that adds a very big risk to anyone attempting to arbitrage that difference.[1]
I think this ambiguity is very dangerous to the hopes of prediction markets as a trustworthy canonical source on controversial events. If Maduro says “See, everyone, I won the election fair and square, the prediction markets agree!”[2], I think it’s very important that the prediction markets be perfectly clear on what they are tracking and what they are not tracking, and I don’t think that’s currently the case.
Related to #2, I think that this ambiguity is vastly more likely to occur in the case of the controversial events where it’s most valuable to have a trustworthy and canonical source. The cases where markets resolve cleanly are exactly the cases where non-prediction-market mechanisms already reach a canonical consensus on their own.
This also makes Manifold’s preferred strategy of dealing with ambiguity by N/A-ing a market less valuable: that’s an acceptable resolution for someone who just did buy-and-hold on that one market, but can be very bad for someone who was trading actively across multiple markets some of which N/A-ed and some of which did not.
This imagines a world where prediction markets are major enough and mainstream enough for people to be looking at them and talking about them: but that’s exactly what prediction market advocates want!
I disagree with the 5% of switching to a Sundar Pichai hairs simile:
Prediction market prices are bounded between 0 and 1
Polymarket has > 1k markets, and maybe 3 to 10 ambiguous resolutions a year. It’s more like 0.3% to 1%.
My sense is this security would be fine? Is there a big issue with this being a security?
In most domains except the most hardened part of the stock market counterparty risk is generally >5%. The key issues come when failure is correlated, but it seems to me indeed that in prediction markets it’s pretty random which way ambiguity resolves, and so you get pretty uncorrelated failures (like, if you are invested in 10,000 markets, while it might be the case that 500 of them resolve in a surprising and ambiguous way, you will pretty randomly be on the losing or winning side of it, so it mostly just cancels out).
This seems quite wrong to me:
High Yield Corporate Bond OAS spreads are <5% according to bloomberg, and most of that is economic risk, not “you will get screwed by a change of rules” risk.
Trades on US stock exchanges almost always succeed, many more 9s than just one.
If I buy a product in a box in a supermarket the contents of the box match the label >>95% of the time.
Banks make errors with depositor balances <<5% of the time.
Most employers manage to pay fortnightly wages on time without missing one or more paycheques per year.
Once you’re seated in an Uber or Taxi they take you to your destination almost all the time.
Your utility company fulfills its obligations to supply your house >>95% of the time under all but the most extreme circumstances.
Most employees turn up >95% of non-holiday days, and most students maintain >95% attendance.
Sorry, I wanted to say “except the most hardened parts of the world (like the stock market)”. I agree with you that basically anything in the stock market has much less counterparty risk than that. I disagree with basically all non-trading examples you give.
My sense is around 1⁄20 Ubers don’t show up, or if they show up, fail to do their job in some pretty obvious and clear way.
True for the most commoditized products. For anything else error rates seem to me to be around 5%. My guess is my overall Amazon error rate has been around 2%, which is lower, but not much lower (usually Amazon sent me something broken that previously was returned where they couldn’t spot the error).
I think that’s false, at least the statistics on wage theft seemed quite substantial to me. I am kind of confused how to interpret these, but various different studies on Wikipedia suggest wage theft on-average to be around 5%-15% (higher among lower-income workers).
I agree this is true for gas and water (and mostly true for electricity, though PG&E is terrible and Berkeley really has a lot of outages).
Overall, I think 5% counterparty risk seems about right for most contracts I sign or business relationship I have. I agree that trading infrastructure is quite robust and in highly commoditized environments you get below that, but that’s not the majority of my economic transactions.
It’s not just the stock market, it’s true for the bond market, the derivatives market, the commodities market… financial markets, a category which includes prediction markets, cannot function effectively with counterparty risk anything like 5%.
If the Uber doesn’t show up I’m not sure that’s counterparty risk: you haven’t paid anything, so it seems more like them declining the contract. The equivalent for a prediction market would be if you hit ‘buy’ and the button didn’t work, not for when you have paid the money and then don’t get the result taken from you. That’s much less bad than if the trade went through and then was settled incorrectly.
I think those studies have significant methodological flaws, though unfortunately I can’t remember the specific issues off the top off my head, so this may not be very convincing to you.
According to the first google hit, PG&E said the average customer suffered 255.9 minutes of outage in 2013, which is a lot higher than I expected, but is still only 100*255.9/(60*24*365) = 0.05%
Hmm, maybe I am just failing to model something here. Isn’t really the only thing that happens when you have 5% randomly-distributed counterparty risk that you end up with like 5% spreads? That seems fine to me.
To be clear, I don’t feel very confident here, I just don’t really understand why you can’t just price in counterparty risk and then maybe end up with some bigger spreads (which I do agree is sad for prediction markets, but for most markets I don’t mind the spread that much).
In the sense that it would find a market-clearing price, it’s fine. But in the sense of its price movements being informative...well. Say the price of that security has just dropped by 10%.
Is the market reflecting that bad news about Google’s new AI model is likely to reflect poor long-term prospects? Is it indicating that increased regulatory scrutiny is likely to be bad for Google’s profitability?
Or is Sundar Pichai going bald?
I mean, I feel like random things affect the price of securities all the time. During early COVID random fiscal policy decisions had a much bigger effect on the stock price of companies than their actual competence. Similarly, COVID itself of course had huge effects.
I feel like it’s normal that when the stock price of a company moves, this often has little to do with the company, but can be chased back to kind of “random” other things. In this case, the stock price would go down, and it would be pretty easy to check whether that was because something related to the resolution criteria changed, or whether something “core” to the company changed.
Most Polymarket markets resolve neatly, I’d also estimate <5% contentious.
For myself, and I’d guess many LW users, the AI-related questions on Manifold and Metaculus are of particular interest though, and these are a lot worse. My guesses as to the state of affairs there:
33% of AI-related questions on Metaculus having significant ambiguity (shifting my credence by >10%).
66% of AI-related questions on Manifold having significant ambiguity
For example, most AI benchmarking questions do not specify whether or not they allow things like N-trajectory majority vote or web search. And, most of the ambiguities I’m thinking of are worse than this.
On AI, I expect bringing down the ambiguity rate by a factor of 2 would be quite easy, but getting to 5% sounds hard. I wrote up my suggestions for Manifold here a few days ago. For Metaculus, I think they’d benefit from having a dedicated AI-benchmarking mod who is familiar with common ambiguities in that area (they might already have one, but they should be assigned by default).