Oh interesting, that’s news to me if true, but could be the case! I’ll have to ask Sinclair.
David Chee
Karma: 143
Manifold Predicted the AI Extinction Statement and CAIS Wanted it Deleted
How well did Manifold predict GPT-4?
FYI the creator of the market decided to resolve the market to N/A.
This means everyone’s mana is fully restored as if they never interacted with the market.
David here from Manifold.
Really cool to see you taking the time to present your thoughts in a way that is constructive whilst not holding back with your criticisms. A lot of people would have just gotten frustrated with their experience on the site and have just moved on, so thank you for this post! We definitely think about these things a lot as we design Manifold and this post could definitely have an impact on where we lean within tough design problems.
First off, let me briefly discuss why we are bullish on using a creator-based resolution mechanism.
Creator reputation. This is definitely the most important point. Regular users quickly learn which markets are run by credible creators who know to set clear criteria. And in the rare instance an unexpected variable occurs making the resolution less clear, these creators will often defer to what the comments agree should be the correct resolution. Admittedly this is definitely not obvious for new users who are interacting with our markets for the first time and are often times trading in good faith that all the markets are reliable. We are aiming to get a reputation system set up at some point, but we haven’t had the capacity to do so yet
Creator resolution facilitates types of questions that otherwise couldn’t exist.
Fun, personal markets that often only the creator will know the results. eg. this one from James Medlock.
Creators resolving works >95% of the time. For the majority of markets, there are no issues. Very rarely does a creator’s internal bias or ulterior motive affect the resolution. It’s unfortunate that the first market you interacted with may be one of the minority and we will be more conscious about ensuring big/featured markets are reliable. I strongly believe if you were to use the site more that you would find a lot of your concerns are exacerbated by your experience with an outlier.
Interpretation of a result is often necessary. As others have alluded to in the comments, there is often conflict between a market that is useful and answers what you want to know vs one that is extremely well-defined in advance. It will always be the case that questions come up where someone has to make a call and it makes sense that the person creating the market is the best person to do so. We have heard some good suggestions regarding crowd-sourcing resolutions so might be open to possible types of implementations of that.
Now that that’s out of the way, let’s discuss some of the specific criticisms you outlined. The following is more a reflection of my own thoughts and hasn’t necessarily been discussed in depth with the team.
The market probability becomes more reflective of the creator’s tendencies and bias and not the event itself.
I will agree that this is definitely true with some of our markets. But I actually think that is okay! One thing that sets Manifold apart from other prediction markets is that we support a wide range of markets that is only possible by giving creators full autonomy. I think it is a good thing that each individual creator is expressed in their market. Not all markets have to provide actionable, informative data even if this may not align with what is traditionally sought after with forecasting. Just to be clear though, one of our primary goals is to facilitate markets that provide actionable and informative discussion and data. But, I suspect as the site grows that it will become more obvious what the intent of each individual market is. There very well could be multiple instances of the same market which will actually have different probabilities based on who is the creator and subtle differences in resolution criteria.
Markets on Manifold are too ambiguous.
I don’t have too much to say on this as I agree with you that our markets are generally too ambiguous and what Jack said covers a lot of my own thoughts. We do encourage creators to try and be as clear as possible, but many people are still learning how to create questions. One of the main purposes of the site is to lower the entry-barrier to prediction markets and allow anyone to create one easily so it isn’t a surprise a large number of markets aren’t that great or clear. Fortunately, there is already a clear culture developing where users will comment when they see unambiguity or a possible loophole in a market. And I think you would find that our big serious markets tend to be fairly unambiguous at this point.
Manifold is untrustworthy.
I’m not sure I agree with this but that may definitely be my bias haha. I do think though that users should regard every market they bet on with mild scepticism and not default to trusting the creator. Now, as a site is it optimal to have a bunch of unreliable content? Maybe not, particularly for the new user experience and I can empathise with why this may cause you to mistrust the site. But it’s unrealistic to expect a site that gives its users full autonomy and does minimal curation to have only trustworthy content. But it is something we will need to think about more. Hopefully, it will be mostly solved once we introduce a reputation system of some sort. And from my experience, the number of markets that have been “untrustworthy” can be counted on one hand so far.
Thanks again for taking the time to try out our site (even if it didn’t go as you expected) and for doing this write-up. Am happy to discuss things further. Maybe what you’ve outlined is a bigger problem than we’ve realised and we need to spend a lot more effort designing the site around it, but I’m not yet convinced. It is good to know though that it is turning some users away.
It’s true that markets involving existential risk aren’t nearly as reliable as other markets. Perhaps I should have shown more care when using one as an example.
Ultimately for a market to be efficient, users have to be incentivised to trade to the best of their ability. Whilst the AI risk markets aren’t going to be particularly accurate, I do think that the one I linked is reasonably representative of our user’s beliefs. There is a good amount of upward and downward buying pressure on either side. My main concern with its accuracy is how many Yes shares Catnee has, however, the market does seem fairly stable at 25% and isn’t being held there with huge limit orders or anything like that.
One of the advantages of play money is that the incentive for truth-seeking can actually outweigh the desire to profit. This sort of market would obviously never work if real money was on the line!