The latest episode of Hard Fork has a 20 min section dedicated to prediction markets! Kevin Roose and Casey Newton go into much more depth in this podcast than in the earlier NYT article, covering the history of prediction markets, the rationalist movement, Google’s internal markets, insider trading, etc. They even talk about embedding prediction markets in the New York Times!
You can listen to the full podcast here (the segment runs from 29:30 to 48:15); a transcript is available there too. Some highlights:
Excerpts
Manifest, LK99 and Manifold
Casey Newton
Kevin, you did some great reporting this week about prediction markets, and it all started at something called Manifest, which — I read that and I thought, well, is this just a festival for men? Tell us about what manifest is.
Kevin Roose
[LAUGHS]: So this is a field trip that I had been planning for a while. This was a very fun and interesting reporting trip to a conference for what they call forecasting nerds, so people who like to predict the future and bet on the future. And this was actually something that came out of an episode that we did several months ago about LK-99. Do you remember this episode?
Casey Newton
Yes, of course.
Kevin Roose
So this was the room-temperature superconductor that a group of scientists in South Korea had claimed to have come up with. And there was this period of maybe a week or two where people were hotly debating whether this was real or not. And we mentioned on the show the existence of something called Manifold Markets, which is a prediction-markets platform where people can go and wager fake money on real-world events. And one of the most popular markets was about LK-99. And it was a way to track, like, what the smart-money people thought was going to happen and whether this prediction of a room-temperature superconductor would pan out. Now, it did not, right? LK-99 did not turn out to be a room-temperature superconductor. But I heard from one of the founders of Manifold Markets who said, if you’re interested in prediction markets, we’re actually having a big conference in a few weeks in Berkeley called Manifest, and you should come report on it. And I thought, well, that sounds like a fun trip.
Casey Newton
Yeah, I’d actually predicted that you were going to go to that, so that was interesting. [KEVIN LAUGHS] So you get there. And sort of describe the scene. Because what you’ve described — I’ll say it — sounds a little bit dull, but then I read your story, and it actually seemed like it might be a good time.
Kevin Roose
Yeah, it was a very strange event. And I say that — I had a good time and I learned a lot, but it was definitely not what I was expecting. I was expecting, like, a sort of statistics conference where people in dress shirts and Dockers would be going around, like, comparing their predictive models of the world. But it was more like a party than I thought. I described it in the article as sort of a cross between a math Olympiad and Burning Man. Like, there was actually an orgy at this conference. And I know that because there was also a prediction market asking whether or not there was going to be an orgy. I think when I got there, it was, like, 28 percent possibility, and by the time I left, someone had had an orgy and closed out the market.
History of prediction markets
Kevin Roose
When we talked about it on the podcast in the context of LK-99, I believe we made some snarky comments about, oh, these are just gamblers who like to bet on everything.
Casey Newton
Yeah.
Kevin Roose
But I would say, after going to this Manifest conference, there’s also a real movement that I think is worth paying attention to here. Prediction markets — this is not a new idea, right? People have been betting on things like elections for centuries. Actually, in the 19th and early-20th centuries, it was common to open up the newspaper and see a sort of betting-odds breakdown of who people thought was going to win the next election.
Casey Newton
And I feel like my entire life, I’ve been hearing about the wisdom of crowds.
Kevin Roose
Yes. That was a very popular idea. This idea of prediction markets was sort of revived in the 1990s by a group of economists who thought, well, markets collect information. You can bet on the price of a company’s stock, or you can bet on corn futures — what will the price of corn be a year from now? You can also bet on sports games. Why can’t you bet on other things? Why can’t you bet about scientific discoveries? Why can’t you bet about policy implementation? Why can’t you bet about silly things like whether there’s going to be an orgy at a statistics conference?
On Rationalists
So there’s been sort of a real resurgence in the last few years, led by this group of people called the Rationalists. Do much about the Rationalists?
Casey Newton
A lot of it from reading your reporting, but yeah, tell us a little bit more about the Rationalists.
Kevin Roose
So Rationalists are a sort of loose collective of people who are sort of committed to examining their own beliefs. They want to get closer to the truth. Big figures in the movement are people like the guy who runs this blog, “Astral Codex Ten,” which used to be known as “Slate Star Codex.” Eliezer Yudkowsky is sort of an AI safety researcher and a prominent Rationalist blogger who started a website called “LessWrong.” So there’s a crew of people, largely based in the Bay Area but also spread out throughout the world, who are sort of doing what they would describe as rigorous empirical testing of everything that they believe and do. They love attaching probabilities to things. So I want to sketch out the vision for what they believe prediction markets could do. Because they’re not just saying like this could be a way to make money by betting right on things. They’re saying, if you have everyone betting on everything, then you end up with a system where people are incentivized to understand the truth.
Google’s internal prediction markets
But interestingly, I was told that this is not a new idea to the world of tech, that actually, Google has run its own prediction markets internally for employees. So if you worked at Google, you could bet —
Casey Newton
On whether it would become a search monopoly.
Kevin Roose
[LAUGHS]: Yeah, and everyone who made that bet got paid. No, it was — they used a fake currency called googles, and you could bet on things like, will this project launch in time, or will Gmail get to this many users by this date. And the company’s leaders use this as a way to gauge employee sentiment. And sort of, when people could bet anonymously, they could actually get people’s true opinions.
Casey Newton
Wait, that’s fascinating. And do they not do this anymore?
Kevin Roose
Well, they were playing around with this as recently as 2021. And there were people I talked to at Manifest who believed that this is ultimately how all companies should run. Like, you get a job at a company and a prediction market opens up that says, in a year, will Casey Newton be more or less successful than we expect him to? And everyone in the company would bet on whether they think you will succeed at your job or not. And over time, you would essentially see who is the best at forecasting people’s performance. And you could put them in charge of your hiring process.
Casey Newton
That sounds so unbelievably stressful. [KEVIN LAUGHS] By the way, if you are really good at the Google predictions market, and so you had more googles than anyone else, and then they shut down the market and you were stuck with all these googles you couldn’t use, we want to hear from you. I think that would make for a good story.
Insider trading
Well, the Google story is really interesting. But it brings up something else I want to ask you about, which is the potential for people to manipulate these markets, right? You set up a market, and then you either have some insider information, or you just sort of try really hard to make the thing happen.
Kevin Roose
Yes, I actually saw insider trading happen. Because I was interviewing someone at this conference, and he pulled out his phone, and he showed me a prediction market that had been placed on Manifold about whether “The New York Times” would cover Manifold in an article in the year 2023. And as I was talking to him for this article, he was placing a large bet on yes on that market with his insider information, which is that a reporter from “The New York Times” was, in fact, interviewing him for an article.
Casey Newton
And how does the platform view that? Is it just sort of, well, all’s fair in love and prediction markets?
Kevin Roose
So they actually think that insider information and insider trading is good. Because people with inside information have the best information, and they can bring it to a market. They have some sort of elaborate theoretical underpinning for not believing that insider trading actually should be illegal. Right now, all of this is play money, right? Because of our gambling laws in the US, there are a couple sort of small real-money prediction markets that are very limited, and it’s not worth going into why. But most forms of real-money prediction markets are not legal in the US.
Casey Newton
Right. And at Manifold, they use something called mana as the virtual currency, which is also the same currency that you use to cast spells and Magic the Gathering and Diablo, so you’re going to want to manage your resources wisely.
Kevin Roose
Exactly. So this is this play currency called mana that you can use on the platform. They have leaderboards for who’s got the most mana. You can also convert it into charity donations. They are not allowed to pay out real money for people who are right on these gambles. And the people at this conference were upset about that. They think this should be legal. I have some concerns about that. I just don’t know what it would look like for a society to be gambling on everything all the time. But they are of the mind that the benefits of legalizing this kind of prediction market would outweigh the costs.
Skepticism of “skin in the game”
Casey Newton
You know, I got to say, Kevin, I’m of really mixed mind about this. Because on one hand, the idea of people, like, betting play money to guess what might happen seems totally innocuous. Have a good time. It seems like you had a great time at this conference, seemed like all the other people who were there did, too. But I start to hear things like, well, these folks think that insider trading should be legal, and I just start to think, keep them away from the real economy. And this whole idea that we make better decisions when we have skin in the game, I just think, has been really challenged over the past few years, right? Like, this was one of the big arguments for crypto. And crypto is the place where we used to hear all the time, you got to have skin in the game. I use — we were told for years, you can make a better social network if you have skin in the game. Right? You can develop a better relationship between musicians and fans if the fans have skin in the game. The whole idea of the Bored Ape Yacht Club was, give people skin in the game and they’ll be able to make movies. And it all just kind of came to nothing. And one of the reasons was that when you give people skin in the game and everything just has this, like, gross economic incentive tied to it, it just changes behavior. And people start, kind of, behaving in antisocial ways. So what do you think about the value of people having skin in the game? And is it possible that they’re overstating the benefits here?
Prediction markets in the New York Times?
Kevin Roose
One application of this that I actually think is kind of interesting would be, in our industry, in media — I had a conversation with the guy who runs Astral Codex Ten, and one thing that he was saying is like, if “The New York Times” put little prediction-market things at the bottom of articles, for example, that might give readers a better sense of what the probabilities behind the news events that they’re reading about are. So you could have an article about who will be the next speaker of the House, and then at the bottom of the article, you could have a little widget that sort of gave you the prediction market for someone specific or sort of an indicator of where the betting odds were on various people. And that might actually help you come to a better conclusion than just reading the article alone. Do you think that makes any sense?
Casey Newton
Yeah, I think you subscribe to “The New York Times,” and you’re given a certain amount of New York toobles, and then you sort of bet your toobles on who will be the next House speaker and — I mean, what that makes me think of is the way that polls would be gamed on Twitter in the heyday, right? People would say, like, hey, do you think this thing is going to happen? And then, it would get gamed as the most zealous partisans would stuff the ballot box until the poll was over. And I wonder what mechanisms might be put in place to prevent something like that from happening here.
But on balance, I’m persuaded that this is an interesting technology. And one thing that I have just observed in moving through Silicon Valley is that you do just constantly meet people who are into prediction markets. You know, it’s like, along with poker, these are the two preferred forms of gambling and, increasingly, ways of socializing here in our strange little corner.
Calling pundits to account
Kevin Roose
Totally. I mean, the first place I saw this take off was among AI researchers, who love to bet on, for example, what year we will get AGI or, like, when the first AI-generated screenplay will win an Oscar and things like that. And so they really are sort of running prediction models in their heads at all times. There’s this sort of cohort of people who are very into what they call Bayesian analysis or, like, attaching probabilities to things and living their lives that way. Do I think that is the way most people live their lives? Absolutely not. But it is sort of an interesting idea. And as someone who makes predictions sometimes as part of my job, it’s interesting to contemplate a world in which your position as a pundit or a columnist or a newsletter writer would be quantifiable in some way. Like, readers of “Platformer” could go in and say, OK, Casey’s predictions were 75 percent right last year, so I’m going to trust him more. But if his predictions fall to only 50 percent right next year, maybe I’ll cancel my subscription.
Casey Newton
And, like, by the way, how amazing would it be to have the pundit score that sort of said this person predicted these 50 concrete things in the past year and 12 of them happened? That feels like — that feels like the sort of information that the right person would sue to get taken off of Google.
I work at Manifold. I think it’s notable that these two experienced tech journalists have had lots of repeated exposure to the idea of prediction markets, but it sounds like they only sort of figured out the basic concept?
In the discussion on insider trading, nobody mentions the extremely obvious point, which is that the prediction market is trying to incentivize the people with private information (maybe “insider” information, or maybe just something they haven’t said out loud) to publicize what they know. If Casey actually cares about whether Linda Yaccarino will be the CEO of X next year, he should be excited by the idea that some guy at Twitter will come and insider trade in his market. But they never said anything like this—they just said that maybe the market was supposed to very generally aggregate the wisdom of crowds.
It also sounds like they don’t really understand why it would aggregate the wisdom of crowds better than, for example, a poll. Casey was like “well, when people have a Twitter poll, then partisans stuff the ballot box”, implying that a similar result would be likely to happen with a prediction market on who will be the next Speaker, ignoring the obvious point that it costs a bunch of money to “stuff the ballot box” on a prediction market that isn’t a self-fulfilling prophecy.
Perhaps relatedly, it sounded like Kevin and/or Casey had absolutely no clue how a prediction market actually works, numerically. At the end when they were making the market, Casey wasn’t like “OK, bet it to 25%, since I think that’s the chance.” Instead Kevin was like “OK, I’ll bet 100 mana,” and then they were like “Huh, how about that, now it says 10%. Oops, I bet 100 more and now it says 8%.” It seems like they are totally missing the core concept that the point of the prediction market is trying to specifically incentivize you to move the market to the probability you believe, which is like the first thing I ever learned about prediction markets in my life?
In the end, their feelings about prediction markets seemed totally vague and vibes-based. On the one hand, the wisdom of crowds has good vibes. On the other hand, insider trading and crypto/transactionalization of everyday things have bad vibes. On the gripping hand, gambling with play money is cute and harmless. Therefore, prediction markets are a land of contrasts.
My takeaway is that prediction markets are harder to understand than I think and I am not sure what to do about that.
Do you mean “closer to the probability you believe”? Moving a liquid prediction market right to the probability you believe involves spending a bunch of capital in an inefficient way—your last dollar spent on such a trade gets almost no expected return. If you have many markets available, or are risk averse, or have a limited budget, you will not behave this way.
Haha, this actually seems normal and fine. We who work on prediction markets, understand the nuances and implementation of these markets (what it means in mathematical terms when a market says 25%). And Kevin and Casey haven’t quite gotten it yet, based on a couple of days of talking to prediction markets enthusiasts.
But that’s okay! Ideas are actually super hard to understand by explanation, and much easier to understand by experience (aka trial and error). My sense is that if Kevin follows up and bets on a few other markets, he’d start to wonder “hm, why did I get M100 for winning this market but only M50 on that one?” and then learn that the odds at which you place the bet actually matter. This principle underpins the idea of Manifold—you can argue all day about whether prediction markets are good for X or Y, or… you can try using them with play money and find out.
It’s reasonable for their reporting to be vibes-based for now—so long as they are reasonably accurate in characterizing the vibes, it sets the stage for other people to explore Manifold or other prediction markets.
Never mind bettors—part of my project for improving the world is, I want people like Casey to look at a prediction market and be like, “Oh, a prediction market. I take this probability seriously, because if it was obviously wrong, someone could come in and make money by fixing it, and then it would be right.” If he doesn’t understand that line of argument, then indeed, why is Casey ever going to take the probability any more seriously than a Twitter poll?
I feel like right now he might have the vibe of that argument, even if he doesn’t actually understand it? But I think you have to really comprehend the argument before you will take the prediction market more seriously than your own uninformed feeling about the topic, or your colleague’s opinion, or one research paper you skimmed.
If it’s known that insider trading is allowed, wouldn’t that make some people more reluctant to bet? Would this reduce the liquidity of the market enough to be damaging overall?
Zvi mentions the insider trading problem in this post.