Yep! I’ve been thinking about this quite a bit lately. May write another post later.
It seems to me that we make social progress by imposing a competitive or game-like structure on meandering human activity. This gives it direction and allows judgment of expertise by a uniform set of standards.
This might be a strategy for the “move fast and break things” approach to political activism.
For example, imagine the town of Libertyville is considering whether to take on debt to pay for connecting itself to the sewage system of the nearby metropolis, so that a Walmart will be able to be built.
Some people think this is bad for the local economy; they don’t want to take on debt, and they think the Walmart will destroy local businesses. Others think it’s good for the local economy because of the jobs it will bring.
Currently, everybody will debate this, and then they’ll vote and see what happens. The valuable thing for the whole community that is lost is a record of who thought what, who changed their minds and how, and who had the greatest insight.
Repeat over and over again for many other decisions. What’s happening when we don’t do forecasting tournaments is we waste a tremendous amount of information about who in our communities is most politically engaged, and who has good judgment. Forecasting tournaments have an opportunity to preserve this information in a distilled form that we can use to make communities stronger.
If Berg wanted to do better, they could organize such a tournament in which they ask each side in the debate to specify the most significant verifiable impacts a bill would have, year by year, after it was passed. Forecasters vote on their predicted outcomes for both results; each forecaster is graded on the forecasts they made for the bill chosen. For example, all forecasters might vote on:
If the bill is passed:
a) Construction on a Walmart will begin
b) Construction on a Walmart will be complete
c) Population will increase by at least 2%
d) At least three currently registered businesses will go out of business within two years
e) At least 5 new businesses in the city limits will open up within two years.
If the bill is not passed:
a) Population will decrease
d) At least three currently registered businesses will go out of business within two years
e) At least 5 new businesses in the city limits will open up within two years.
I imagine that at first, the forecast results would be kind of useless. It would just be blabbing and partisan cheerleading. People might rope their friends into making thoughtless “forecasts” just to up the numbers for their preferred side.
But over time, forecasters would accumulate a track record. The overall community forecast could be weighted by that track record. Perhaps newer people have something equivalent to “lightweight credit” and consistently bad forecasters have “heavy, poor credit.”
The forecasts wouldn’t displace voting. But they would impose some beneficial structure into the city-wide political conversation, shape people’s conversation in a way that might be helpful, and influence votes by creating some record of the thoughts and track records of the best forecasters.
I would be absolutely delighted if such a competition existed in my city. It seems like it could be set up amongst a small group of avid forecasters without asking anybody for permission, grow by invitation, and naturally come to be a fixture of local politics.
In order to facilitate that, I think it would be wonderful if there was a website or app in which forecasting competitions could be created.
Why don’t the media make pundits and politicians go on record and challenge or ignore them when they get things wrong? One theory might be they aren’t interested in informing as much as getting ratings and accuracy isn’t a big part of that. Another more conspiracy minded theory might be that they are interesting in driving a certain narrative.
It’s even more interesting to think about why companies don’t use prediction markets. Seems like a company would be more profitable if they had the best predictors....so why aren’t they doing more to attract them? Robin Hanson has, what I think, are some great thoughts on this.
Again, agree with everything you’ve said in this post, but I don’t feel we have a great theory on why people don’t take forecasting more seriously and until we do are unlikely to increase their use.
In reality, many things happen at the same time. It is difficult to give a prediction of what X will achieve, if you don’t know the outcomes of Y and Z, which will be decided tomorrow. Perhaps the decision on Y and Z will depend on the outcome of votes for X.
For example, imagine making a prediction about a future of a startup that breaks some regulations, such as Uber. (Depending on your prediction, citizens will vote whether to invite Uber into your city.) Well, the prediction depends on how strongly the breaking of regulations will be prosecuted and punished. Suppose your political opponents have a strong influence on prosecutors and judges. Now, whatever prediction about the future of Uber you make, they can make the opposite happen. It would cost them some political points, but maybe it is worth doing for them, if it results in hugely discrediting you and eliminating you from the competition.
Now not only can your enemies make your project fail, they can also make you seem like a bad predictor if that happens. And if your prediction contains hundreds of “if”s, no one will take it seriously. “My project will increase the GDP of our community by 2% withing 5 years, assuming the government will not do something else that decreases the GDP, nor change the way GDP is measured, nor adopt new laws that make parts of my project illegal, nor change the tax code to make my project more expensive, nor … organize a Twitter mob that would attack people participating in my project, nor ….”
But thats true of pretty much every prediction market. Unless you’re opposed to prediction markets very broadly, I’m not sure this changes anything I’ve said.
prediction market + future depends on the voters’ actions = moral hazard
I guess this is an argument against prediction markets in general, unless the results are independent on what voters do (or a voter activity aimed to change the outcome is prosecuted as a fraud).
We can see this in politics. The Blue Party says that a major Green Party policy will fail. When the Green Party enacts it, the Blues sabotage it, then use its failure as evidence that Blue ideology is correct.
Likewise, partisan pollsters run polls designed to exaggerate support for their preferred candidate, under the theory that predictions of success can cause success.
It also seems to me that any form of prediction market has moral hazard. They’ve been criticized as assassination markets, and any questions that have anything to do with the behavior of an individual person set up an incentive for murder.
Hence, the key question is not whether moral hazard exists, but whether the tradeoffs are acceptable.
It’s hard to say if the entire world should be run by prediction markets. But I think we should be experimenting more with forecasting and prediction markets at more local levels.
The Blue Party says that a major Green Party policy will fail. When the Green Party enacts it, the Blues sabotage it, then use its failure as evidence that Blue ideology is correct.
Can you give an example? What is the difference between “sabotaging” a policy and simply opposing a policy you don’t support?
Sure. Republicans couldn’t afford to repeal the Affordable Care Act (I’m assuming you’re American, let me know if not and I’ll explain further), because it was too popular even among republicans.
One of the gears that makes it work is the individual mandate, a tax penalty for not having health insurance. This penalty was meant to incentivize young healthy people to sign up for health insurance, effectively subsidizing the insurance policies of the old and sick. The Republicans repealed this component, which should have the effect of gutting the revenue stream that makes the ACA work, driving up the cost of insurance.
Opposing the policy would look like Republicans repealing the ACA. Sabotaging it looks like keeping it in place, but repealing a tax penalty that effectively funds it, making it untenable in the long run as insurance prices spike. They can blame this on Democrats, arguing that a big government insurance program was bound to fail, while taking credit for the tax cut.
The dynamic I’m describing is meant only to illustrate the difference between political opposition and sabotage, not a serious piece of contemporary political analysis.
To define them more generally, political opposition is about making the consequences of your opposing actions clear.
Political sabotage is hiding the consequences of your opposing actions.
Well, in politics, you’re dealing with an issue where enforcing compliance would be difficult. It’s famously hard to get a politician to give a clear answer to a clear question.
I think that partly, the idea of forecasting as practiced by Tetlock et al is just relatively new. It hasn’t seeped into society to the extent that it might someday. Companies probably need to be of a certain size to make such tournaments seem worthwhile.
But I wouldn’t be surprised if some companies do in fact run internal forecasting competitions or prediction markets. The Pentagon tried running a prediction market to identify security threats, but shut it down when it was criticized as as “terrorism futures market.”
My particular interest is not the macro question of “why isn’t this done more,” but the micro question of “what’s the easiest way that an individual or small group could start a forecasting competition?”
One way is to build better prediction markets. If we would have better prediction markets that move a decent chunk of money those prediction could be used.
Another is to think through the application of forcasting in individual fields. In my post on cancer I for example wrote about Prediction-Based Medicine for compassionate use. While it’s likely impossible to convince people of Prediction-Based Medicine for those decisions that are currently allegedly made via Evidence-Based Medicine campaigning for it being used for in cases where compassionate use is currently done without any need for any evidence might be more promising.
On the one hand you have people who don’t like that doctors can promise patients whatever they want in cases of compassionate use and on the other hand you have people who find compassionate use important because it allows patients to get life saving drugs before they are approved by the FDA. Prediction-Based Medicine looks to me like a great compromize between the two sides.
When it comes to computer programming on tool I would like to see is a program that asks you to make a prediction whenever you run your unit test about whether or not those unit tests will fail.
There are a lot of cases of specific expertise where it’s worth thinking about how to build forcasting based systems for them.
When it comes to journalism, a new way to do journalism can be invented that works on the Blockchain. It’s possible to raise sizeable amounts of capital in ICO’s if you have a well thought out idea.
Yep! I’ve been thinking about this quite a bit lately. May write another post later.
It seems to me that we make social progress by imposing a competitive or game-like structure on meandering human activity. This gives it direction and allows judgment of expertise by a uniform set of standards.
This might be a strategy for the “move fast and break things” approach to political activism.
For example, imagine the town of Libertyville is considering whether to take on debt to pay for connecting itself to the sewage system of the nearby metropolis, so that a Walmart will be able to be built.
Some people think this is bad for the local economy; they don’t want to take on debt, and they think the Walmart will destroy local businesses. Others think it’s good for the local economy because of the jobs it will bring.
Currently, everybody will debate this, and then they’ll vote and see what happens. The valuable thing for the whole community that is lost is a record of who thought what, who changed their minds and how, and who had the greatest insight.
Repeat over and over again for many other decisions. What’s happening when we don’t do forecasting tournaments is we waste a tremendous amount of information about who in our communities is most politically engaged, and who has good judgment. Forecasting tournaments have an opportunity to preserve this information in a distilled form that we can use to make communities stronger.
If Berg wanted to do better, they could organize such a tournament in which they ask each side in the debate to specify the most significant verifiable impacts a bill would have, year by year, after it was passed. Forecasters vote on their predicted outcomes for both results; each forecaster is graded on the forecasts they made for the bill chosen. For example, all forecasters might vote on:
If the bill is passed:
a) Construction on a Walmart will begin
b) Construction on a Walmart will be complete
c) Population will increase by at least 2%
d) At least three currently registered businesses will go out of business within two years
e) At least 5 new businesses in the city limits will open up within two years.
If the bill is not passed:
a) Population will decrease
d) At least three currently registered businesses will go out of business within two years
e) At least 5 new businesses in the city limits will open up within two years.
I imagine that at first, the forecast results would be kind of useless. It would just be blabbing and partisan cheerleading. People might rope their friends into making thoughtless “forecasts” just to up the numbers for their preferred side.
But over time, forecasters would accumulate a track record. The overall community forecast could be weighted by that track record. Perhaps newer people have something equivalent to “lightweight credit” and consistently bad forecasters have “heavy, poor credit.”
The forecasts wouldn’t displace voting. But they would impose some beneficial structure into the city-wide political conversation, shape people’s conversation in a way that might be helpful, and influence votes by creating some record of the thoughts and track records of the best forecasters.
I would be absolutely delighted if such a competition existed in my city. It seems like it could be set up amongst a small group of avid forecasters without asking anybody for permission, grow by invitation, and naturally come to be a fixture of local politics.
In order to facilitate that, I think it would be wonderful if there was a website or app in which forecasting competitions could be created.
I agree with all that.
My question is why isn’t it already being done?
Why don’t the media make pundits and politicians go on record and challenge or ignore them when they get things wrong? One theory might be they aren’t interested in informing as much as getting ratings and accuracy isn’t a big part of that. Another more conspiracy minded theory might be that they are interesting in driving a certain narrative.
It’s even more interesting to think about why companies don’t use prediction markets. Seems like a company would be more profitable if they had the best predictors....so why aren’t they doing more to attract them? Robin Hanson has, what I think, are some great thoughts on this.
Again, agree with everything you’ve said in this post, but I don’t feel we have a great theory on why people don’t take forecasting more seriously and until we do are unlikely to increase their use.
In reality, many things happen at the same time. It is difficult to give a prediction of what X will achieve, if you don’t know the outcomes of Y and Z, which will be decided tomorrow. Perhaps the decision on Y and Z will depend on the outcome of votes for X.
For example, imagine making a prediction about a future of a startup that breaks some regulations, such as Uber. (Depending on your prediction, citizens will vote whether to invite Uber into your city.) Well, the prediction depends on how strongly the breaking of regulations will be prosecuted and punished. Suppose your political opponents have a strong influence on prosecutors and judges. Now, whatever prediction about the future of Uber you make, they can make the opposite happen. It would cost them some political points, but maybe it is worth doing for them, if it results in hugely discrediting you and eliminating you from the competition.
Now not only can your enemies make your project fail, they can also make you seem like a bad predictor if that happens. And if your prediction contains hundreds of “if”s, no one will take it seriously. “My project will increase the GDP of our community by 2% withing 5 years, assuming the government will not do something else that decreases the GDP, nor change the way GDP is measured, nor adopt new laws that make parts of my project illegal, nor change the tax code to make my project more expensive, nor … organize a Twitter mob that would attack people participating in my project, nor ….”
This is a really good point. It would be a terrible idea to judge economic policy according to whether GDP went up after it was enacted.
What is the way to judge economic policy?
Sure.
But thats true of pretty much every prediction market. Unless you’re opposed to prediction markets very broadly, I’m not sure this changes anything I’ve said.
prediction market + future depends on the voters’ actions = moral hazard
I guess this is an argument against prediction markets in general, unless the results are independent on what voters do (or a voter activity aimed to change the outcome is prosecuted as a fraud).
We can see this in politics. The Blue Party says that a major Green Party policy will fail. When the Green Party enacts it, the Blues sabotage it, then use its failure as evidence that Blue ideology is correct.
Likewise, partisan pollsters run polls designed to exaggerate support for their preferred candidate, under the theory that predictions of success can cause success.
It also seems to me that any form of prediction market has moral hazard. They’ve been criticized as assassination markets, and any questions that have anything to do with the behavior of an individual person set up an incentive for murder.
Hence, the key question is not whether moral hazard exists, but whether the tradeoffs are acceptable.
It’s hard to say if the entire world should be run by prediction markets. But I think we should be experimenting more with forecasting and prediction markets at more local levels.
Can you give an example? What is the difference between “sabotaging” a policy and simply opposing a policy you don’t support?
Sure. Republicans couldn’t afford to repeal the Affordable Care Act (I’m assuming you’re American, let me know if not and I’ll explain further), because it was too popular even among republicans.
One of the gears that makes it work is the individual mandate, a tax penalty for not having health insurance. This penalty was meant to incentivize young healthy people to sign up for health insurance, effectively subsidizing the insurance policies of the old and sick. The Republicans repealed this component, which should have the effect of gutting the revenue stream that makes the ACA work, driving up the cost of insurance.
Opposing the policy would look like Republicans repealing the ACA. Sabotaging it looks like keeping it in place, but repealing a tax penalty that effectively funds it, making it untenable in the long run as insurance prices spike. They can blame this on Democrats, arguing that a big government insurance program was bound to fail, while taking credit for the tax cut.
The dynamic I’m describing is meant only to illustrate the difference between political opposition and sabotage, not a serious piece of contemporary political analysis.
To define them more generally, political opposition is about making the consequences of your opposing actions clear.
Political sabotage is hiding the consequences of your opposing actions.
Well, in politics, you’re dealing with an issue where enforcing compliance would be difficult. It’s famously hard to get a politician to give a clear answer to a clear question.
I think that partly, the idea of forecasting as practiced by Tetlock et al is just relatively new. It hasn’t seeped into society to the extent that it might someday. Companies probably need to be of a certain size to make such tournaments seem worthwhile.
But I wouldn’t be surprised if some companies do in fact run internal forecasting competitions or prediction markets. The Pentagon tried running a prediction market to identify security threats, but shut it down when it was criticized as as “terrorism futures market.”
My particular interest is not the macro question of “why isn’t this done more,” but the micro question of “what’s the easiest way that an individual or small group could start a forecasting competition?”
One way is to build better prediction markets. If we would have better prediction markets that move a decent chunk of money those prediction could be used.
Another is to think through the application of forcasting in individual fields. In my post on cancer I for example wrote about Prediction-Based Medicine for compassionate use. While it’s likely impossible to convince people of Prediction-Based Medicine for those decisions that are currently allegedly made via Evidence-Based Medicine campaigning for it being used for in cases where compassionate use is currently done without any need for any evidence might be more promising.
On the one hand you have people who don’t like that doctors can promise patients whatever they want in cases of compassionate use and on the other hand you have people who find compassionate use important because it allows patients to get life saving drugs before they are approved by the FDA. Prediction-Based Medicine looks to me like a great compromize between the two sides.
When it comes to computer programming on tool I would like to see is a program that asks you to make a prediction whenever you run your unit test about whether or not those unit tests will fail.
There are a lot of cases of specific expertise where it’s worth thinking about how to build forcasting based systems for them.
When it comes to journalism, a new way to do journalism can be invented that works on the Blockchain. It’s possible to raise sizeable amounts of capital in ICO’s if you have a well thought out idea.