According to some people we here at less wrong are good at determining the truth. Other people are notoriouslynot.
I don’t know that Less Wrong is the appropriate venue for this, but I have felt for some time that I trust the truth-seeking capability here and that it could be used for something more productive than arguments about meta-ethics (no offense to the meta-ethicists intended). I also realize that people are fairly supportive of SIAI here in terms of giving spare cash away, but I feel like the community would be a good jumping-off point for a polling organization.
So I guess this leads to a few questions:
-Is anyone at LW currently involved with a polling firm?
-Is anyone (else) at LW interested in doing polls?
-Is LW an appropriate place to create a truth-seeking business, such as a pollster or a sponsor for studies?
None of these questions are immediate since I am a broke undergrad rather than an entrepreneur.
I’m not sure I understand the connection between truth-seeking and polling, unless the specific truth you seek is simply the percentage of people who give a particular answer to a poll. Are you simply talking about a more accurate polling company or using polling to find other truths?
Possibly related: I have a bet going with a reddit-acquaintance; basically, I gave him an upvote, and if x turns out to be true, he donates $1000 to SIAI.
If members of this community have an accurate, well calibrated map, making bets could be a cost-effective way to pump money into SIAI or other non-profits/charities (which signals caring as well as integrity).
Yes, a more accurate polling company; potentially polling on alternative subjects, I also had scientific studies (grant-writing, peer-reviewing) in mind but I have even less idea how that works and how to express what I would actually think about it.
The two examples you linked of bad polling seem to be examples of polling fraud rather than incompetence. It is not that these companies did not understand how to conduct an accurate poll, rather that they don’t appear to have been motivated to do so.
It seems to me that accurate polling is quite a well understood problem. Legitimate polling companies exist that are reasonably good at it. In many cases I don’t think there is much value (from a truth seeking perspective) in the poll data but I think it generally answers the question “what percentage of people give answer Y to question X?” fairly well. That’s just not a very useful piece of data in many cases.
Here’s an idea for how a LW-based commercial polling website could operate. Basically it is a variation on PredictionBook with a business model similar to TopCoder.
The website has business clients, and a large number of “forecasters” who have accounts on the website. Clients pay to have their questions added to the website, and forecasters give their probability estimates for whichever questions they like. Once the answer to a question has been verified, each forecaster is financially rewarded using some proper scoring rule. The more money assigned to a question, the higher the incentive for a forecaster to have good discrimination and calibration. Some clever software would also be needed to combine and summarize data in a way that is useful to clients.
The main advantage of this over other prediction markets is that the scoring rule encourages forecasters to give accurate probability estimates.
I’m not interested, in particular, with polling but I’m interested in it insofar as it is a way of getting data that people don’t otherwise have and thus improving our predictions. That said, LW totally is the place to create a truth-seeking business and as another broke undergrad (and putting off grad school) if you, anyone else and I can come up with a profitable venture that involves employing truth seeking I definitely want in.
The obvious way to make money with this is consulting, but I’m not sure why anyone would hire a bunch of philosophy/math/CS types to do the job.
People would hire the firm if it could be demonstrated that the firm consistently produced accurate results. So initial interest might be low, but pick up over time as the track record gets longer.
Perhaps start by giving it away, or sell it to small buyers (eg. individuals).
But I’ve got to admit I don’t have experience in this area, so my suggestions are mostly naive speculation (but hopefully my speculation is of high quality!). Research into existing prediction companies is called for.
According to some people we here at less wrong are good at determining the truth. Other people are notoriously not.
I don’t know that Less Wrong is the appropriate venue for this, but I have felt for some time that I trust the truth-seeking capability here and that it could be used for something more productive than arguments about meta-ethics (no offense to the meta-ethicists intended). I also realize that people are fairly supportive of SIAI here in terms of giving spare cash away, but I feel like the community would be a good jumping-off point for a polling organization.
So I guess this leads to a few questions:
-Is anyone at LW currently involved with a polling firm?
-Is anyone (else) at LW interested in doing polls?
-Is LW an appropriate place to create a truth-seeking business, such as a pollster or a sponsor for studies?
None of these questions are immediate since I am a broke undergrad rather than an entrepreneur.
I’m not sure I understand the connection between truth-seeking and polling, unless the specific truth you seek is simply the percentage of people who give a particular answer to a poll. Are you simply talking about a more accurate polling company or using polling to find other truths?
All that, and how does it make money?
Possibly related: I have a bet going with a reddit-acquaintance; basically, I gave him an upvote, and if x turns out to be true, he donates $1000 to SIAI.
If members of this community have an accurate, well calibrated map, making bets could be a cost-effective way to pump money into SIAI or other non-profits/charities (which signals caring as well as integrity).
Is such a thing in the realm of Dark Arts?
Yes, a more accurate polling company; potentially polling on alternative subjects, I also had scientific studies (grant-writing, peer-reviewing) in mind but I have even less idea how that works and how to express what I would actually think about it.
The two examples you linked of bad polling seem to be examples of polling fraud rather than incompetence. It is not that these companies did not understand how to conduct an accurate poll, rather that they don’t appear to have been motivated to do so.
It seems to me that accurate polling is quite a well understood problem. Legitimate polling companies exist that are reasonably good at it. In many cases I don’t think there is much value (from a truth seeking perspective) in the poll data but I think it generally answers the question “what percentage of people give answer Y to question X?” fairly well. That’s just not a very useful piece of data in many cases.
Here’s an idea for how a LW-based commercial polling website could operate. Basically it is a variation on PredictionBook with a business model similar to TopCoder.
The website has business clients, and a large number of “forecasters” who have accounts on the website. Clients pay to have their questions added to the website, and forecasters give their probability estimates for whichever questions they like. Once the answer to a question has been verified, each forecaster is financially rewarded using some proper scoring rule. The more money assigned to a question, the higher the incentive for a forecaster to have good discrimination and calibration. Some clever software would also be needed to combine and summarize data in a way that is useful to clients.
The main advantage of this over other prediction markets is that the scoring rule encourages forecasters to give accurate probability estimates.
I’m not interested, in particular, with polling but I’m interested in it insofar as it is a way of getting data that people don’t otherwise have and thus improving our predictions. That said, LW totally is the place to create a truth-seeking business and as another broke undergrad (and putting off grad school) if you, anyone else and I can come up with a profitable venture that involves employing truth seeking I definitely want in.
The obvious way to make money with this is consulting, but I’m not sure why anyone would hire a bunch of philosophy/math/CS types to do the job.
People would hire the firm if it could be demonstrated that the firm consistently produced accurate results. So initial interest might be low, but pick up over time as the track record gets longer.
Right, but how do you get started? Begin by giving away the service? Work on spec? What kind of companies/organizations would hire such a firm?
Bet on propositions on InTrade. If you are good, you will make money from the exercise, as well as establish crediblility.
Perhaps start by giving it away, or sell it to small buyers (eg. individuals).
But I’ve got to admit I don’t have experience in this area, so my suggestions are mostly naive speculation (but hopefully my speculation is of high quality!). Research into existing prediction companies is called for.