Plus, of course, there is huge selection bias. How many people with regular jobs, for example, do you think spend their evenings doing MTurk jobs?
I discuss this in the “Diversity of the Sample” subsection of the “Is Mechanical Turk a Reliable Source of Data?” section.
The question is not “is MTurk representative?” but rather “Is MTurk representative enough to be useful in answering the kinds of questions we want to answer and quicker / cheaper than our alternative sample sources?”.
The first question is “Can you trust the data coming out of MTurk surveys?”
The paper which your link references is behind the paywall but it seems likely to me that they gathered the data on representativeness of MTurk workers through a survey of MTurk workers. Is there a reason to trust these numbers?
The paper which your link references is behind the paywall
Which one? I can make it publicly available.
but it seems likely to me that they gathered the data on representativeness of MTurk workers through a survey of MTurk workers. Is there a reason to trust these numbers?
You can compare the answers to other samples.
Unless, of course, your concern is that the subjects are lying about their demographics, which is certainly possible. But then, it would be pretty amazing that this mix of lies and truths creates a believable sample. And what would be the motivation to lie about demographics? Would this motivation be any higher than other surveys? Do you doubt the demographics in non-MTurk samples?
I actually do agree this is a risk, so we’d have to (1) maybe run a study first to gauge how often MTurkers lie, perhaps using the Marlowe-Crowne Social Desirability Inventory, and/or (2) look through MTurker forums to see if people talk about lying on demographics. (One demographic that is known to be fabricated fairly often is nationality, because many MTurk tasks are restricted to Americans.)
Instead of dismissing MTurk based on expectations that it would be useless for research, I think it would be important to test it. After all, published social science has made use of MTurk samples, so we have some basis for expecting it to be at least worth testing to see if it’s legitimate.
Unless, of course, your concern is that the subjects are lying about their demographics
Yes. Or, rather, the subjects submit noise as data.
Consider, e.g. a Vietnamese teenager who knows some English and has declared himself as an American to MTurk. He’ll fill out a survey because he’ll get paid for it, but there is zero incentive for him to give true answers (and some questions like “Did you vote for Obama?” are meaningless for him). The rational thing for him to do is to put checkmarks into boxes as quickly as he can without being obvious about his answers being random.
Instead of dismissing MTurk based on expectations that it would be useless for research, I think it would be important to test it.
I’ll rephrase this as “it would be useful and necessary to test it before we use MTurk samples for research”.
The rational thing for him to do is to put checkmarks into boxes as quickly as he can without being obvious about his answers being random.
This is a good point. You still would be able to match the resulting demographics to known trends and see how reliable your sample is, however. Random answers should show, either overtly on checks, or subtlety through aggregate statistics.
~
I’ll rephrase this as “it would be useful and necessary to test it before we use MTurk samples for research”.
I discuss this in the “Diversity of the Sample” subsection of the “Is Mechanical Turk a Reliable Source of Data?” section.
The question is not “is MTurk representative?” but rather “Is MTurk representative enough to be useful in answering the kinds of questions we want to answer and quicker / cheaper than our alternative sample sources?”.
The first question is “Can you trust the data coming out of MTurk surveys?”
The paper which your link references is behind the paywall but it seems likely to me that they gathered the data on representativeness of MTurk workers through a survey of MTurk workers. Is there a reason to trust these numbers?
Which one? I can make it publicly available.
You can compare the answers to other samples.
Unless, of course, your concern is that the subjects are lying about their demographics, which is certainly possible. But then, it would be pretty amazing that this mix of lies and truths creates a believable sample. And what would be the motivation to lie about demographics? Would this motivation be any higher than other surveys? Do you doubt the demographics in non-MTurk samples?
I actually do agree this is a risk, so we’d have to (1) maybe run a study first to gauge how often MTurkers lie, perhaps using the Marlowe-Crowne Social Desirability Inventory, and/or (2) look through MTurker forums to see if people talk about lying on demographics. (One demographic that is known to be fabricated fairly often is nationality, because many MTurk tasks are restricted to Americans.)
Instead of dismissing MTurk based on expectations that it would be useless for research, I think it would be important to test it. After all, published social science has made use of MTurk samples, so we have some basis for expecting it to be at least worth testing to see if it’s legitimate.
The paper I mean is that one: http://cpx.sagepub.com/content/early/2013/01/31/2167702612469015.abstract
Yes. Or, rather, the subjects submit noise as data.
Consider, e.g. a Vietnamese teenager who knows some English and has declared himself as an American to MTurk. He’ll fill out a survey because he’ll get paid for it, but there is zero incentive for him to give true answers (and some questions like “Did you vote for Obama?” are meaningless for him). The rational thing for him to do is to put checkmarks into boxes as quickly as he can without being obvious about his answers being random.
I’ll rephrase this as “it would be useful and necessary to test it before we use MTurk samples for research”.
Here you go.
~
This is a good point. You still would be able to match the resulting demographics to known trends and see how reliable your sample is, however. Random answers should show, either overtly on checks, or subtlety through aggregate statistics.
~
Definitely.