This isn’t evidence of rigging as such, but I should note that some of my psychologist friends are rather skeptical about the methodology of Moral Foundations Theory. E.g. in his 2009 paper, Haidt reports the Cronbach’s alphas for the three MFT studies:
Study 1 [...] Cronbach’s alphas for the three-item measures of each foundation were .62 (Harm), .67 (Fairness), .59 (Ingroup), .39 (Authority), and .70 (Purity).
Study 2 [...] Cronbach’s alphas for each foundation were .71 (Harm), .70 (Fairness), .71 (Ingroup), .64 (Authority), and .76 (Purity).”
Study 3 [...] Cronbach’s alphas for each foundation were .69 (Harm), .69 (Fairness), .69 (Ingroup), .67 (Authority), and .58 (Purity).
I’m no expert in statistics myself, but I’m told that an alpha of .70 indicates a measure for which half of the result is just noise/error and half something real, while alphas of less than .7 are composed more of noise than anything else. As can be seen in the above numbers, Haidt’s measures occasionally reach that minimum level, but more frequently (at least in that paper) they don’t. Which implies that the MFT questions may not really be measuring what Haidt thinks they’re measuring.
(Still, many of Haidt’s claims seem intuitively right, so I’m inclined to believe that he’s roughly on the right track.)
(Here is Haidt’s response, which I find rather unconvincing.)
In fact, the more I think about Haidt’s questions, the more heavily biased they seem. For example, one of his “authority” questions asks for how much money you’d curse your parents in their face, and have to wait for a year to explain and apologize. Imagine if he instead asked for how much money you’d yell racial insults at a black person. Now, Haidt would presumably say that the latter falls properly under “harm,” since it would be greatly emotionally hurtful to this person. But how does this same argument not apply to someone being cursed by their own child?!
This isn’t evidence of rigging as such, but I should note that some of my psychologist friends are rather skeptical about the methodology of Moral Foundations Theory. E.g. in his 2009 paper, Haidt reports the Cronbach’s alphas for the three MFT studies:
I’m no expert in statistics myself, but I’m told that an alpha of .70 indicates a measure for which half of the result is just noise/error and half something real, while alphas of less than .7 are composed more of noise than anything else. As can be seen in the above numbers, Haidt’s measures occasionally reach that minimum level, but more frequently (at least in that paper) they don’t. Which implies that the MFT questions may not really be measuring what Haidt thinks they’re measuring.
(Still, many of Haidt’s claims seem intuitively right, so I’m inclined to believe that he’s roughly on the right track.)
Looking again at the questions listed in this paper, I remembered a blog post by Bryan Caplan in which he proposed some skillfully thought up alternative questions that make Haidt’s biases especially apparent:
http://econlog.econlib.org/archives/2010/03/do_liberals_use.html
(Here is Haidt’s response, which I find rather unconvincing.)
In fact, the more I think about Haidt’s questions, the more heavily biased they seem. For example, one of his “authority” questions asks for how much money you’d curse your parents in their face, and have to wait for a year to explain and apologize. Imagine if he instead asked for how much money you’d yell racial insults at a black person. Now, Haidt would presumably say that the latter falls properly under “harm,” since it would be greatly emotionally hurtful to this person. But how does this same argument not apply to someone being cursed by their own child?!