The amusing thing is that Mitchel’s argument proves much more than he wants it to prove.
Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way. Unless direct replications are conducted by flawless experimenters, nothing interesting can be learned from them.
Notice that the above argument applies just as well to the original experiment being replicated.
Has anyone read his entire article? Does he attempt any justification for why this particular argument doesn’t equally apply to the original experiment?
One principle I try to keep in mind is “The other guy is probably not a total moron. If it seems that way, you’re probably missing something.”
I read it. He has a section titled “The asymmetry between positive and negative evidence”.
His argument is that a positive result is like seeing a black swan, and a null result is like seeing a white swan, and once you see a black swan, then no matter how many white swans you see it doesn’t prove that all swans are white.
He addresses the objection that this leaves us unable to ever reject a spurious claim. His answer is that, since negative evidence is always meaningless, we should get positive evidence that the experimenter was wrong.
I think this is a fair summary of the section. It’s not long, so you can check for yourself. I am… not impressed.
His argument is that a positive result is like seeing a black swan
Actually, it’s like hearing a report of a black swan, which is why the burden of proof is generally put on the report.
It’s even worse than that for him. What a bad analogy for him to rest his case on. Surely, the purpose of these social science studies is not to make a claim about the existence of some bizarre subset of the population (a black swan), but that the results will generalize to the population at large (all swans are black).
There’s a lot wrong with the argument; he has no actual justification for assuming that social science is anything like swan-spotting.
But even within his unjustified analogy… apparently if someone reports a new color of swan in Australia, he might give polygraphs and vision tests to the reporter, but sending an expedition to Australia to check it out would be of no scientific value.
My interpretation is that he’s saying false positives are unlikely but false negatives are common. If that’s the case, then getting a positive result should greatly affect your beliefs; but a negative replication shouldn’t shift them much.
(I can’t actually justify this interpretation by reading the text, but it does make the text seem fairly reasonable.)
Congratulations, you’re smarter than a Harvard professor. I noticed the same thing, so I’m also smarter than a Harvard professor. I would hope that everyone on this site is smarter than a Harvard professor. A Harvard professor who got his B.A. and M.S. from Yale and his Ph.D. from Harvard.
This is sociologically interesting: is it an isolated incident that a Harvard professor—an award-winning one, no less—would so loudly fail to comprehend how science and statistics work? That doesn’t seem likely, for two reasons. First, he probably talked it over with other people in the field, or at least mentioned why he thought replication was bad and wrong, and no one talked him out of it. Second, the system he came through produced him, and it’s unlikely that that sort of error would only be produced once, since education is fairly standardized. So: what does this say about the relevant institutions? (The social sciences, HYP, academia, etc.)
I don’t think that means you are smarter than that Harvard professor. He is a very successful person and has reached heights coveted by many very smart people. It just means that the game he is playing is not one where you get ahead by saying things that make sense.
For example, if you listen to a successful politician and spot a false statement he utters, that does not mean that you are smarter than that politician.
Yes, academics are supposed to raise the status of their institution. This brings in money, which helps educate students, which makes the world a better place. Unsuccessful replication threatens this. Plus, replication does nothing to advance social justice.
Jason employs functional neuroimaging (fMRI) and behavioral methods to study how we infer the thoughts, feelings, and opinions of others (i.e., how we mentalize) as well as how we reason about counterfactual experiences.
That doesn’t seem to me like someone who focus on advancing social justice.
You should have looked at his vita for a more accurate description of his activities. If you had looked at his paper titles, some of them indicate he’s not a stranger to social justice like theorizing and investigation, and likewise his funding sources, on top of Harvard’s well-earned reputation: eg. “What’s in a forename?: Cue familiarity and stereotypical thinking”, “Gender differences in implicit weight identity”, “Deflecting negative self-relevant stereotype activation: The effects of individuation”, “Me and my group: Cultural status can disrupt cognitive consistency”, and the funding:
June 2007 – May 2010: National Science Foundation (BCS 0642448), “The neural basis of stereotyping”, $609,800 (co-PI: Mahzarin Banaji)...September 2010 – August 2012: Templeton Foundation for Positive Neuroscience, “Vicarious Neural Response to Others as a Basis for Altruistic Behavior”, $180,000 (co-PI: Jamil Zaki)
Sorry my mistake. As far as the paper titles goes.
As far as the funding goes the National Science Foundation isn’t an entity that I would see as spearheading the social justice movement.
The same goes for the Templeton Foundation. They have the reputation of wanting “progress in spiritual discoveries” instead of “advancing social justice”.
Both groups are quite big and may fund more than you think; not that the grantors always get what they think they’re getting or are the only people who are then allowed to draw upon the research. For example, consider “stereotype threat”, much beloved of social-justice types for explaining how bad white people keep test scores low for women and blacks; you can see the NSF certainly has been involved in that research in the past just with a cursory google: http://scholar.google.com/scholar?q=%22stereotype%20threat%22%20%22National%20Science%20Foundation%22 (I count ~4 NSF grants attested to just from the snippet-view for the first page).
Both groups are quite big and may fund more than you think; not that the grantors always get what they think they’re getting or are the only people who are then allowed to draw upon the research.
I do accept that both groups do fund a large variety of courses but I still wouldn’t conclude from the funding source that there a bias in the direction of the social justice movement.
When thinking about the title “The neural basis of stereotyping” you might be right that it smells like pseudoscience. It’s a bit like the “The neural basis of acupuncture”. Instead of searching for the neural basis it would make much more sense to focus of studying the actual effect.
I wonder if I picked up to much stereotypes about fMRI research ;)
I do accept that both groups do fund a large variety of courses but I still wouldn’t conclude from the funding source that there a bias in the direction of the social justice movement.
I never said the NSF is biased in favor of SJ. I said if you had bothered to look at the vita instead of stopping at the most convenient place, you would have found a number of paper and grant titles which indicate a more than theoretical interest in topics strongly associated with SJ on top of his affiliation with an institute with a strong background both current and historical in liberal thought & SJ-like figures such as Cornell West and to borrow from parallel ongoing conversations, crucified Sumners for his well-founded suggestion that the elite math achievement gap might be a necessary consequence of gender differences. And then you said the NSF was evidence against SJ association, which is either wrong or weak evidence since they fund related research all the time.
The might have crucified Summers but they led him rise to be president at Harvard in the first place and didn’t seem to successfully taught him not to say things like that.
Cornell West has professor for religion and then professor for African-American Studies. As far as my instincts go I wouldn’t expect the same thing from Harvard department of psychology than I would expect from the department of African-American Studies.
And then you said the NSF was evidence against SJ association, which is either wrong or weak evidence since they fund related research all the time.
I didn’t. I took your post as saying that I should update in the direction of him being part of the social justice movement based on his funding source and wanted to reject updating based on that information in that direction.
Either you don’t read your own comments, or you classify all social justice as signaling.
In the latter case, I’m sure you’ll come up with reasons why you aren’t making a fully general counterargument. I would urge you to stop for a second and ask concretely what you would have thought about, say, abolitionists.
I don’t understand what you mean. Social justice is a label for a set of beliefs. I do not think that social justice is just about signaling. If it helps to understand what I mean, I’m a politically incorrect college professor.
academics are supposed to raise the status of their institution...He teaches at an elite university so it’s highly probably he greatly values social justice.
There may indeed be a contradiction, but the real question—if we grant the first part—is what do you know and how do you know it? This seems to me like a really obvious point.
The amusing thing is that Mitchel’s argument proves much more than he wants it to prove.
Notice that the above argument applies just as well to the original experiment being replicated.
Yes, noticed.
Has anyone read his entire article? Does he attempt any justification for why this particular argument doesn’t equally apply to the original experiment?
One principle I try to keep in mind is “The other guy is probably not a total moron. If it seems that way, you’re probably missing something.”
I read it. He has a section titled “The asymmetry between positive and negative evidence”.
His argument is that a positive result is like seeing a black swan, and a null result is like seeing a white swan, and once you see a black swan, then no matter how many white swans you see it doesn’t prove that all swans are white.
He addresses the objection that this leaves us unable to ever reject a spurious claim. His answer is that, since negative evidence is always meaningless, we should get positive evidence that the experimenter was wrong.
I think this is a fair summary of the section. It’s not long, so you can check for yourself. I am… not impressed.
Actually, it’s like hearing a report of a black swan, which is why the burden of proof is generally put on the report.
It’s even worse than that for him. What a bad analogy for him to rest his case on. Surely, the purpose of these social science studies is not to make a claim about the existence of some bizarre subset of the population (a black swan), but that the results will generalize to the population at large (all swans are black).
That’s more than enough for me.
Thanks for taking the bullet for us.
There’s a lot wrong with the argument; he has no actual justification for assuming that social science is anything like swan-spotting.
But even within his unjustified analogy… apparently if someone reports a new color of swan in Australia, he might give polygraphs and vision tests to the reporter, but sending an expedition to Australia to check it out would be of no scientific value.
Medicine has had its Ioannidis. Is anyone doing the same for psychology and social science?
Uri Simonsohn?
There’s El Vul’s Voodoo Correlations in Social Neuroscience.
My interpretation is that he’s saying false positives are unlikely but false negatives are common. If that’s the case, then getting a positive result should greatly affect your beliefs; but a negative replication shouldn’t shift them much.
(I can’t actually justify this interpretation by reading the text, but it does make the text seem fairly reasonable.)
Beautiful point.
Congratulations, you’re smarter than a Harvard professor. I noticed the same thing, so I’m also smarter than a Harvard professor. I would hope that everyone on this site is smarter than a Harvard professor. A Harvard professor who got his B.A. and M.S. from Yale and his Ph.D. from Harvard.
This is sociologically interesting: is it an isolated incident that a Harvard professor—an award-winning one, no less—would so loudly fail to comprehend how science and statistics work? That doesn’t seem likely, for two reasons. First, he probably talked it over with other people in the field, or at least mentioned why he thought replication was bad and wrong, and no one talked him out of it. Second, the system he came through produced him, and it’s unlikely that that sort of error would only be produced once, since education is fairly standardized. So: what does this say about the relevant institutions? (The social sciences, HYP, academia, etc.)
I don’t think that means you are smarter than that Harvard professor. He is a very successful person and has reached heights coveted by many very smart people. It just means that the game he is playing is not one where you get ahead by saying things that make sense.
For example, if you listen to a successful politician and spot a false statement he utters, that does not mean that you are smarter than that politician.
Yes, academics are supposed to raise the status of their institution. This brings in money, which helps educate students, which makes the world a better place. Unsuccessful replication threatens this. Plus, replication does nothing to advance social justice.
His website describes his research as:
That doesn’t seem to me like someone who focus on advancing social justice.
You should have looked at his vita for a more accurate description of his activities. If you had looked at his paper titles, some of them indicate he’s not a stranger to social justice like theorizing and investigation, and likewise his funding sources, on top of Harvard’s well-earned reputation: eg. “What’s in a forename?: Cue familiarity and stereotypical thinking”, “Gender differences in implicit weight identity”, “Deflecting negative self-relevant stereotype activation: The effects of individuation”, “Me and my group: Cultural status can disrupt cognitive consistency”, and the funding:
Sorry my mistake. As far as the paper titles goes.
As far as the funding goes the National Science Foundation isn’t an entity that I would see as spearheading the social justice movement.
The same goes for the Templeton Foundation. They have the reputation of wanting “progress in spiritual discoveries” instead of “advancing social justice”.
Both groups are quite big and may fund more than you think; not that the grantors always get what they think they’re getting or are the only people who are then allowed to draw upon the research. For example, consider “stereotype threat”, much beloved of social-justice types for explaining how bad white people keep test scores low for women and blacks; you can see the NSF certainly has been involved in that research in the past just with a cursory google: http://scholar.google.com/scholar?q=%22stereotype%20threat%22%20%22National%20Science%20Foundation%22 (I count ~4 NSF grants attested to just from the snippet-view for the first page).
I do accept that both groups do fund a large variety of courses but I still wouldn’t conclude from the funding source that there a bias in the direction of the social justice movement.
When thinking about the title “The neural basis of stereotyping” you might be right that it smells like pseudoscience. It’s a bit like the “The neural basis of acupuncture”. Instead of searching for the neural basis it would make much more sense to focus of studying the actual effect.
I wonder if I picked up to much stereotypes about fMRI research ;)
I never said the NSF is biased in favor of SJ. I said if you had bothered to look at the vita instead of stopping at the most convenient place, you would have found a number of paper and grant titles which indicate a more than theoretical interest in topics strongly associated with SJ on top of his affiliation with an institute with a strong background both current and historical in liberal thought & SJ-like figures such as Cornell West and to borrow from parallel ongoing conversations, crucified Sumners for his well-founded suggestion that the elite math achievement gap might be a necessary consequence of gender differences. And then you said the NSF was evidence against SJ association, which is either wrong or weak evidence since they fund related research all the time.
The might have crucified Summers but they led him rise to be president at Harvard in the first place and didn’t seem to successfully taught him not to say things like that.
Cornell West has professor for religion and then professor for African-American Studies. As far as my instincts go I wouldn’t expect the same thing from Harvard department of psychology than I would expect from the department of African-American Studies.
I didn’t. I took your post as saying that I should update in the direction of him being part of the social justice movement based on his funding source and wanted to reject updating based on that information in that direction.
He teaches at an elite university so it’s highly probably he greatly values social justice.
Either you don’t read your own comments, or you classify all social justice as signaling.
In the latter case, I’m sure you’ll come up with reasons why you aren’t making a fully general counterargument. I would urge you to stop for a second and ask concretely what you would have thought about, say, abolitionists.
I don’t understand what you mean. Social justice is a label for a set of beliefs. I do not think that social justice is just about signaling. If it helps to understand what I mean, I’m a politically incorrect college professor.
Are you implying that there is a contradiction in what you quoted? If so, I don’t see it.
There may indeed be a contradiction, but the real question—if we grant the first part—is what do you know and how do you know it? This seems to me like a really obvious point.
Reasoning for this claim? Do you also believe that original research can do nothing to advance social justice? That doesn’t seem likely.