For a few years, I attended a meeting called Animal Behavior Lunch where we discussed new animal behavior articles. All of the meetings consisted of graduate students talking at great length about the flaws of that week’s paper. The professors in attendance knew better but somehow we did not manage to teach this. The students seemed to have a strong bias to criticize. Perhaps they had been told that “critical thinking” is good. They may have never been told that appreciation should come first. I suspect failure to teach graduate students to see clearly the virtues of flawed research is the beginning of the problem I discuss here: Mature researchers who don’t do this or that because they have been told not to do it (it has obvious flaws) and as a result do nothing.
Seth Roberts, ‘Something is better than nothing’, Nutrition, vol. 23, no. 11 (November, 2007), p. 912
Roberts, naturally, has substantial interest in avoiding any criticism, and the work of people like Ioannides and the eternal life of the publication bias says that if anything, we are insufficiently critical...
I think we’re looking at the wrong kind of criticism. Like, the kind of criticism you can make with almost equal ease of results that will and won’t turn out to replicate later.
As you know, I agree with you that Roberts is incorrigibly biased, and I liked your earlier post on this. But I think we can be critical in the sense you have in mind, and still try to cultivate the attitude that I take Roberts to be hinting at. Perhaps this is not very clear in the passage I chose to quote though.
From an outside view, how can we distinguish this virtue-of-flawed-research from insiders refraining from criticizing each other for the sake of the reputation of the research field?
Virtue of flawed research insiders won’t not criticise the flaws, but they will follow up on them with further studies expanding on a point or fixing a methodology.
The problem that Roberts might be criticising is the sort of thinking that goes: I’ve made a criticism, now we can forget about the thing.
One of the more useful class discussions I had consciously started with the opposite. The first question was what was good and useful in the week’s reading. We proceeded to criticism, but starting with “is there anything useful here?” made the discussion more useful and positive.
Seth Roberts, ‘Something is better than nothing’, Nutrition, vol. 23, no. 11 (November, 2007), p. 912
Roberts, naturally, has substantial interest in avoiding any criticism, and the work of people like Ioannides and the eternal life of the publication bias says that if anything, we are insufficiently critical...
I think we’re looking at the wrong kind of criticism. Like, the kind of criticism you can make with almost equal ease of results that will and won’t turn out to replicate later.
As you know, I agree with you that Roberts is incorrigibly biased, and I liked your earlier post on this. But I think we can be critical in the sense you have in mind, and still try to cultivate the attitude that I take Roberts to be hinting at. Perhaps this is not very clear in the passage I chose to quote though.
From an outside view, how can we distinguish this virtue-of-flawed-research from insiders refraining from criticizing each other for the sake of the reputation of the research field?
Virtue of flawed research insiders won’t not criticise the flaws, but they will follow up on them with further studies expanding on a point or fixing a methodology.
The problem that Roberts might be criticising is the sort of thinking that goes: I’ve made a criticism, now we can forget about the thing.
One of the more useful class discussions I had consciously started with the opposite. The first question was what was good and useful in the week’s reading. We proceeded to criticism, but starting with “is there anything useful here?” made the discussion more useful and positive.