Figuring out that a paper contains fake research requires a lot of domain knowledge. For instance, I have read enough software engineering papers to spot fake research, but would have a lot of trouble spotting fake research in related fields, e.g., database systems. What counts as fake research, everybody has their own specific opinions.
My approach, based on experience reading very many software engineering, is to treat all papers as having a low value (fake or otherwise) until proven otherwise.
Emailing the author asking for a copy of their data is always interesting; around a third don’t reply, and a third have lost/not kept the data.
Spotting fake research is a (very important) niche topic. A more generally useful proposal would be to teach people how to read papers. Reading one paper might almost be worse than reading none at all, because of the false feeling of knowing it gives the reader. I always tell people to read the thesis from which the paper was derived (if there is one); a thesis provides a lot more context and is a much easier read than a paper (which is a very condensed summary of the thesis). Researchers much prefer to have their paper cited, because thesis citations don’t ‘count’.
Is a Fake journal club worth the effort? It’s possible to spend more time debunking a paper than was spent doing the original research, and for nothing to happen.
Knowing if North Korea is going to do a hydrogen bomb test this year also requires a lot of domain knowledge, and one can invest arbitrary effort into obtaining new data like smuggling oneself into North Korea or interrogating defectors, and may in fact require knowledge it is impossible to obtain outside a particular skull in North Korea. Yet, calibration training still exists and will improve forecasts on both North Korea and on how many M&Ms are in that big jar over there.
Figuring out that a paper contains fake research requires a lot of domain knowledge. For instance, I have read enough software engineering papers to spot fake research, but would have a lot of trouble spotting fake research in related fields, e.g., database systems. What counts as fake research, everybody has their own specific opinions.
My approach, based on experience reading very many software engineering, is to treat all papers as having a low value (fake or otherwise) until proven otherwise.
Emailing the author asking for a copy of their data is always interesting; around a third don’t reply, and a third have lost/not kept the data.
Spotting fake research is a (very important) niche topic. A more generally useful proposal would be to teach people how to read papers. Reading one paper might almost be worse than reading none at all, because of the false feeling of knowing it gives the reader. I always tell people to read the thesis from which the paper was derived (if there is one); a thesis provides a lot more context and is a much easier read than a paper (which is a very condensed summary of the thesis). Researchers much prefer to have their paper cited, because thesis citations don’t ‘count’.
Is a Fake journal club worth the effort? It’s possible to spend more time debunking a paper than was spent doing the original research, and for nothing to happen.
Knowing if North Korea is going to do a hydrogen bomb test this year also requires a lot of domain knowledge, and one can invest arbitrary effort into obtaining new data like smuggling oneself into North Korea or interrogating defectors, and may in fact require knowledge it is impossible to obtain outside a particular skull in North Korea. Yet, calibration training still exists and will improve forecasts on both North Korea and on how many M&Ms are in that big jar over there.