Are any LWer’s familiar with adversarial publishing? The basic idea is that two researchers who disagree on some empirically testable proposition come together with an arbiter to design an experiment to resolve their disagreement.
Here’s a summary of the process from an article (pdf) I recently read (where Daniel Kahneman was one of the adversaries).
When tempted to write a critique or to run an experimental refutation of a recent publication, consider the possibility of proposing joint
research under an agreed protocol. We call the scholars engaged in such an effort participants. If theoretical differences are deep or if
there are large differences in experimental routines between the laboratories, consider the possibility of asking a trusted colleague to
coordinate the effort, referee disagreements, and collect the data. We call that person an arbiter.
Agree on the details of an initial study, designed to subject the opposing claims to an informative empirical test. The participants should
seek to identify results that would change their mind, at least to some extent, and should explicitly anticipate their interpretations of
outcomes that would be inconsistent with their theoretical expectations. These predictions should be recorded by the arbiter to prevent
future disagreements about remembered interpretations.
If there are disagreements about unpublished data, a replication that is agreed to by both participants should be included in the initial
study.
Accept in advance that the initial study will be inconclusive. Allow each side to propose an additional experiment to exploit the fount of
hindsight wisdom that commonly becomes available when disliked results are obtained. Additional studies should be planned jointly,
with the arbiter resolving disagreements as they occur.
Agree in advance to produce an article with all participants as authors. The arbiter can take responsibility for several parts of the article:
an introduction to the debate, the report of experimental results, and a statement of agreed-upon conclusions. If significant disagreements
remain, the participants should write individual discussions. The length of these discussions should be determined in advance
and monitored by the arbiter. An author who has more to say than the arbiter allows should indicate this fact in a footnote and provide
readers with a way to obtain the added material.
The data should be under the control of the arbiter, who should be free to publish with only one of the original participants if the other
refuses to cooperate. Naturally, the circumstances of such an event should be part of the report.
All experimentation and writing should be done quickly, within deadlines agreed to in advance. Delay is likely to breed discord.
The arbiter should have the casting vote in selecting a venue for publication, and editors should be informed that requests for major
revisions are likely to create impossible problems for the participants in the exercise.
This seems like a great way to resolve academic disputes. Philip Tetlock appears to be an advocate. What do you think?
Are any LWer’s familiar with adversarial publishing? The basic idea is that two researchers who disagree on some empirically testable proposition come together with an arbiter to design an experiment to resolve their disagreement.
Here’s a summary of the process from an article (pdf) I recently read (where Daniel Kahneman was one of the adversaries).
When tempted to write a critique or to run an experimental refutation of a recent publication, consider the possibility of proposing joint research under an agreed protocol. We call the scholars engaged in such an effort participants. If theoretical differences are deep or if there are large differences in experimental routines between the laboratories, consider the possibility of asking a trusted colleague to coordinate the effort, referee disagreements, and collect the data. We call that person an arbiter.
Agree on the details of an initial study, designed to subject the opposing claims to an informative empirical test. The participants should seek to identify results that would change their mind, at least to some extent, and should explicitly anticipate their interpretations of outcomes that would be inconsistent with their theoretical expectations. These predictions should be recorded by the arbiter to prevent future disagreements about remembered interpretations.
If there are disagreements about unpublished data, a replication that is agreed to by both participants should be included in the initial study.
Accept in advance that the initial study will be inconclusive. Allow each side to propose an additional experiment to exploit the fount of hindsight wisdom that commonly becomes available when disliked results are obtained. Additional studies should be planned jointly, with the arbiter resolving disagreements as they occur.
Agree in advance to produce an article with all participants as authors. The arbiter can take responsibility for several parts of the article: an introduction to the debate, the report of experimental results, and a statement of agreed-upon conclusions. If significant disagreements remain, the participants should write individual discussions. The length of these discussions should be determined in advance and monitored by the arbiter. An author who has more to say than the arbiter allows should indicate this fact in a footnote and provide readers with a way to obtain the added material.
The data should be under the control of the arbiter, who should be free to publish with only one of the original participants if the other refuses to cooperate. Naturally, the circumstances of such an event should be part of the report.
All experimentation and writing should be done quickly, within deadlines agreed to in advance. Delay is likely to breed discord.
The arbiter should have the casting vote in selecting a venue for publication, and editors should be informed that requests for major revisions are likely to create impossible problems for the participants in the exercise.
This seems like a great way to resolve academic disputes. Philip Tetlock appears to be an advocate. What do you think?