I don’t think your point applies to this specific graph, since it is a cumulative odds ratio graph selected for initially high error in one direction.
Associations from Ioannides’ analysis were selected for inclusion on this graph because there were dramatic changes from the first study to the next studies—in this case, because the first study had a high effect size and the others showed lower effect sizes. Studies were selected for Ioannides’ analysis in the first place because there were meta-analyses around them—which means lots of people tried to replicate them—which means the initial result must have been surprising and interesting. So these have been double-selected already for original studies likely to have high errors.
In a cumulative odds ratio graph (unlike the individual odds ratio graphs of the classic funnel plot), most of the work of subsequent studies will go to bringing the trend line closer to the mean. Even a study that shows an effect in the opposite direction as the original won’t move the trend line to the other side of the identity line if the effect is smaller. So if the graph is arranged so that the most surprising and deviant result is the first, then it could very well look like this one even if there were no publication bias.
This is doubly true when the graphs are not actually converging to one—Ioannides’ paper admits that three of these are probably real associations, and this is most obvious here on the DRD2 line, which converges around .5.
[A] cumulative odds ratio graph [is] unlike the individual odds ratio graphs of the classic funnel plot[.]
The original article should be edited to include note of this. As it stands, it is misleading. The original plot may or may not show problems, but expecting it to look like the “generic funnel plot” image is wrong.
I don’t think your point applies to this specific graph, since it is a cumulative odds ratio graph selected for initially high error in one direction.
Associations from Ioannides’ analysis were selected for inclusion on this graph because there were dramatic changes from the first study to the next studies—in this case, because the first study had a high effect size and the others showed lower effect sizes. Studies were selected for Ioannides’ analysis in the first place because there were meta-analyses around them—which means lots of people tried to replicate them—which means the initial result must have been surprising and interesting. So these have been double-selected already for original studies likely to have high errors.
In a cumulative odds ratio graph (unlike the individual odds ratio graphs of the classic funnel plot), most of the work of subsequent studies will go to bringing the trend line closer to the mean. Even a study that shows an effect in the opposite direction as the original won’t move the trend line to the other side of the identity line if the effect is smaller. So if the graph is arranged so that the most surprising and deviant result is the first, then it could very well look like this one even if there were no publication bias.
This is doubly true when the graphs are not actually converging to one—Ioannides’ paper admits that three of these are probably real associations, and this is most obvious here on the DRD2 line, which converges around .5.
The original article should be edited to include note of this. As it stands, it is misleading. The original plot may or may not show problems, but expecting it to look like the “generic funnel plot” image is wrong.