I’m interested if you have a toy example showing how Simpsons paradox could have an impact here?
I assume that has a placebo/doesn’t have a placebo is a binary variable, and I also assume that the number of people in each arm in each experiment is the same. I can’t really see how you would end up with Simpsons paradox with that set up.
It’s not exactly Simpson’s, but we don’t even need a toy model as in their updated analysis it highlights details in line with exactly what I described above (down to tying in earlier PiPC research), and describe precisely the issue with pooled results across different subgroupings of placebo interventions:
It can be difficult to interpret whether a pooled standardised mean difference is large enough to be of clinical relevance. A consensus paper found that an analgesic effect of 10 mm on a 100 mm visual analogue scale represented a ‘minimal effect’ (Dworkin 2008). The pooled effect of placebo on pain based on the four German acupuncture trials corresponded to 16 mm on a 100 mm visual analogue scale, which amounts to approximately 75% of the effect of non‐steroidal anti‐inflammatory drugs on arthritis‐related pain (Gøtzsche 1990). However, the pooled effect of the three other pain trials with low risk of bias corresponded to 3 mm. Thus, the analgesic effect of placebo seems clinically relevant in some situations and not in others.
Putting subgroups with a physical intervention where there’s a 16⁄100 result with 10⁄100 as significant in with subgroups where there’s a 3⁄100 result and only looking at the pooled result might lead someone to thinking “there’s no significant effect” as occurred with OP, even though there’s clearly a significant effect for one subgroup when they aren’t pooled.
This is part of why in the discussion they explicitly state:
However, our findings do not imply that placebo interventions have no effect. We found an effect on patient‐reported outcomes, especially on pain. Several trials of low risk of bias reported large effects of placebo on pain, but other similar trials reported negligible effect of placebo, indicating the importance of background factors. We identified three clinical factors that were associated with higher effects of placebo: physical placebos...
Additionally, the criticism they raise in their implications section about there being no open label placebo data is no longer true, which was the research I was pointing OP towards.
The problem here was that the aggregate analysis at face value presents a very different result from a detailed review of the subgroups, particularly along physical vs pharmacological placebos, all of which has been explored further in research since this analysis.
I’m interested if you have a toy example showing how Simpsons paradox could have an impact here?
I assume that has a placebo/doesn’t have a placebo is a binary variable, and I also assume that the number of people in each arm in each experiment is the same. I can’t really see how you would end up with Simpsons paradox with that set up.
It’s not exactly Simpson’s, but we don’t even need a toy model as in their updated analysis it highlights details in line with exactly what I described above (down to tying in earlier PiPC research), and describe precisely the issue with pooled results across different subgroupings of placebo interventions:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7156905/
Putting subgroups with a physical intervention where there’s a 16⁄100 result with 10⁄100 as significant in with subgroups where there’s a 3⁄100 result and only looking at the pooled result might lead someone to thinking “there’s no significant effect” as occurred with OP, even though there’s clearly a significant effect for one subgroup when they aren’t pooled.
This is part of why in the discussion they explicitly state:
Additionally, the criticism they raise in their implications section about there being no open label placebo data is no longer true, which was the research I was pointing OP towards.
The problem here was that the aggregate analysis at face value presents a very different result from a detailed review of the subgroups, particularly along physical vs pharmacological placebos, all of which has been explored further in research since this analysis.