For comparison: imagine some medical researchers are interested in whether a particular medicine helps with a particular medical condition, so they set up a placebo controlled trial. A bunch of people with the medical condition all get their symptoms tested, then they flip a coin and half get pills with the medicine while the other half get sugar pills, and they don’t know whether they have the real pills. Then, some time later, they all get their symptoms tested again.
Now, imagine that I’m interested in “placebo effects”—I want to see if the ritual of taking sugar pills which you think might be medicine improves people’s health, or causes side effects, and I want to piggyback on this medical trial. I could just look at the pre vs post results for the set of people who got the sugar pills, but unfortunately this medical condition varies over time so I can’t disentangle effects of the pill-taking ritual from changes over time. I wish the study had a third “no-pill” group who (knowingly) didn’t get any treatment, in addition to the medical pill group and the inert pill group. Then I could just compare the results of the sugar pill group to the no pill group. But it doesn’t.
So I have the clever idea of getting the researchers to add a question to the tests at the end of the study, where they ask the patients whether they think they got the medicine pill or the sugar pill. That gives me a nice 2x2 design, where patients differ both in whether they got the medicine pill or the sugar pill, and separately in whether they believe they got the medicine pill or the sugar pill. So I can look separately at each of the 4 groups to see how much their condition improved, and what side effects they got. Changes that are associated with beliefs, I can claim, are based on the psychological effects of this pill taking ritual rather than the physiological effects of the substance they ingested.
This is a terrible study design. Who’s going to believe they got the real medicine? Well, people whose condition improved will tend to think they must’ve gotten the real medicine. And people who noticed physiological states like nausea or dry mouth will tend to think they’ve gotten the real medicine. This study design will say that improved condition & nausea are caused by people’s beliefs about whether they got the medicine, when in reality it’s the reverse: the beliefs are caused by these physical changes.
If I’m especially meddlesome, I might even tell the original researchers that they should use this 2x2 design to evaluate the original study. Instead of just comparing the outcomes for the medicine pill group and the sugar pill group, they should compare the outcomes while controlling for people’s beliefs about whether they got the medicine. That would mess up their study. It would be asking how effective the medicine is, after removing any effects that allowed patients to realize that they’d gotten the medicine (as if belief-entangled effects couldn’t be physiological effects of the substance).
For comparison: imagine some medical researchers are interested in whether a particular medicine helps with a particular medical condition, so they set up a placebo controlled trial. A bunch of people with the medical condition all get their symptoms tested, then they flip a coin and half get pills with the medicine while the other half get sugar pills, and they don’t know whether they have the real pills. Then, some time later, they all get their symptoms tested again.
Now, imagine that I’m interested in “placebo effects”—I want to see if the ritual of taking sugar pills which you think might be medicine improves people’s health, or causes side effects, and I want to piggyback on this medical trial. I could just look at the pre vs post results for the set of people who got the sugar pills, but unfortunately this medical condition varies over time so I can’t disentangle effects of the pill-taking ritual from changes over time. I wish the study had a third “no-pill” group who (knowingly) didn’t get any treatment, in addition to the medical pill group and the inert pill group. Then I could just compare the results of the sugar pill group to the no pill group. But it doesn’t.
So I have the clever idea of getting the researchers to add a question to the tests at the end of the study, where they ask the patients whether they think they got the medicine pill or the sugar pill. That gives me a nice 2x2 design, where patients differ both in whether they got the medicine pill or the sugar pill, and separately in whether they believe they got the medicine pill or the sugar pill. So I can look separately at each of the 4 groups to see how much their condition improved, and what side effects they got. Changes that are associated with beliefs, I can claim, are based on the psychological effects of this pill taking ritual rather than the physiological effects of the substance they ingested.
This is a terrible study design. Who’s going to believe they got the real medicine? Well, people whose condition improved will tend to think they must’ve gotten the real medicine. And people who noticed physiological states like nausea or dry mouth will tend to think they’ve gotten the real medicine. This study design will say that improved condition & nausea are caused by people’s beliefs about whether they got the medicine, when in reality it’s the reverse: the beliefs are caused by these physical changes.
If I’m especially meddlesome, I might even tell the original researchers that they should use this 2x2 design to evaluate the original study. Instead of just comparing the outcomes for the medicine pill group and the sugar pill group, they should compare the outcomes while controlling for people’s beliefs about whether they got the medicine. That would mess up their study. It would be asking how effective the medicine is, after removing any effects that allowed patients to realize that they’d gotten the medicine (as if belief-entangled effects couldn’t be physiological effects of the substance).