The effect size I quoted includes studies in which cognitive testing was done in the sleep inertia period(and, in fact, they account for about 30% of the weight), and the mean nap duration among the studies in that meta-analysis was 55.4 ± 29.4 min (quite a bit longer than 20-30 minutes), so I don’t think that was selective reporting.
A second selective reporting complaint that I would make about how the researchers framed their findings is that after outlier exclusion or after analyzing only RCTs, their global effect size drops from 0.18 to 0.07-0.08, with a 95% CI on both sides of 0. However, when describing this result, the researchers consistently, in both the abstract and in the sensitivity analysis section, refer to this as “similar results,” without qualification and without reprinting the effect size number.
To be fair, the only RCT model still finds a moderate effect size 61-120 minutes after napping, which tracks with the common sense idea that a nap can temporarily boost cognitive function once the sleep inertial period is over. It’s odd, though, that their global model finds that napping provides the biggest benefits to alertness, while their only RCT analysis finds the worst results for napping on alertness, with the central estimate showing a negative impact.
Possibly, this can all be explained if the few RCTs they found focused on testing less than 30 minutes after the nap, so that the apparent effect of napping on performance would be diminished. I haven’t looked at them, so I don’t want to be too strident in this critique.
their only RCT analysis finds the worst results for napping on alertness, with the central estimate showing a negative impact
What you mean is that it showed a difference between the napping and control groups at t1 that favored the control group. This seems to have been due to baseline differences between the groups, because when you look at how the mean alertness of each group changed in the RCT analysis, they both improved, with that of the nap group having improved a bit more:
That’s a good catch. So it sounds like they’re comparing the “effects of a nap” by comparing the control group performance at t1 with the nap group performance at t1, rather than the difference in the magnitude of improvement between the groups from time t0 to t1?
Example:
Controls score 1 for alertness at time t0, and a 3 at time t1.
Nappers score a −1 for alertness at time t0, and a 2 at time t1.
The nappers improved by 3, while the controls only improved by 2.
Yet the controls scored higher at t1, because they started at higher baseline alertness.
If this is what’s going on, my credence in napping being effective is increased. The gains in figure S4 vs. S7 are easy to discern, just by eye.
So it sounds like they’re comparing the “effects of a nap” by comparing the control group performance at t1 with the nap group performance at t1, rather than the difference in the magnitude of improvement between the groups from time t0 to t1?
Yeah, that is what Figure 4 (which you attached in your comment) shows, and what the effect sizes reported in the abstract refer to. There does seem to be a trend towards nap groups across the studies having a bit lower baseline cognitive performance:
Figure S6 directly compares the change in performance of the groups, though the effect sizes are harder to interpret in it.
The effect size I quoted includes studies in which cognitive testing was done in the sleep inertia period (and, in fact, they account for about 30% of the weight), and the mean nap duration among the studies in that meta-analysis was 55.4 ± 29.4 min (quite a bit longer than 20-30 minutes), so I don’t think that was selective reporting.
That’s a fair objection.
A second selective reporting complaint that I would make about how the researchers framed their findings is that after outlier exclusion or after analyzing only RCTs, their global effect size drops from 0.18 to 0.07-0.08, with a 95% CI on both sides of 0. However, when describing this result, the researchers consistently, in both the abstract and in the sensitivity analysis section, refer to this as “similar results,” without qualification and without reprinting the effect size number.
To be fair, the only RCT model still finds a moderate effect size 61-120 minutes after napping, which tracks with the common sense idea that a nap can temporarily boost cognitive function once the sleep inertial period is over. It’s odd, though, that their global model finds that napping provides the biggest benefits to alertness, while their only RCT analysis finds the worst results for napping on alertness, with the central estimate showing a negative impact.
Possibly, this can all be explained if the few RCTs they found focused on testing less than 30 minutes after the nap, so that the apparent effect of napping on performance would be diminished. I haven’t looked at them, so I don’t want to be too strident in this critique.
What you mean is that it showed a difference between the napping and control groups at t1 that favored the control group. This seems to have been due to baseline differences between the groups, because when you look at how the mean alertness of each group changed in the RCT analysis, they both improved, with that of the nap group having improved a bit more:
That’s a good catch. So it sounds like they’re comparing the “effects of a nap” by comparing the control group performance at t1 with the nap group performance at t1, rather than the difference in the magnitude of improvement between the groups from time t0 to t1?
Example:
Controls score 1 for alertness at time t0, and a 3 at time t1.
Nappers score a −1 for alertness at time t0, and a 2 at time t1.
The nappers improved by 3, while the controls only improved by 2.
Yet the controls scored higher at t1, because they started at higher baseline alertness.
If this is what’s going on, my credence in napping being effective is increased. The gains in figure S4 vs. S7 are easy to discern, just by eye.
Yeah, that is what Figure 4 (which you attached in your comment) shows, and what the effect sizes reported in the abstract refer to. There does seem to be a trend towards nap groups across the studies having a bit lower baseline cognitive performance:
Figure S6 directly compares the change in performance of the groups, though the effect sizes are harder to interpret in it.
Thanks for digging out the supplemental figures! Interesting...