“However, all authors must walk a thin rope between appeasing the Carl Shulman’s of the world who have read everything and introducing some background for naive readers beyond telling them to simply catch up on their own.”
I don’t need to be appeased, and I strongly endorse the project of providing that introduction. My post was about ways for readers and authors to manage some of the drawbacks of success.
I agree that sample and effect sizes are grossly under-reported, often concealing that an experiment with a sexy conclusion had almost no statistical power, or that an effect was statistically significant but too small to matter. It seems possible for this to become a general science journalism norm, but only if a word-efficient and consistent format for in-text description can be devised, like conflict-of-interest acknowledgments.
How about this? “The study had 27 participants, 15 women and 12 men. The difference between men and women was on average 2 points on a questionnaire ranging from 0 to 100 points.” This clearly explains the (small) sample size and (weak) effect size without requiring any complicated statistics.
Actually, it doesn’t tell you the effect size, since it doesn’t include information about how much individuals in each group differ from each other. If the difference between the group means is 2 points and the standard deviation in each group is 5 points, that’s the same effect size (in the technical Cohen sense) as if the difference is 10 points and the standard deviation is 25 points.
I think a useful way to report data like this might be a variation on, “If you chose one of the women and random and one of the men at random, the probability that the woman would have a higher score would be 53%.”
Aaaand in order not to completely miss the point of the original article, ETA: I’m not sure how much of that suggestion is my own thinking, but I was certainly influenced by reading about the binomial effect size display which solves a related problem in a similar way, and after I had the idea myself I came across something very similar in Rebecca Jordan-Young’s Brainstorm (p.52, and endnote 4 on p.299): mental rotation ability is “considered to be the largest and most reliable gender difference in cognitive ability”; using data from a meta-analysis, she notes that if you tried to guess someone’s gender based on their score in a mental rotation test, using your best strategy you’d get it right 60% of the time. (I checked that math a while ago and got the same result, assuming normal distributions with equal variances in each group, with Cohen’s d=.56; the meta-analysis is Voyer, Voyer & Bryden, 1995.)
It’s annoying that IIRC, “guess the gender” and “in a random pair, who has the higher score” don’t give the same number, though. Average readers will probably just see a percentage in each case and derive some measure of affect from the number, whichever interpretation you give.
It is easier to read comments that include quoted paragraphs when they are quoted with markdown syntax. Instead of using quotation marks for isolated paragraphs use a greater than > sign at the start of the intended quote.
“However, all authors must walk a thin rope between appeasing the Carl Shulman’s of the world who have read everything and introducing some background for naive readers beyond telling them to simply catch up on their own.”
I don’t need to be appeased, and I strongly endorse the project of providing that introduction. My post was about ways for readers and authors to manage some of the drawbacks of success.
I agree that sample and effect sizes are grossly under-reported, often concealing that an experiment with a sexy conclusion had almost no statistical power, or that an effect was statistically significant but too small to matter. It seems possible for this to become a general science journalism norm, but only if a word-efficient and consistent format for in-text description can be devised, like conflict-of-interest acknowledgments.
How about this? “The study had 27 participants, 15 women and 12 men. The difference between men and women was on average 2 points on a questionnaire ranging from 0 to 100 points.” This clearly explains the (small) sample size and (weak) effect size without requiring any complicated statistics.
Actually, it doesn’t tell you the effect size, since it doesn’t include information about how much individuals in each group differ from each other. If the difference between the group means is 2 points and the standard deviation in each group is 5 points, that’s the same effect size (in the technical Cohen sense) as if the difference is 10 points and the standard deviation is 25 points.
I think a useful way to report data like this might be a variation on, “If you chose one of the women and random and one of the men at random, the probability that the woman would have a higher score would be 53%.”
Aaaand in order not to completely miss the point of the original article, ETA: I’m not sure how much of that suggestion is my own thinking, but I was certainly influenced by reading about the binomial effect size display which solves a related problem in a similar way, and after I had the idea myself I came across something very similar in Rebecca Jordan-Young’s Brainstorm (p.52, and endnote 4 on p.299): mental rotation ability is “considered to be the largest and most reliable gender difference in cognitive ability”; using data from a meta-analysis, she notes that if you tried to guess someone’s gender based on their score in a mental rotation test, using your best strategy you’d get it right 60% of the time. (I checked that math a while ago and got the same result, assuming normal distributions with equal variances in each group, with Cohen’s d=.56; the meta-analysis is Voyer, Voyer & Bryden, 1995.)
It’s annoying that IIRC, “guess the gender” and “in a random pair, who has the higher score” don’t give the same number, though. Average readers will probably just see a percentage in each case and derive some measure of affect from the number, whichever interpretation you give.
It is easier to read comments that include quoted paragraphs when they are quoted with markdown syntax. Instead of using quotation marks for isolated paragraphs use a greater than > sign at the start of the intended quote.