For people ignorant of statistics (like I am), I checked and d measures the change of mean between two distributions, proportional to the variance (i.e. (mean1−mean2)÷standardDeviation).
An effect size in the range 0.2−0.5 is considered ‘small’ by Cohen, and conscientiousness was increased (with p<0.001) with d=0.24 which seems to me notable for such a small intervention (~4 days), though I haven’t got much intuition around these things and maybe it’s actually quite a big intervention. Neuroticism decreased with d=0.17.
If you want to jump to the bit in the above article that lists all the effect sizes measured, search for ‘The table below summarizes the study results’. The biggest effect sizes were in the places (I think) where CFAR tried to come up with their own metrics to measure what they’re trying to do better.
The biggest effect size was on one of these measures, for the metric ‘Emotions Help Rather Than Hinder’, and I’ll quote that section in full below.
Emotions Help Rather Than Hinder
We created a single item measure of how participants relate to their emotions: “When you feel emotions, do they mostly help or hinder you in pursuing your goals?” The CFAR workshop emphasizes the value of emotions as sources of data and motivation, and includes techniques for increasing the alignment between one’s emotions, goals, and behavior. Research on emotion regulation provides some of the theoretical background for this approach, with the findings that it generally is not helpful to suppress emotions and it often is helpful to reframe situations so that they elicit different emotions (Gross, 2002).
There was an increase in the extent to which participants evaluated their emotions as helping them rather than hindering them, d = 0.41 (t(129) = 4.13, p < .001).
I feel like I’ll be bringing this up in conversation any time that anyone says the rationality community is too focused on hyper-logical/S2 thinking.
Anyway, this is just me highlighting some stuff I found interesting, to overcome trivial inconveniences for others, but there’s a ton more interesting stuff in the post.
I think its very confusing to call d = 0.2 to 0.5 ‘small’, especially in the context of a 4 day workshop. Imagine the variable is IQ. Then a ‘small’ effect increases iq by 3 to 7.5 points. That boost in iq would be much better described as ‘huge’. However IQ has a relatively large standard deviation compared to its mean (roughly 15 and 100).
Lets look at male height. In the USA male height has a mean around 70 inches and a standard deviation around 4 inches. (Note 4⁄70 is 38% of 15⁄100). A d of 0.2 to 0.5 would correspond to an increase in height of 0.8 to 2 inches. Some people are willing to undergo costly, time consuming and painful length lengthening surgery to gain 4-5 inches of height. If a four day, 4000 dollar workshop gave increased your height by 0.8 to 2 inches millions of men would be on the waiting list. I know I would be. That doesnt really sound ‘small’ to me.
IQ has a relatively large standard deviation compared to its mean
No, the mean here is an arbitrary convention, so 15 and 100 don’t tell us anything relevant. The appropriate comparison is to what other interventions have accomplished.
For people ignorant of statistics (like I am), I checked and d measures the change of mean between two distributions, proportional to the variance (i.e. (mean1−mean2)÷standardDeviation).
An effect size in the range 0.2−0.5 is considered ‘small’ by Cohen, and conscientiousness was increased (with p<0.001) with d=0.24 which seems to me notable for such a small intervention (~4 days), though I haven’t got much intuition around these things and maybe it’s actually quite a big intervention. Neuroticism decreased with d=0.17.
If you want to jump to the bit in the above article that lists all the effect sizes measured, search for ‘The table below summarizes the study results’. The biggest effect sizes were in the places (I think) where CFAR tried to come up with their own metrics to measure what they’re trying to do better.
The biggest effect size was on one of these measures, for the metric ‘Emotions Help Rather Than Hinder’, and I’ll quote that section in full below.
I feel like I’ll be bringing this up in conversation any time that anyone says the rationality community is too focused on hyper-logical/S2 thinking.
Anyway, this is just me highlighting some stuff I found interesting, to overcome trivial inconveniences for others, but there’s a ton more interesting stuff in the post.
I think its very confusing to call d = 0.2 to 0.5 ‘small’, especially in the context of a 4 day workshop. Imagine the variable is IQ. Then a ‘small’ effect increases iq by 3 to 7.5 points. That boost in iq would be much better described as ‘huge’. However IQ has a relatively large standard deviation compared to its mean (roughly 15 and 100).
Lets look at male height. In the USA male height has a mean around 70 inches and a standard deviation around 4 inches. (Note 4⁄70 is 38% of 15⁄100). A d of 0.2 to 0.5 would correspond to an increase in height of 0.8 to 2 inches. Some people are willing to undergo costly, time consuming and painful length lengthening surgery to gain 4-5 inches of height. If a four day, 4000 dollar workshop gave increased your height by 0.8 to 2 inches millions of men would be on the waiting list. I know I would be. That doesnt really sound ‘small’ to me.
No, the mean here is an arbitrary convention, so 15 and 100 don’t tell us anything relevant. The appropriate comparison is to what other interventions have accomplished.
Here’s a nice visualisation of what d represents.
For an effect size of 0.4, this means that 66% of the study group will have results above the 50th percentile of the control.