Raven’s matrices would seem like an odd choice, since they measure fluid intelligence and basically every attempt to increase fluid intelligence by training has failed. Given that CFAR’s curriculum isn’t specifically targeting fluid intelligence and there’s no reason to expect their interventions to affect it in particular, it would be really unlikely for improvements to show up in Raven’s.
Another thing I’d be particularly interested in is longer term follow-up. It would be impressive if the changes to conscientiousness etc. observed in the 2015 study persist now.
Here are relatively brief responses on these 3 particular points; I’ve made a separate comment which lays out my thinking on metrics like the Big 5 which provides some context for these responses.
We have continued to collect measures like the ones in the 2015 longitudinal study. We are mainly analyzing them in large batches, rather than workshop to workshop, because the sample size isn’t big enough to distinguish signal from noise for single workshops. One of the projects that I’m currently working on is an analysis of a couple years of these data.
The 2017 impact report was not intended as a comprehensive account of all of CFAR’s metrics, it was just focused on CFAR’s EA impact. So it looked at the data that were most directly related to CFAR alums’ impact on the world, and “on average alums have some increase in conscientiousness” seemed less relevant than the information that we did include. The first few paragraphs of the report say more about this.
I’m curious why you’re especially interested in Raven’s Progressive Matrices. I haven’t looked closely at the literature on it, but my impression is that it’s one of many metrics which are loosely related to the thing that we mean by “rationality.” It has the methodological advantage of being a performance score rather than self-report (though this is partially offset by the possibility of practice effects and effort effects). The big disadvantage is the one that Kaj pointed to: it seems to track relatively stable aspects of a person’s thinking skills, and might not change much even if a person made large improvements. For instance, I could imagine a person developing MacGyver-level problem-solving ability while having little or no change in their Raven’s score.
Thanks for this comment; I found it really useful :-)
I’m curious why you’re especially interested in Raven’s Progressive Matrices.
In part interested because it’s a performance measure rather than self-report.
Also speaking from my experience, my performance on tests like Raven’s has been heavily mediated by things that don’t seem directly related to g, and that I’d imagine could be affected by CFAR’s curriculum.
e.g. I perform better on tests like Raven’s when I’m feeling low-anxiety & emotionally cohered. (Seems plausible that CFAR could lower anxiety & increase emotional coherence.)
Thanks, I didn’t know about the 2015 study :-)
Any plans to track the 2015 study measures on a rolling basis? (i.e. for every workshop cohort?) Seems useful to measure performance over time.
Why did the 2017 impact report move away from measuring Big 5 traits? (and the other measures looked at in 2015?)
Any thoughts re: using Raven’s Matrices?
Raven’s matrices would seem like an odd choice, since they measure fluid intelligence and basically every attempt to increase fluid intelligence by training has failed. Given that CFAR’s curriculum isn’t specifically targeting fluid intelligence and there’s no reason to expect their interventions to affect it in particular, it would be really unlikely for improvements to show up in Raven’s.
Another thing I’d be particularly interested in is longer term follow-up. It would be impressive if the changes to conscientiousness etc. observed in the 2015 study persist now.
Here are relatively brief responses on these 3 particular points; I’ve made a separate comment which lays out my thinking on metrics like the Big 5 which provides some context for these responses.
We have continued to collect measures like the ones in the 2015 longitudinal study. We are mainly analyzing them in large batches, rather than workshop to workshop, because the sample size isn’t big enough to distinguish signal from noise for single workshops. One of the projects that I’m currently working on is an analysis of a couple years of these data.
The 2017 impact report was not intended as a comprehensive account of all of CFAR’s metrics, it was just focused on CFAR’s EA impact. So it looked at the data that were most directly related to CFAR alums’ impact on the world, and “on average alums have some increase in conscientiousness” seemed less relevant than the information that we did include. The first few paragraphs of the report say more about this.
I’m curious why you’re especially interested in Raven’s Progressive Matrices. I haven’t looked closely at the literature on it, but my impression is that it’s one of many metrics which are loosely related to the thing that we mean by “rationality.” It has the methodological advantage of being a performance score rather than self-report (though this is partially offset by the possibility of practice effects and effort effects). The big disadvantage is the one that Kaj pointed to: it seems to track relatively stable aspects of a person’s thinking skills, and might not change much even if a person made large improvements. For instance, I could imagine a person developing MacGyver-level problem-solving ability while having little or no change in their Raven’s score.
Thanks for this comment; I found it really useful :-)
In part interested because it’s a performance measure rather than self-report.
Also speaking from my experience, my performance on tests like Raven’s has been heavily mediated by things that don’t seem directly related to g, and that I’d imagine could be affected by CFAR’s curriculum.
e.g. I perform better on tests like Raven’s when I’m feeling low-anxiety & emotionally cohered. (Seems plausible that CFAR could lower anxiety & increase emotional coherence.)