In the process, we’re breaking new ground in studying the long-term effects of rationality training on life outcomes using randomized controlled trials.
Despite CFAR’s 2-3 year existence (probably longer informally, as well) they have yet to publish a single paper on these “randomized controlled trials”. I would advise not donating until they make good on their claims.
edit: I’ve also made some notes on CFAR and their use of science as an applause light in previous comments.
Yes, small ones—but not peer-reviewed or published (which is much harder). We did a tiny randomized admissions experiment early on (with the admissions in summer 2012, the follow-up surveys one year later), and a small randomized online experiment; we will probably run another randomized admissions set sometime in 2015, w/ results coming in in 2016. (In some years, if/when we get to a point where funding is more secure, we will likely do more.)
Yep; that would be good to do, and I suspect we’ll get to it; I’m a bit embarrassed that we haven’t done it, but we are in fact getting better with time at doing the things that it’s embarrassing not to do, while also prioritizing the few things that most yield compound progress.
One of the future premises of CFAR is that we can eventually apply the full scientific method to the problem of constructing a rationality curriculum (by measuring variations, counting things, re-testing, etc.) -- we aim to eventually be an evidence-based organization. In our present state this continues to be a lot harder than we would like; and our 2014 workshop, for example, was done via crude “what do you feel you learnt?” surveys and our own gut impressions.
Ok. That’s a little more worrisome. So how much of that situation is itself caused by lack of funding and the currently small nature of the organization?
The response was directed towards you, I should have maybe phrased it as adding something like “We should then ask” before the question. If this is caused by a lack of funding then it isn’t by itself that much of a worry.
On CFAR’s front page:
Despite CFAR’s 2-3 year existence (probably longer informally, as well) they have yet to publish a single paper on these “randomized controlled trials”. I would advise not donating until they make good on their claims.
edit: I’ve also made some notes on CFAR and their use of science as an applause light in previous comments.
Our vision page sure is out of date there; which I agree reflects badly on us. We should not make that claim at this time.
I do suspect we’re a good use of donation, though, for reasons discussed in the links above; happy to engage on specifics.
Are there any plans to try randomized controlled trials in the future?
Yes, small ones—but not peer-reviewed or published (which is much harder). We did a tiny randomized admissions experiment early on (with the admissions in summer 2012, the follow-up surveys one year later), and a small randomized online experiment; we will probably run another randomized admissions set sometime in 2015, w/ results coming in in 2016. (In some years, if/when we get to a point where funding is more secure, we will likely do more.)
Is there a chance that a written up version of these small results will get posted on the CFAR website?
Yep; that would be good to do, and I suspect we’ll get to it; I’m a bit embarrassed that we haven’t done it, but we are in fact getting better with time at doing the things that it’s embarrassing not to do, while also prioritizing the few things that most yield compound progress.
2-3 years seems like a reasonable time span to not have published if one is trying to measure some sort of long-term effect.
As noted in http://lesswrong.com/lw/lfg/cfar_in_2014_continuing_to_climb_out_of_the/, they haven’t even started yet. Also, just replicating a study they cite in their rationality training would be a good step.
Ok. That’s a little more worrisome. So how much of that situation is itself caused by lack of funding and the currently small nature of the organization?
I’m not sure if this response was directed towards me, because I don’t know what their reasonings are.
The response was directed towards you, I should have maybe phrased it as adding something like “We should then ask” before the question. If this is caused by a lack of funding then it isn’t by itself that much of a worry.