Know of any randomised trials teaching debiasing in schools? Want to help review them, or would that review help your work?

Last year Elon Musk tweeted a list of cognitive biases saying they should be be taught at a young age.

He got over 325,000 likes, but is he right? We want to find out.

We’re planning a systematic review of randomised trials on debiasing interventions in schools and universities. We think a review like this will help us figure out how best to improve this component of rationality. We hope an academic paper will provide credibility so—if the teaching works—it can be embedded in curricula. If you think that’s worthwhile, there are a few ways you could help. If you think we’ve got come critical blind spots, please let us know too.

Let us know about any randomised trials we might have missed

We’ve collated 10 ‘starter’ references for the review. Basically, the review’s goal will be to ‘find everything like this.’ We can be more sure we’re likely to find everything if we have more of those references up-front. Do you know of any randomised trials testing debiasing? If it’s not on the list, can you add them to this draft protocol or to the comments below?

Let us know if this review has already been done

We’ve found systematic reviews on a range of nearby topics (see Our Rationale), but haven’t seen a systematic review on this topic specifically. If there is a recent systematic review that covers the same territory, we’d be basically wasting our time. We’d love to know that is the case before we dive too deep. So, if you know of any, please let us know in the comments.

If you work on teaching rationality, maybe join the study as an advisor

We want to make sure this review directly helps the work of places like CFAR, Clearer Thinking, SPARC, and so on. If you work at a place like these formally teaching rationality, we’re looking for a few advisors who can help make the review more useful for real-world implementation. If that sounds like you, please email Mike with what you do and why this would be valuable.

If you want to help to make the paper happen, volunteer as a reviewer

If you’re not working at an organisation like that but still want to help, our team at READI Research will be calling on volunteers who want to learn how to do a systematic review. We recently published a meta-review on how to reduce animal product consumption, and have one about promoting charitable donations under second review (preprint here). Both reviews are stacked with amazing authors who volunteered their time to make the review happen (e.g., screening studies, extracting data, assessing the quality of the papers). If you want to join the team for a project like this, please add your details to this form by the 28th of March.

Why do a systematic review of randomised trials on debiasing?

I don’t see many systematic reviews described on LessWong so forgive me if this is condescending, but I don’t want to assume you know what a systematic review is. Across most disciplines—from environmental science to education, policing to policy, health to homelessness—systematic reviews and meta-analyses are considered the most robust evidence for establishing what works because they sit atop the hierarchy of evidence.^[1] Rather than making conclusions on the back of a few—possibly cherry-picked—studies, systematic reviews of randomised trials aim to find all the best studies on an intervention (for example) and pool the effects. Their methods are designed to eliminate as many biases as possible (including publication bias).^[2] The top journals in many fields (e.g., psychology and education) only accept systematic reviews, likely because those reviews are cited heavily and because of the role they play in changing policy and practice.

Systematic reviews have shown we can improve students’ critical thinking abilities in school and at university. As a result, critical thinking is a core part of many curricula. However, to our knowledge, no reviews have explored how well we can teach students to directly overcome biases. Overcoming these biases is an essential component of learning to reason clearly—the skills make up large chunks of the Comprehensive Assessment of Rational Thinking (CART).

Systematic reviews have looked at debiasing training applied to reducing professional biases in social work, medical diagnostics, reducing health-related biases, and mental ill-health among young people. Reviews have assessed the influence of game- and video-based debiasing interventions. These reviews have shown that people can learn about biases, and in some cases, doing so leads to better decisions. But what do we know about debiasing education in schools and universities? Is Elon Musk right? Can we teach young people to reduce the influence of cognitive biases? This review aims to assess the effects of debiasing training in formal educational settings (schools and universities) to identify whether they would add value to the curricula. If you know this has been done, please let us know.

Here are some educational debiasing randomised trials. Know of anything else?

If you know of any randomised trials that aim to teach students to reduce cognitive biases, please add them to the google doc or comment on this post.

A Short Educational Intervention Diminishes Causal Illusions and Specific Paranormal Beliefs in Undergraduates (Barberia et al. 2018)
Implementation and Assessment of an Intervention to Debias Adolescents against Causal Illusions (Barberia et al. 2013)
Is It Time for Studying Real-Life Debiasing? Evaluation of the Effectiveness of an Analogical Intervention Technique (Aczel et al. 2015)
Removing the Blinders: Increasing Students’ Awareness of Self-Perception Biases and Real-World Ethical Challenges Through an Educational Intervention (Tomlin et al. 2021)
Teaching on Video as an Instructional Strategy to Reduce Confirmation Bias—A Pre-Registered Study (van Brussel et al. 2021)
Training Anchoring and Representativeness Bias Mitigation Through a Digital Game (Lee et al. 2016)
Mitigation of Cognitive Bias with a Serious Game: Two Experiments Testing Feedback Timing and Source (Dunbar et al. 2017)
Mitigating Bias Blind Spot via a Serious Videogame (Bessarabova et al. 2016)
Implicit and explicit training in the mitigation of cognitive bias through the use of a serious game (Dunbar et al. 2014)
Improved Learning in US History and Decision Competence with Decision-Focused Curriculum (Jacobson et al. 2012)

Thanks in advance, and let us know in the comments if you have any feedback. If you want to see the concrete details of what’s involved, they’re available in this draft review protocol.

^
Some good examples of reviews applied to policy include the What Works network in the UK. These evidence centres create, curate, and disseminate systematic reviews to help people use what works. They also conduct primary studies to fill in gaps, but usually do that after a review has struggled to draw useful conclusions.
^
The best summary of the methods that systematic reviews use to achieve this is at the Cochrane Collaboration who basically set the standards for evidence-based medicine https://www.cochranelibrary.com/about/about-cochrane-reviews