I think it is right to be skeptical of the science around meditation. Meditation perfectly fits into the Bermuda triangle of phenomenon for which our current scientific institutions and practices are not well-prepared to study.
It shares with psychological studies the challenge that the thing under investigation is the internal mental state of the subject. When there are studies with objective endpoints, usually the objective endpoint isn’t the thing we want to get out of it, it’s just a more reliable metric so we know the subjects aren’t fooling themselves. As Science-based Medicine says:
But the more concrete and physiological the outcome, the smaller the placebo effect. Survival from serious forms of cancer, for example, has no demonstrable placebo effect. There is a “clinical trial effect,” as described above – being a subject in a trial tends to improve care and compliance, but no placebo effect beyond that. There is no compelling evidence that mood or thought alone can help fight off cancer or any similar disease.
In the case of meditation, people usually begin the practice to have mental well-being or greater happiness, which is among the outcomes least amenable to reliable objective observation. If it happens to also do something that could be reliably measured with a medical instrument, that would be a bit outside the point.
Meditation shares with nutritional science (also a wrecked landscape of low-quality studies that fail to answer our real questions) that performing the study relies on the subjects to reliably do something with a huge, short-term cost and an uncertain, long-term benefit, which humans are bad at.
High-quality studies on nutritional interventions rarely answer the questions that normal fitness-minded folks want answered, because we want the answer to the question “assuming I perfectly adhere to diet X, what results would I obtain.” Studies can only measure “assuming we take a random sampling of people with varying levels of conscientiousness and investment in their diet, and tell them to do X, what happens” which is too big a difference to be useful.
Similarly with meditation, what meditators want to know is “is it worth my time meditating if I do it approximately perfectly” not “is it worth someone ‘intervening’ to tell me about meditation taking into account the possibility that I’m too lazy to really follow through with it.” The second has more clinical relevance, but less personal relevance for the kind of people on Less Wrong.
All of that is a long precursor to saying “Is there any scientific evidence for benefits of meditation?” and “Are there good reasons for a typical reader of LessWrong to invest their time and effort into meditation practice?” are subtly different questions, so it would be wrong to literally equate them. The second is the answer we really care about, the first is one input which would, if available, fully resolve the question instead of leaving is in a state of uncertainty. You’re entitled to arguments, but not (that particular) proof.
There is objective evidence that meditation does something real (EEG studies of Tibetan monks, for example), but the evidence it does something both real and valuable is probably not up to that standard.
Another, smaller, point I’d like to make is that this post is attempting to perform its own meta-analysis, but with a higher quality bar than academic meta analyses. I don’t think crowdsourcing the best studies of meditation is likely to work this way. If you are interested in running a project to identify the top studies of meditation, I think you would need to identify all the relevant studies, get individuals who are interested in your project to review them, then collate the results. Just asking “the crowd” for the best studies they happen to have on hand I think is likely to fail regardless of what the evidence is.
Thanks for a thoughtful and provoking comment. I wanted to elaborate on my methodology before I start my search, and your comment was an excellent prompt for that.
All of that is a long precursor to saying “Is there any scientific evidence for benefits of meditation?” and “Are there good reasons for a typical reader of LessWrong to invest their time and effort into meditation practice?” are subtly different questions, so it would be wrong to literally equate them.
I agree those are different questions. My purpose in starting this post is gathering scientific data that helps answer the latter question. I admit there might be reasons to invest in meditation practice that are not based on scientifically proven benefits (e.g., curiosity, sense of novelty, sense of belonging to a community). At the same time, I hope that most LW readers attach very little weight to those non-evidence-based reasons to meditate, just like I do.
So I want to answer the first question. The second question reveals my motivation and limits the scope of the first question. For example, I’m not interested in such potential benefits like ‘gaining enough willpower to voluntarily starve yourself to death or voluntarily set yourself aflame’.
I’m glad that you raise this point because I wanted to comment today on my standards of proof anyway.
Let’s start with the easy to verify claims that I generalise as: ‘After only few weeks of regular practice, I sometimes notice impulses when they arise and simply let them fade away, instead of succumbing to them. Sure, often I still act like I used to, but I’m starting to see a change for the better.’ This is very easy to verify. For such claims, I will treat an absence of a strong proof as a strong proof of absence. Here is a sample study protocol (for illustrative purposes only, I don’t claim it’s well thought-out):
1. Gather people who claim to spend too much time compulsively on social media (and in this day and age, who doesn’t?)
2. Give them a smartphone app and a browser extension that tracks how much time they spend on social media.
3. Randomly instruct them to meditate for x minutes a day or lie down for a nap for x minutes a day.
4. Each day ask them on their smartphone whether they did their meditation session/nap time and give them a shame-free, nuisance-free option to answer ‘yes’ or ‘no’.
5. Do some statistics juggling and publish.
There are of course those who claim that the benefits require more investment. For example, the Goenka’s organisation is one of the leading schools of insight meditation and claims that one needs to finish their 10-day retreat and then meditate for a year, two hours a day, in order to reap benefits (or, more precisely, that this is the maximum investment needed before the benefits become apparent). This is still within reach of a sufficiently motivated person, but well beyond what can be tested through a random trial. In that case, a sample study that I would deem sufficient would be:
1. Gather people undertaking their 10-day retreat.
2. Before they start, measure the metrics of interest (more on that later)
3. After the retreat, poll them periodically (e.g., monthly or weekly) to see whether they maintain their 2 hours a day practice.
4. After a year measure again the metrics of interest.
5. Do some statistics juggling and publish.
If the benefits require even greater commitment, I would argue that any evidence of such benefits becomes irrelevant, thanks to the second question. Even if you promise me living the rest of my days in a state of never-ending orgasm, on the condition that I first spend a few years locked up in a monastery, I’m not going to do that.
In the case of meditation, people usually begin the practice to have mental well-being or greater happiness
Do you have some sources to back this up? I’ve heard many declared reasons why people begin their meditation practice, and it was quite a diverse set, none seemed dominant. But my sample was never representative. Until proven… er, until argumented otherwise, I reject this assumption.
which is among the outcomes least amenable to reliable objective observation.
(Side note: it would be fascinating to see whether those measures diverge more after a meditation training. That is, whether meditators report higher well-being but don’t show the usual signs of it.)
(Confession: I have only skimmed through that paper, after finding it referenced in The Happiness Lab)
If it happens to also do something that could be reliably measured with a medical instrument, that would be a bit outside the point.
I don’t buy this at all. If the only observable benefit of me meditating is that I used to self-report average well-being of 5.17 out of 10, and now I self-report 7.39 on average, but:
my measurable indicators of stress levels and anxiety remain unchanged
there is no significant change to my relationships
people around me don’t see me being any happier
there is no change in my addictive/compulsive behaviours
the frequency of me using the word ‘d**khead’ in my tweets has not dropped
Then:
1. I don’t find this change of self-reported well-being something that’s worth pursuing at all.
2. Even if I wanted to pursue it, I can achieve it with less demanding training protocol: Install an app for spaced repetition and commit to memory: ‘Whenever asked, claim that you are super happy’. I can achieve this lasting result in probably less than 5 minutes of total investment.
Meditation shares with nutritional science (also a wrecked landscape of low-quality studies that fail to answer our real questions) that performing the study relies on the subjects to reliably do something with a huge, short-term cost and an uncertain, long-term benefit
I agree that nutritional science is a mess but I disagree with your diagnosis of why that is the case. There are several characteristics of nutrition and eating that make scientific scrutiny very difficult, and those characteristics are not shared with meditation:
What we eat is largely determined by the environment we find ourselves in (e.g, it’s nearly impossible for me to have a fresh salad if I don’t have a fridge in the office or an affordable store selling good quality salads nearby). On the other hand, most of us have the power to decide whether to meditate at some point during the day or not. (Ok, I guess parents of small children should be excluded from any meditation studies.)
We eat many times a day, almost always with different parameters. So accurate tracking is extremely tedious and would probably require carrying a scale with you at all times, and demanding detailed datasheets from all the restaurants you visit. On the other hand, we can reliably tell how much time we spend meditating. (Situation would be much more difficult if the meditation teachers suggested that for example we meditate for 30 seconds, within every 5-minute period. Luckily, meditation recommendations seem to focus on designated meditation sessions, with clear start and finish.) Also, society is not going to shame you for breathing without being aware of it, so you have less incentive to lie.
We have habits pertaining to eating. Some of those are even unconscious. Imposing changes on our eating habits might cause significant suffering that persists until the change is completely abandoned. You are not going to be grumpy for an entire day, feel pain in your stomach and fantasise about being un-mindful just because you promised a researcher to observe your breath for 20 minutes later today.
Another, smaller, point I’d like to make is that this post is attempting to perform its own meta-analysis, but with a higher quality bar than academic meta analyses. I don’t think crowdsourcing the best studies of meditation is likely to work this way. If you are interested in running a project to identify the top studies of meditation, I think you would need to identify all the relevant studies, get individuals who are interested in your project to review them, then collate the results. Just asking “the crowd” for the best studies they happen to have on hand I think is likely to fail regardless of what the evidence is.
That’s a valid point. Crowdsourcing meta-analysis would be great but that’s not my intention. I expect this post to turn into my own notebook. I still expect that this will be beneficial to me because:
I hate writing, and in this format I can split a long meta-analysis article into smaller pieces, published as soon as they are ready, which makes it more bearable.
I still expect that some people will read what I write, and keep me in check if I write something stupid or biased.
While I don’t expect to be flooded by high-quality studies, I do hope that people will suggest new ways of how I might find them.
The added motivation of a possible ‘I told you so’ at the end of this road.
Increased confidence in my findings, so that hopefully I don’t revisit this subject for the third time in a few years.
I admit there might be reasons to invest in meditation practice that are not based on scientifically proven benefits (e.g., curiosity, sense of novelty, sense of belonging to a community). At the same time, I hope that most LW readers attach very little weight to those non-evidence-based reasons to meditate, just like I do.
I suppose I should admit the main reason I started meditating a long time ago was curiosity. I read Mastering the Core Teachings of the Buddha (reviewed on SSC here) and thought “well, this person sounds like they are explaining mental states that seem pretty unbelievable to me, I wonder if this is all BS.” I was/am mentally healthy and emotionally stable more than the average person. I don’t meditate that consistently anymore, only when things are more stressful than usual. Having it in the toolbox, like fitness, is enough for me. I did enough practice to know that what MCTB is pointing at is a real phenomenon, but that’s it. I actually think that viewing it as a hobby is the healthiest way to approach the kind of serious practice needed for enlightenment.
Let’s start with the easy to verify claims that I generalise as...
In my experience, these claims are false. I occasionally tried to use mindfulness to help me with dieting or exercising, since those are also things I do, and it never helped in a way I could discern.
Do you have some sources to back this up? I’ve heard many declared reasons why people begin their meditation practice, and it was quite a diverse set, none seemed dominant.
Thank you for challenging me on this, that was based only on personal observation, which as I admit above doesn’t even square with my own experience! This survey has concrete data on why people meditate in Fig. 1. The top reason is “General wellness and general disease prevention.” None of them are specifically happiness related, so maybe that’s an overly specific claim.
I don’t buy this at all. If the only observable benefit of me meditating is that I used to self-report average well-being of 5.17 out of 10, and now I self-report 7.39 on average
Based on my mental model of meditation, you probably would be dissatisfied with the results. In section IV of the post above, Scott Alexander summarizes thus:
Ingram dedicates himself hard to debunking a lot of the things people would use to fill the gap. Pages 261-328 discuss the various claims Buddhist schools have made about enlightenment, mostly to deny them all. He has nothing but contempt for the obviously silly ones, like how enlightened people can fly around and zap you with their third eyes. But he’s equally dismissive of things that sort of seem like the basics. He denies claims about how enlightened people can’t get angry, or effortlessly resist temptation, or feel universal unconditional love, or things like that. Some of this he supports with stories of enlightened leaders behaving badly; other times he cites himself as an enlightened person who frequently experiences anger, pain, and the like. Once he’s stripped everything else away, he says the only thing one can say about enlightenment is that it grants a powerful true experience of the non-dual nature of the world. [9eb1: I’ve excluded the possible counterargument here for brevity]
There are external benefits I think meditation has given me that feel like they are real, but the effect size is too small for studies to realistically find them. I can fall asleep reliably by using meditation as a tool. I can tactically break my own rumination thought cycles by meditating as a tool (or I can workout, but sometimes you’ve already worked out that day). I definitely feel like I am harder to surprise (lack of “jump”), but that’s not a particularly practical superpower.
I’ve meditated for >300 hours (maybe 4 or 5). I don’t regret my hours. It is a hobby, it satisfies my curiosity, it makes me happy when I need it. Lack of personal transformation is fine.
To be clear, my values with regard to self-rated wellness are different from yours. I am glad to improve my self-rated wellness even if it has no measurable outward impact on my behavior. My happiness is super important to me. If I move the needle on that, that’s great even if I’m still an asshole. I have no interest in being a miserable saint.
There are several characteristics of nutrition and eating that make scientific scrutiny very difficult, and those characteristics are not shared with meditation
Those differences are subsumed in the “high short-term costs” side of my statement. The exact costs are different, that’s all. You can tell people not to do all the things you mentioned during a diet study, but they won’t follow your instruction.
I think it is right to be skeptical of the science around meditation. Meditation perfectly fits into the Bermuda triangle of phenomenon for which our current scientific institutions and practices are not well-prepared to study.
It shares with psychological studies the challenge that the thing under investigation is the internal mental state of the subject. When there are studies with objective endpoints, usually the objective endpoint isn’t the thing we want to get out of it, it’s just a more reliable metric so we know the subjects aren’t fooling themselves. As Science-based Medicine says:
In the case of meditation, people usually begin the practice to have mental well-being or greater happiness, which is among the outcomes least amenable to reliable objective observation. If it happens to also do something that could be reliably measured with a medical instrument, that would be a bit outside the point.
Meditation shares with nutritional science (also a wrecked landscape of low-quality studies that fail to answer our real questions) that performing the study relies on the subjects to reliably do something with a huge, short-term cost and an uncertain, long-term benefit, which humans are bad at.
High-quality studies on nutritional interventions rarely answer the questions that normal fitness-minded folks want answered, because we want the answer to the question “assuming I perfectly adhere to diet X, what results would I obtain.” Studies can only measure “assuming we take a random sampling of people with varying levels of conscientiousness and investment in their diet, and tell them to do X, what happens” which is too big a difference to be useful.
Similarly with meditation, what meditators want to know is “is it worth my time meditating if I do it approximately perfectly” not “is it worth someone ‘intervening’ to tell me about meditation taking into account the possibility that I’m too lazy to really follow through with it.” The second has more clinical relevance, but less personal relevance for the kind of people on Less Wrong.
All of that is a long precursor to saying “Is there any scientific evidence for benefits of meditation?” and “Are there good reasons for a typical reader of LessWrong to invest their time and effort into meditation practice?” are subtly different questions, so it would be wrong to literally equate them. The second is the answer we really care about, the first is one input which would, if available, fully resolve the question instead of leaving is in a state of uncertainty. You’re entitled to arguments, but not (that particular) proof.
There is objective evidence that meditation does something real (EEG studies of Tibetan monks, for example), but the evidence it does something both real and valuable is probably not up to that standard.
Another, smaller, point I’d like to make is that this post is attempting to perform its own meta-analysis, but with a higher quality bar than academic meta analyses. I don’t think crowdsourcing the best studies of meditation is likely to work this way. If you are interested in running a project to identify the top studies of meditation, I think you would need to identify all the relevant studies, get individuals who are interested in your project to review them, then collate the results. Just asking “the crowd” for the best studies they happen to have on hand I think is likely to fail regardless of what the evidence is.
Thanks for a thoughtful and provoking comment. I wanted to elaborate on my methodology before I start my search, and your comment was an excellent prompt for that.
I agree those are different questions. My purpose in starting this post is gathering scientific data that helps answer the latter question. I admit there might be reasons to invest in meditation practice that are not based on scientifically proven benefits (e.g., curiosity, sense of novelty, sense of belonging to a community). At the same time, I hope that most LW readers attach very little weight to those non-evidence-based reasons to meditate, just like I do.
So I want to answer the first question. The second question reveals my motivation and limits the scope of the first question. For example, I’m not interested in such potential benefits like ‘gaining enough willpower to voluntarily starve yourself to death or voluntarily set yourself aflame’.
I’m glad that you raise this point because I wanted to comment today on my standards of proof anyway.
Let’s start with the easy to verify claims that I generalise as: ‘After only few weeks of regular practice, I sometimes notice impulses when they arise and simply let them fade away, instead of succumbing to them. Sure, often I still act like I used to, but I’m starting to see a change for the better.’ This is very easy to verify. For such claims, I will treat an absence of a strong proof as a strong proof of absence. Here is a sample study protocol (for illustrative purposes only, I don’t claim it’s well thought-out):
1. Gather people who claim to spend too much time compulsively on social media (and in this day and age, who doesn’t?)
2. Give them a smartphone app and a browser extension that tracks how much time they spend on social media.
3. Randomly instruct them to meditate for x minutes a day or lie down for a nap for x minutes a day.
4. Each day ask them on their smartphone whether they did their meditation session/nap time and give them a shame-free, nuisance-free option to answer ‘yes’ or ‘no’.
5. Do some statistics juggling and publish.
There are of course those who claim that the benefits require more investment. For example, the Goenka’s organisation is one of the leading schools of insight meditation and claims that one needs to finish their 10-day retreat and then meditate for a year, two hours a day, in order to reap benefits (or, more precisely, that this is the maximum investment needed before the benefits become apparent). This is still within reach of a sufficiently motivated person, but well beyond what can be tested through a random trial. In that case, a sample study that I would deem sufficient would be:
1. Gather people undertaking their 10-day retreat.
2. Before they start, measure the metrics of interest (more on that later)
3. After the retreat, poll them periodically (e.g., monthly or weekly) to see whether they maintain their 2 hours a day practice.
4. After a year measure again the metrics of interest.
5. Do some statistics juggling and publish.
If the benefits require even greater commitment, I would argue that any evidence of such benefits becomes irrelevant, thanks to the second question. Even if you promise me living the rest of my days in a state of never-ending orgasm, on the condition that I first spend a few years locked up in a monastery, I’m not going to do that.
Do you have some sources to back this up? I’ve heard many declared reasons why people begin their meditation practice, and it was quite a diverse set, none seemed dominant. But my sample was never representative. Until proven… er, until argumented otherwise, I reject this assumption.
As per Sandvik, E., Diener, E., & Seidlitz, L. (1993). Subjective well-being: The convergence and stability of self-report and non-self-report measures. Journal of Personality, 61(3), 317-342, there are other measures that are well correlated with subjective well-being under normal circumstances. For example, you can simply ask subject’s friends and family how happy they seem. Or you can request the subject to recall negative and positive events from their life and see how many they come up with in a short amount of time.
(Side note: it would be fascinating to see whether those measures diverge more after a meditation training. That is, whether meditators report higher well-being but don’t show the usual signs of it.)
(Confession: I have only skimmed through that paper, after finding it referenced in The Happiness Lab)
I don’t buy this at all. If the only observable benefit of me meditating is that I used to self-report average well-being of 5.17 out of 10, and now I self-report 7.39 on average, but:
my measurable indicators of stress levels and anxiety remain unchanged
there is no significant change to my relationships
people around me don’t see me being any happier
there is no change in my addictive/compulsive behaviours
the frequency of me using the word ‘d**khead’ in my tweets has not dropped
Then:
1. I don’t find this change of self-reported well-being something that’s worth pursuing at all.
2. Even if I wanted to pursue it, I can achieve it with less demanding training protocol: Install an app for spaced repetition and commit to memory: ‘Whenever asked, claim that you are super happy’. I can achieve this lasting result in probably less than 5 minutes of total investment.
I agree that nutritional science is a mess but I disagree with your diagnosis of why that is the case. There are several characteristics of nutrition and eating that make scientific scrutiny very difficult, and those characteristics are not shared with meditation:
What we eat is largely determined by the environment we find ourselves in (e.g, it’s nearly impossible for me to have a fresh salad if I don’t have a fridge in the office or an affordable store selling good quality salads nearby). On the other hand, most of us have the power to decide whether to meditate at some point during the day or not. (Ok, I guess parents of small children should be excluded from any meditation studies.)
We eat many times a day, almost always with different parameters. So accurate tracking is extremely tedious and would probably require carrying a scale with you at all times, and demanding detailed datasheets from all the restaurants you visit. On the other hand, we can reliably tell how much time we spend meditating. (Situation would be much more difficult if the meditation teachers suggested that for example we meditate for 30 seconds, within every 5-minute period. Luckily, meditation recommendations seem to focus on designated meditation sessions, with clear start and finish.) Also, society is not going to shame you for breathing without being aware of it, so you have less incentive to lie.
We have habits pertaining to eating. Some of those are even unconscious. Imposing changes on our eating habits might cause significant suffering that persists until the change is completely abandoned. You are not going to be grumpy for an entire day, feel pain in your stomach and fantasise about being un-mindful just because you promised a researcher to observe your breath for 20 minutes later today.
That’s a valid point. Crowdsourcing meta-analysis would be great but that’s not my intention. I expect this post to turn into my own notebook. I still expect that this will be beneficial to me because:
I hate writing, and in this format I can split a long meta-analysis article into smaller pieces, published as soon as they are ready, which makes it more bearable.
I still expect that some people will read what I write, and keep me in check if I write something stupid or biased.
While I don’t expect to be flooded by high-quality studies, I do hope that people will suggest new ways of how I might find them.
The added motivation of a possible ‘I told you so’ at the end of this road.
Increased confidence in my findings, so that hopefully I don’t revisit this subject for the third time in a few years.
I suppose I should admit the main reason I started meditating a long time ago was curiosity. I read Mastering the Core Teachings of the Buddha (reviewed on SSC here) and thought “well, this person sounds like they are explaining mental states that seem pretty unbelievable to me, I wonder if this is all BS.” I was/am mentally healthy and emotionally stable more than the average person. I don’t meditate that consistently anymore, only when things are more stressful than usual. Having it in the toolbox, like fitness, is enough for me. I did enough practice to know that what MCTB is pointing at is a real phenomenon, but that’s it. I actually think that viewing it as a hobby is the healthiest way to approach the kind of serious practice needed for enlightenment.
In my experience, these claims are false. I occasionally tried to use mindfulness to help me with dieting or exercising, since those are also things I do, and it never helped in a way I could discern.
Thank you for challenging me on this, that was based only on personal observation, which as I admit above doesn’t even square with my own experience! This survey has concrete data on why people meditate in Fig. 1. The top reason is “General wellness and general disease prevention.” None of them are specifically happiness related, so maybe that’s an overly specific claim.
Based on my mental model of meditation, you probably would be dissatisfied with the results. In section IV of the post above, Scott Alexander summarizes thus:
There are external benefits I think meditation has given me that feel like they are real, but the effect size is too small for studies to realistically find them. I can fall asleep reliably by using meditation as a tool. I can tactically break my own rumination thought cycles by meditating as a tool (or I can workout, but sometimes you’ve already worked out that day). I definitely feel like I am harder to surprise (lack of “jump”), but that’s not a particularly practical superpower.
I’ve meditated for >300 hours (maybe 4 or 5). I don’t regret my hours. It is a hobby, it satisfies my curiosity, it makes me happy when I need it. Lack of personal transformation is fine.
To be clear, my values with regard to self-rated wellness are different from yours. I am glad to improve my self-rated wellness even if it has no measurable outward impact on my behavior. My happiness is super important to me. If I move the needle on that, that’s great even if I’m still an asshole. I have no interest in being a miserable saint.
Those differences are subsumed in the “high short-term costs” side of my statement. The exact costs are different, that’s all. You can tell people not to do all the things you mentioned during a diet study, but they won’t follow your instruction.