So far as I can tell, the central question Elizabeth has been trying to answer is “Do the people who convert to veganism because they get involved in EA have systemic health problems?” Those health problems might be easily solvable with supplementation (Great!), systemic to having a fully vegan diet but only requires some modest amount of animal product, or something more complicated. She has several self-reported people coming to her saying they tried veganism, had health problems, and stopped. So, “At what rate do vegans desist for health reasons?” seems like an important question to me. It will tell you at least some of what you are missing when surveying current vegans only.
Analogously, a survey of healing crystal buyers doesn’t reliably tell us whether healing crystals improve health. Even if such a survey is useful for explaining motives, it’s clearly less valuable than an RCT when it comes to the important question of whether they actually work.
I agree that if your prior probability of something being true is near 0, you need very strong evidence to update. Was your prior probability that someone would desist from the vegan diet for health reasons actually that low? If not, why is the crystal healing metaphor analogous?
So, “At what rate do vegans desist for health reasons?” seems like an important question to me. It will tell you at least some of what you are missing when surveying current vegans only.
As I argued in my original comment, self-reported data is unreliable for answering this question. I simply do not trust people’s ability to attribute the causal impact of diets on their health. Separately, I think people frequently misreport their motives. Even if vegan diets caused no health effects, a substantial fraction of people could still report desisting from veganism for health reasons.
I’m honestly not sure why think that self-reported data is more reliable than proper scientific studies like RTCs when trying to shed light on this question. The RCTs should be better able to tell us the actual health effects of adopting veganism, which is key to understanding how many people would be forced to abandon the diet for health reasons.
As I argued in my original comment, self-reported data is unreliable for answering this question. I simply do not trust people’s ability to attribute the causal impact of diets on their health.
It seems to me like even if this isn’t relevant to our beliefs about underlying reality (“teachers think criticism works because they’re not correcting for regression to the mean, and so we can’t rely on teacher’s beliefs”) it should be relevant to our beliefs about individual decision-making (“when people desist from vegan diets, 20% of the time it’s because they think their health is worse”), which should direct our views on advocacy (if we want fewer people to desist from vegan diets (and possibly then dissuade others from trying them), we should try to make sure they don’t think their health is worse on them).
I’m honestly not sure why think that self-reported data is more reliable than proper scientific studies like RTCs when trying to shed light on this question.
Often there’s question-substitution going on. If the proper scientific studies are measuring easily quantifiable things like blood pressure and what people care about more / make decisions based off of is the difficult-to-quantify “how much pep is in my step”, then the improper survey may point more directly at the thing that’s relevant, even if it does so less precisely.
The question I assumed Stephen was asking (and at least my question for myself) here is, “okay, but what do we believe in the meanwhile?”.
Natalia responded with a process that might find some good evidence (but, might not, and looks like at least several hours of skilled-labor search to find out). I agree someone should do that labor and find out if better evidence exists.
I also realize Vaniver did explicitly ask “what alternate framework you prefer?” and it makes sense that your framework is interested in different questions than mine or Elizabeths or Stephens or whatnot. But, for me, the question is “what should vegan activist’s best guess be right now”, not “what might it turn out to be after doing a bunch more research that maybe turns out to have good data and maybe doesn’t.”
for me, the question is “what should vegan activist’s best guess be right now”
This is fair and a completely reasonable question to ask, even if we agree that the self-reported data is unreliable. I agree the self-reported data could be a useful first step towards answering this question if we had almost no other information. I also haven’t looked deeply into the RCTs and scientific data and so I don’t have a confident view on the health value of vegan diets.
On the other hand, personally, my understanding is that multiple mainstream scientific institutions say that vegan diets are generally healthy (in the vast majority of cases) unless you don’t take the proper supplementation regularly. If you put a gun to my head right now and asked me to submit my beliefs about this question, I would defer heavily to (my perception of) the mainstream consensus, rather than the self-reported data. That’s not because I think mainstream consensus isn’t sometimes wrong or biased, but—in the spirit of your question—that’s just what I think is most reliable out of all the easily accessible facts that I have available right now, including the self-reported data.
To be clear, I didn’t really want to take this line, and talk about how The Experts disagree with Elizabeth, and so the burden of proof is on her rather than me, because that’s often a conversation stopper and not helpful for fruitful discussion, especially given my relative ignorance about the empirical data. But if we’re interested in what a good “best guess” should be, then yes, I think mainstream scientific institutions are generally reliable on questions they have strong opinions about. That’s not my response to everything in this discussion, but it’s my response to your specific point about what we should believe in the meantime.
In this case I don’t think the claim you’re ascribing to the experts are Elizabeth are actually in conflict. You say:
vegan diets are generally healthy (in the vast majority of cases) unless you don’t take the proper supplementation regularly.
And I think Elizabeth said several times “If you actually are taking the supplementation, it’s healthy, but I know many people who aren’t taking that supplementation. I think EA vegan activists should put more effort into providing good recommendations to people they convince to go vegan.” So I’m not sure why you’re thinking of the expert consensus here as saying a different thing.
I feel a bit confused about what the argument is about here. I think the local point of “hey, you should be quite skeptical of self-reports” is a good, important point (thanks for bringing it up. I don’t think I agree with you on how much I should discount this data, but I wasn’t modeling all the possible failure modes you’re pointing out). But it feels from your phrasing like there’s something else going on, or the thread is overall getting into a cycle of arguing-for-the-sake-of-arguing, or something. (Maybe it’s just that Elizabeth’s post is long and it’s easy to lose track of the various disclaimers she made? Maybe it’s more of a “how much are you supposed to even have an opinion if all your evidence is weak?” frame clash)
Could you (or Natalie) say more about what this thread is about from your perspective?
So I’m not sure why you’re thinking of the expert consensus here as saying a different thing.
As far as I can tell, I didn’t directly assert that expert consensus disagreed with Elizabeth in this thread. Indeed I mentioned that I “didn’t really” want to make claims about that. I only brought up expert consensus to reply to a narrow question that you asked about what we should rely on as a best guess. I didn’t mention expert consensus in any of my comments prior to that one, at least in this thread.
My primary point in this thread was to talk about the unreliability of self-reported data and the pitfalls of relying on it. Secondarily, I commented that most of the people she’s critiquing in this post don’t seem obviously guilty of the allegation in the title. I think it’s important to push back against accusations that a bunch of people (or in this case, a whole sub-community) is “not truthseeking” on the basis of weak evidence. And my general reply here is that if indeed vegan diets are generally healthy as long as one takes the standard precautions, then I think it is reasonable for others to complain about someone emphasizing health tradeoffs excessively (which is what I interpreted many of the quoted people in the post as doing).
(At the very least, if you think these people are being unreasonable, I would maintain that the sweeping accusation in the title requires stronger evidence than what was presented. I am putting this in parentheses though to emphasize that this is not my main point.)
Also, I think it’s possible that Elizabeth doesn’t agree with the scientific consensus, or thinks it’s at least slightly wrong. I don’t want to put words in her mouth, though. Partly I think the scientific consensus is important to mention at some point because I don’t fully know what she believes, and I think that bringing up expert consensus is a good way to ground our discussion and make our premises more transparent. However, if she agrees with the consensus, then I’m still OK saying what I said, because I think almost all of it stands or falls independently of whether she agrees with the consensus.
But it feels from your phrasing like there’s something else going on, or the thread is overall getting into a cycle of arguing-for-the-sake-of-arguing, or something.
That’s possible too. I do think I might be getting too deep into this over what is mostly a few pointless quibbles about what type of data is reliable and what isn’t. You’re right to raise the possibility that things are going off the rails in an unintended way.
I think the original post was a bit confusing in what it claimed the Faunalytics study was useful for.
For example, the section
The ideal study is a longitudinal RCT where diet is randomly assigned, cost (across all dimensions, not just money) is held constant, and participants are studied over multiple years to track cumulative effects. I assume that doesn’t exist, but the closer we can get the better.
I’ve spent several hours looking for good studies on vegan nutrition, of which the only one that was even passable was the Faunalytics study.
[...]
A non-exhaustive list of common flaws:
Studies rarely control for supplements. [...]
makes it sound like the author is interested on the effects of vegan diets on health, both with and without supplementation, and that they’re claiming that the Faunalytics study is the best study we have to answer that question. This is what I and Matthew would strongly disagree with.
This post uses the Faunalytics study in a different (and IMO more reasonable) way, to show which proportion of veg*ans report negative health effects and quit in practice. This is a different question because it can loosely track how much veg*ans follow dietary guidelines. For example, vitamin B12 deficiency should affect close to 100% of vegans who don’t supplement and have been vegan for long enough, and, on the other side of the spectrum, it likely affects close to 0% of those who supplement, monitor their B12 levels and take B12 infusions when necessary.
A “longitudinal RCT where diet is randomly assigned” and that controls for supplements would not be useful for answering the second question, and neither would the RCTs and systematic reviews I brought up. But they would be more useful than the Faunalytcis survey for answering the first question.
Elizabeth has put at least dozens of hours into seeking good RCTs on vegan nutrition, and has come up nearly empty. At this point, if you want to say there is an expert consensus that disagrees with her, you need to find a particular study that you are willing to stand behind, so that we can discuss it. This is why Elizabeth wrote a post on the Adventist study—because that was the best that people were throwing at her.
Elizabeth has put at least dozens of hours into seeking good RCTs on vegan nutrition, and has come up nearly empty.
Given that I haven’t looked deeply into the RCTs and observational studies, I fully admit that I can’t completely address this comment and defend the merits of the scientific data. That said, I find it very unlikely that the RCTs and observational studies are so flawed that the self-reported survey data is more reliable. Although I find it quite plausible that the diet studies are flawed, why would the self-reported data be better?
The scientific studies might be bad, but that doesn’t mean we should anchor to an even more unreliable source of information.
At this point, if you want to say there is an expert consensus that disagrees with her, you need to find a particular study that you are willing to stand behind, so that we can discuss it.
I was very careful in my comment to say that I was only bringing up expert consensus to respond purely to a narrow point about what the “best guess” of vegan activists should be in the absence of a thorough investigation. Moreover, expert consensus is not generally revealed via studies, and so I don’t think I need to bring one up in order to make this point. Expert consensus is usually revealed by statements from mainstream institutions and prominent scientists, and sometimes survey data from scientists. If you’re asking me to show expert consensus, then I’d refer you to statements from the American Dietetic Association and the British Dietetic Association as a start. But I also want to emphasize that I really do not see expert consensus as the primary point of contention here.
So far as I can tell, the central question Elizabeth has been trying to answer is “Do the people who convert to veganism because they get involved in EA have systemic health problems?” Those health problems might be easily solvable with supplementation (Great!), systemic to having a fully vegan diet but only requires some modest amount of animal product, or something more complicated. She has several self-reported people coming to her saying they tried veganism, had health problems, and stopped. So, “At what rate do vegans desist for health reasons?” seems like an important question to me. It will tell you at least some of what you are missing when surveying current vegans only.
I agree that if your prior probability of something being true is near 0, you need very strong evidence to update. Was your prior probability that someone would desist from the vegan diet for health reasons actually that low? If not, why is the crystal healing metaphor analogous?
As I argued in my original comment, self-reported data is unreliable for answering this question. I simply do not trust people’s ability to attribute the causal impact of diets on their health. Separately, I think people frequently misreport their motives. Even if vegan diets caused no health effects, a substantial fraction of people could still report desisting from veganism for health reasons.
I’m honestly not sure why think that self-reported data is more reliable than proper scientific studies like RTCs when trying to shed light on this question. The RCTs should be better able to tell us the actual health effects of adopting veganism, which is key to understanding how many people would be forced to abandon the diet for health reasons.
It seems to me like even if this isn’t relevant to our beliefs about underlying reality (“teachers think criticism works because they’re not correcting for regression to the mean, and so we can’t rely on teacher’s beliefs”) it should be relevant to our beliefs about individual decision-making (“when people desist from vegan diets, 20% of the time it’s because they think their health is worse”), which should direct our views on advocacy (if we want fewer people to desist from vegan diets (and possibly then dissuade others from trying them), we should try to make sure they don’t think their health is worse on them).
Often there’s question-substitution going on. If the proper scientific studies are measuring easily quantifiable things like blood pressure and what people care about more / make decisions based off of is the difficult-to-quantify “how much pep is in my step”, then the improper survey may point more directly at the thing that’s relevant, even if it does so less precisely.
The question I assumed Stephen was asking (and at least my question for myself) here is, “okay, but what do we believe in the meanwhile?”.
Natalia responded with a process that might find some good evidence (but, might not, and looks like at least several hours of skilled-labor search to find out). I agree someone should do that labor and find out if better evidence exists.
I also realize Vaniver did explicitly ask “what alternate framework you prefer?” and it makes sense that your framework is interested in different questions than mine or Elizabeths or Stephens or whatnot. But, for me, the question is “what should vegan activist’s best guess be right now”, not “what might it turn out to be after doing a bunch more research that maybe turns out to have good data and maybe doesn’t.”
Best guess of what, specifically?
This is fair and a completely reasonable question to ask, even if we agree that the self-reported data is unreliable. I agree the self-reported data could be a useful first step towards answering this question if we had almost no other information. I also haven’t looked deeply into the RCTs and scientific data and so I don’t have a confident view on the health value of vegan diets.
On the other hand, personally, my understanding is that multiple mainstream scientific institutions say that vegan diets are generally healthy (in the vast majority of cases) unless you don’t take the proper supplementation regularly. If you put a gun to my head right now and asked me to submit my beliefs about this question, I would defer heavily to (my perception of) the mainstream consensus, rather than the self-reported data. That’s not because I think mainstream consensus isn’t sometimes wrong or biased, but—in the spirit of your question—that’s just what I think is most reliable out of all the easily accessible facts that I have available right now, including the self-reported data.
To be clear, I didn’t really want to take this line, and talk about how The Experts disagree with Elizabeth, and so the burden of proof is on her rather than me, because that’s often a conversation stopper and not helpful for fruitful discussion, especially given my relative ignorance about the empirical data. But if we’re interested in what a good “best guess” should be, then yes, I think mainstream scientific institutions are generally reliable on questions they have strong opinions about. That’s not my response to everything in this discussion, but it’s my response to your specific point about what we should believe in the meantime.
Nod.
In this case I don’t think the claim you’re ascribing to the experts are Elizabeth are actually in conflict. You say:
And I think Elizabeth said several times “If you actually are taking the supplementation, it’s healthy, but I know many people who aren’t taking that supplementation. I think EA vegan activists should put more effort into providing good recommendations to people they convince to go vegan.” So I’m not sure why you’re thinking of the expert consensus here as saying a different thing.
I feel a bit confused about what the argument is about here. I think the local point of “hey, you should be quite skeptical of self-reports” is a good, important point (thanks for bringing it up. I don’t think I agree with you on how much I should discount this data, but I wasn’t modeling all the possible failure modes you’re pointing out). But it feels from your phrasing like there’s something else going on, or the thread is overall getting into a cycle of arguing-for-the-sake-of-arguing, or something. (Maybe it’s just that Elizabeth’s post is long and it’s easy to lose track of the various disclaimers she made? Maybe it’s more of a “how much are you supposed to even have an opinion if all your evidence is weak?” frame clash)
Could you (or Natalie) say more about what this thread is about from your perspective?
As far as I can tell, I didn’t directly assert that expert consensus disagreed with Elizabeth in this thread. Indeed I mentioned that I “didn’t really” want to make claims about that. I only brought up expert consensus to reply to a narrow question that you asked about what we should rely on as a best guess. I didn’t mention expert consensus in any of my comments prior to that one, at least in this thread.
My primary point in this thread was to talk about the unreliability of self-reported data and the pitfalls of relying on it. Secondarily, I commented that most of the people she’s critiquing in this post don’t seem obviously guilty of the allegation in the title. I think it’s important to push back against accusations that a bunch of people (or in this case, a whole sub-community) is “not truthseeking” on the basis of weak evidence. And my general reply here is that if indeed vegan diets are generally healthy as long as one takes the standard precautions, then I think it is reasonable for others to complain about someone emphasizing health tradeoffs excessively (which is what I interpreted many of the quoted people in the post as doing).
(At the very least, if you think these people are being unreasonable, I would maintain that the sweeping accusation in the title requires stronger evidence than what was presented. I am putting this in parentheses though to emphasize that this is not my main point.)
Also, I think it’s possible that Elizabeth doesn’t agree with the scientific consensus, or thinks it’s at least slightly wrong. I don’t want to put words in her mouth, though. Partly I think the scientific consensus is important to mention at some point because I don’t fully know what she believes, and I think that bringing up expert consensus is a good way to ground our discussion and make our premises more transparent. However, if she agrees with the consensus, then I’m still OK saying what I said, because I think almost all of it stands or falls independently of whether she agrees with the consensus.
That’s possible too. I do think I might be getting too deep into this over what is mostly a few pointless quibbles about what type of data is reliable and what isn’t. You’re right to raise the possibility that things are going off the rails in an unintended way.
I think the original post was a bit confusing in what it claimed the Faunalytics study was useful for.
For example, the section
makes it sound like the author is interested on the effects of vegan diets on health, both with and without supplementation, and that they’re claiming that the Faunalytics study is the best study we have to answer that question. This is what I and Matthew would strongly disagree with.
This post uses the Faunalytics study in a different (and IMO more reasonable) way, to show which proportion of veg*ans report negative health effects and quit in practice. This is a different question because it can loosely track how much veg*ans follow dietary guidelines. For example, vitamin B12 deficiency should affect close to 100% of vegans who don’t supplement and have been vegan for long enough, and, on the other side of the spectrum, it likely affects close to 0% of those who supplement, monitor their B12 levels and take B12 infusions when necessary.
A “longitudinal RCT where diet is randomly assigned” and that controls for supplements would not be useful for answering the second question, and neither would the RCTs and systematic reviews I brought up. But they would be more useful than the Faunalytcis survey for answering the first question.
Elizabeth has put at least dozens of hours into seeking good RCTs on vegan nutrition, and has come up nearly empty. At this point, if you want to say there is an expert consensus that disagrees with her, you need to find a particular study that you are willing to stand behind, so that we can discuss it. This is why Elizabeth wrote a post on the Adventist study—because that was the best that people were throwing at her.
Given that I haven’t looked deeply into the RCTs and observational studies, I fully admit that I can’t completely address this comment and defend the merits of the scientific data. That said, I find it very unlikely that the RCTs and observational studies are so flawed that the self-reported survey data is more reliable. Although I find it quite plausible that the diet studies are flawed, why would the self-reported data be better?
The scientific studies might be bad, but that doesn’t mean we should anchor to an even more unreliable source of information.
I was very careful in my comment to say that I was only bringing up expert consensus to respond purely to a narrow point about what the “best guess” of vegan activists should be in the absence of a thorough investigation. Moreover, expert consensus is not generally revealed via studies, and so I don’t think I need to bring one up in order to make this point. Expert consensus is usually revealed by statements from mainstream institutions and prominent scientists, and sometimes survey data from scientists. If you’re asking me to show expert consensus, then I’d refer you to statements from the American Dietetic Association and the British Dietetic Association as a start. But I also want to emphasize that I really do not see expert consensus as the primary point of contention here.