Outcomes for veganism are [...] worse than everything except for omnivorism in women.
As I explained elsewhere a few days ago (after this post was published), this is a very misleading way to describe that study. The correct takeaway is that they could not find any meaningful difference between each diet’s association with mortality among women, not that “[o]utcomes for veganism are [...] worse than everything except for omnivorism in women.”
It’s very important to consider the confidence intervals in addition to the point estimates when interpreting this study (or any study, really, when confidence intervals are available). They provide valuable context to the data.
Mod note: I count six deleted comments by you on this post. Of these, two had replies (and so were edited to just say “deleted”), one was deleted quickly after posting, and three were deleted after they’d been up for awhile. This is disruptive to the conversation. It’s particularly costly when the subject of the top-level post is about conversation dynamics themselves, which the deleted comments are instances (or counterexamples) of.
You do have the right to remove your post/comments from LessWrong. However, doing so frequently, or in the middle of active conversations, is impolite. If you predict that you’re likely to wind up deleting a comment, it would be better to not post it in the first place. LessWrong has a “retract” button which crosses out text (keeping it technically-readable but making it annoying to read so that people won’t); this is the polite and epistemically-virtuous way to handle comments that you no longer stand by.
Thanks for this information. When I did this, it was because I was misunderstanding someone’s position, and only realized it later. I’ll refrain from deleting comments excessively in the future and will use the “retract” feature when something like this happens again.
That sounds right. When citing a study as finding X is worse than Y, unless you say otherwise people will interpret that as “the study’s confidence intervals for X and Y don’t overlap” and not the much weaker “the study’s point estimate for X is below it’s point estimate for Y”.
(It’s a bit less clear in this context, where Elizabeth is trying to say that, contrary to other people’s claims, the study does not show that veganism is better than other diets. In that case the point estimate for X being below Y does tell us you shouldn’t use the study to argue that X is above Y. But I agree with Natália that people are likely to misinterpret the OP this way.)
To be clear, the study found that veganism and pescetarianism were meaningfully associated with lower mortality among men (aHR 0.72 , 95% CI [0.56, 0.92] and 0.73 , 95% CI [0.57, 0.93], respectively), and that no dietary patterns were meaningfully associated with mortality among women. I don’t think it’s misleading to conclude from this that veganism likely has neutral-to-positive effects on lifespan given this study’s data, which was ~my conclusion in the comment I wrote that Elizabeth linked on that section, which was described as “deeply misleading.”
Thanks Jeff. I’m curious how you feel about my original phrasing, versus what’s here.
I agree that the CIs heavily overlap, and drawing strong conclusions from this would be unjustified. I nonetheless think it’s relevant that even if you treat the results as meaningful, the summaries given by the abstract and some commenters are inaccurate (Natalia has since said she was making a more limited claim). That means they’re making two errors (overstating effect, and effect in wrong direction) rather than just one (overstating effect).
I can see how my phrasing here wouldn’t convey all that if you don’t click through the link, and that seems worth fixing, but I’m curious what you think of the underlying point.
That means they’re making two errors (overstating effect, and effect in wrong direction) rather than just one (overstating effect).
Froolow’s comment claimed that “there’s somewhere between a small signal and no signal that veganism is better with respect to all-cause mortality than omnivorism.” How is that a misleading way of summarizing the adjusted hazard ratio 0.85 (95% CI, 0.73–1.01), in either magnitude or direction? Should he have said that veganism is associated with higher mortality instead?
None of the comments you mentioned in that section claimed that veganism was associated with lower mortality in all subgroups (e.g. women). But even if they had, the hazard ratio for veganism among women was still in the “right” direction (below 1, though just slightly and not meaningfully). Other diets were (just slightly and not meaningfully) better among women, but none of the commenters claimed that veganism was better than all diets either.
Unrelatedly, I noticed that in this comment (and in other comments you’ve made regarding my points about confidence intervals) you don’t seem to argue that the sentence “[o]utcomes for veganism are [...] worse than everything except for omnivorism in women” is not misleading.
I think one issue here is that my phrasing was bad. I meant “support” as in “doesn’t support claims of superiority”, but reading it now it’s obvious that it could be read as “doesn’t support claims as fine”, which is not what I meant. I agree that veganism works for some people. I want to fix that, although am going to wait until I’m sure there aren’t other changes, because all changes need to be made in triplicate.
I appreciate you pointing out that potential reading so I could fix it. However I am extremely frustrated with this conversation overall. You are holding me to an exacting standard while making basic errors yourself, such as: * Quoted me as saying “X is great”, when what I said was “If you believe Y, which you shouldn’t, X is great” * implying that when I wrote this post I ignored a response from you, when the response was made a day after this was posted posted, after I pointed out your non-response in comments. (This was somewhat fixed after another commenter pointed it out) * Your original comment on Change My Mind referred to this study as “missing context”, as if it was something I should have known and deliberately left out. That’s a loaded implication in general, but outright unfair when the post is titled “change my mind” and its entire point was asking for exactly that kind of information.
The second point here was not intended and I fixed it within 2 minutes of orthonormal pointing it out, so it doesn’t seem charitable to bring that up. (Though I just re-edited that comment to make this clearer).
I’m not sure what to say regarding the third point other than that I didn’t mean to imply that you “should have known and deliberately left out” that study. I just thought it was (literally) useful context. Just edited that comment.
All of this also seems unrelated to this discussion. I’m not sure why me addressing your arguments is being construed as “holding [you] to an exacting standard.”
As I explained elsewhere a few days ago (after this post was published), this is a very misleading way to describe that study. The correct takeaway is that they could not find any meaningful difference between each diet’s association with mortality among women, not that “[o]utcomes for veganism are [...] worse than everything except for omnivorism in women.”
It’s very important to consider the confidence intervals in addition to the point estimates when interpreting this study (or any study, really, when confidence intervals are available). They provide valuable context to the data.
Mod note: I count six deleted comments by you on this post. Of these, two had replies (and so were edited to just say “deleted”), one was deleted quickly after posting, and three were deleted after they’d been up for awhile. This is disruptive to the conversation. It’s particularly costly when the subject of the top-level post is about conversation dynamics themselves, which the deleted comments are instances (or counterexamples) of.
You do have the right to remove your post/comments from LessWrong. However, doing so frequently, or in the middle of active conversations, is impolite. If you predict that you’re likely to wind up deleting a comment, it would be better to not post it in the first place. LessWrong has a “retract” button which crosses out text (keeping it technically-readable but making it annoying to read so that people won’t); this is the polite and epistemically-virtuous way to handle comments that you no longer stand by.
Thanks for this information. When I did this, it was because I was misunderstanding someone’s position, and only realized it later. I’ll refrain from deleting comments excessively in the future and will use the “retract” feature when something like this happens again.
That sounds right. When citing a study as finding X is worse than Y, unless you say otherwise people will interpret that as “the study’s confidence intervals for X and Y don’t overlap” and not the much weaker “the study’s point estimate for X is below it’s point estimate for Y”.
(It’s a bit less clear in this context, where Elizabeth is trying to say that, contrary to other people’s claims, the study does not show that veganism is better than other diets. In that case the point estimate for X being below Y does tell us you shouldn’t use the study to argue that X is above Y. But I agree with Natália that people are likely to misinterpret the OP this way.)
To be clear, the study found that veganism and pescetarianism were meaningfully associated with lower mortality among men (aHR 0.72 , 95% CI [0.56, 0.92] and 0.73 , 95% CI [0.57, 0.93], respectively), and that no dietary patterns were meaningfully associated with mortality among women. I don’t think it’s misleading to conclude from this that veganism likely has neutral-to-positive effects on lifespan given this study’s data, which was ~my conclusion in the comment I wrote that Elizabeth linked on that section, which was described as “deeply misleading.”
Thanks Jeff. I’m curious how you feel about my original phrasing, versus what’s here.
I agree that the CIs heavily overlap, and drawing strong conclusions from this would be unjustified. I nonetheless think it’s relevant that even if you treat the results as meaningful, the summaries given by the abstract and some commenters are inaccurate (Natalia has since said she was making a more limited claim). That means they’re making two errors (overstating effect, and effect in wrong direction) rather than just one (overstating effect).
I can see how my phrasing here wouldn’t convey all that if you don’t click through the link, and that seems worth fixing, but I’m curious what you think of the underlying point.
Froolow’s comment claimed that “there’s somewhere between a small signal and no signal that veganism is better with respect to all-cause mortality than omnivorism.” How is that a misleading way of summarizing the adjusted hazard ratio 0.85 (95% CI, 0.73–1.01), in either magnitude or direction? Should he have said that veganism is associated with higher mortality instead?
None of the comments you mentioned in that section claimed that veganism was associated with lower mortality in all subgroups (e.g. women). But even if they had, the hazard ratio for veganism among women was still in the “right” direction (below 1, though just slightly and not meaningfully). Other diets were (just slightly and not meaningfully) better among women, but none of the commenters claimed that veganism was better than all diets either.
Unrelatedly, I noticed that in this comment (and in other comments you’ve made regarding my points about confidence intervals) you don’t seem to argue that the sentence “[o]utcomes for veganism are [...] worse than everything except for omnivorism in women” is not misleading.
I think one issue here is that my phrasing was bad. I meant “support” as in “doesn’t support claims of superiority”, but reading it now it’s obvious that it could be read as “doesn’t support claims as fine”, which is not what I meant. I agree that veganism works for some people. I want to fix that, although am going to wait until I’m sure there aren’t other changes, because all changes need to be made in triplicate.
I appreciate you pointing out that potential reading so I could fix it. However I am extremely frustrated with this conversation overall. You are holding me to an exacting standard while making basic errors yourself, such as:
* Quoted me as saying “X is great”, when what I said was “If you believe Y, which you shouldn’t, X is great”
* implying that when I wrote this post I ignored a response from you, when the response was made a day after this was posted posted, after I pointed out your non-response in comments. (This was somewhat fixed after another commenter pointed it out)
* Your original comment on Change My Mind referred to this study as “missing context”, as if it was something I should have known and deliberately left out. That’s a loaded implication in general, but outright unfair when the post is titled “change my mind” and its entire point was asking for exactly that kind of information.
The second point here was not intended and I fixed it within 2 minutes of orthonormal pointing it out, so it doesn’t seem charitable to bring that up. (Though I just re-edited that comment to make this clearer).
The first point was already addressed here.
I’m not sure what to say regarding the third point other than that I didn’t mean to imply that you “should have known and deliberately left out” that study. I just thought it was (literally) useful context. Just edited that comment.
All of this also seems unrelated to this discussion. I’m not sure why me addressing your arguments is being construed as “holding [you] to an exacting standard.”