On one hand, I’ve been disappointed with SMTM’s handling of this study. I would rather have had more caveats, more pushes to make people more responsible for their own health, and more contextualization around crash diets and monodiets.
On the other hand, I want this data. And I want more experiments in general. When I tried to run my own study, for a much smaller ask than a monodiet, with what I view as acceptable amounts of caveats and contextualization, I really struggled with recruitment and subject follow through. That’s partially a function of audience size, except my smaller audience is in part downstream of me being more cautious and less sweeping in my claims in general. I’m just less fun to read than Slime Mold Time Mold.
[it didn’t help follow through that I was specifically recruiting people with ADHD, who are hell’s own subject group, but the issue with recruitment was audience size and excitement]
This doesn’t undo any of the damage done by SMTM’s lack of caution (it feels unfair judging them on your specific case because they did tell you not to do it, but I’d independently noticed less caution and caveats than I’d like). But if we want this kind of data at all we either need to accept those costs, or become more willing to participate in experiments with accurate caveats.
it feels unfair judging them on your specific case because they did tell you not to do it
I notice I’m very confused. SMTM said not to do it because of bipolarity:
Yeah, we would actually recommend people with bipolar disorder not enroll, both because there might be interactions with medication (especially if we’re right about the lithium thing) and because the potato diet seems to sometimes trigger hypomania even in people without bipolar
(The reason I chose to do it is that I am not under medication and that I am type 2 bipolar, not type 1)
I did end up with more pronounced bipolar symptoms—which I included here to give an exhaustive list of effects the diet has had on me—and that was fine for me because I was warned of the risks and that I had done a risk analysis based on this.
My concerns about refeeding syndrome and physical weakness, as far as I know, have nothing to do with bipolarity. Also, as far as I know, no permanent or drastic damage has been effected in me by all this. I am not writing this so much to say “this went very poorly” but much more that “this could have gone way worse”
I want more experiments in general
Maybe I have not been explicit enough about this, but I do too. There is a reason I was excited to sign up, and I am still very enthusiastic about decentralized science. One of the concern I had in publishing this is that I do not want for the cost of such groups to run an experiment to be too high.
I want this data
(I am a little unsure how exploitable the data are at the moment, besides the weight loss. I think this is in big part because of the exploration/exploitation uncertainty mentioned above, and that had they gone in either direction more clearly, the data would have been a lot more useful)
But if we want this kind of data at all we either need to accept those costs, or become more willing to participate in experiments with accurate caveats.
So to be clear: If SMTM had written in big capital letters “we have not researched into any of the following:
The best practices to stop a diet
Whether potatoes are safe to consume in big quantities for an extended period of time
What kind of signs to look for that are clear indications you should stop
More research to shrink our unknown unknowns
If you sign up for this diet, you understand that we do not know anything and have no ideas about any of these points or any others that are not mentioned in our post.”
Then I would have been completely fine with it. My issue is that there were little to no communication about either what was unsafe, or their level of knowledge about the safety of the diet.
Also, this would have been less problematic for me if they went fully into the “explore” route and were not doing this study as a mean of replication (no set period of time, full communication between participants, frequent updates about the diet, etc). This issue is more relevant for the “please sign up and fill this form” type of study
Part of me strongly agrees with you. At the same time… I don’t think “how much work would it have been to address [single bad thing]?” is the right unit of measurement. The actual question is “what would the cost/benefit analysis of address every single thing in this class of problems?”, and that’s a lot less clear cut.
Some things that make it difficult:
There are just a lot of potential problems.
There are an infinite number of things they didn’t research. Every time they add one to the explicit list of unresearched things they make the list as a whole harder to read (-> people are more likely to miss warnings relevant to them) while also conveying more safety and confidence. God forbid people decide anything not on the list of “things we didn’t check” was in fact checked.
You can say they should estimate the danger from each of these and only explicitly disavow research over a certain level of importance, but that’s an inherently noisy process people are likely to disagree on, and no matter what there has to be something just below the line. So again adding warnings can leave people with an inaccurately increased feeling of safety.
When I share medical info, I always preface it with “anything with a real effect can hurt you, think this through for yourself”. I think that’s a little better than what SMTM did. But I don’t see any easy additions to their statements of “If at any point you get sick or begin having side-effects, stop the diet immediately. We can still use your data up to that point, and we don’t want anything to happen to you” and “We are not doctors. We are 20 rats in a trenchcoat. eee! eee! eee!” that make things obviously better.
You note that you felt obliged to keep going in order to provide better data. Maybe there was pressure in private communication, but AFAICT they explicitly spoke against that in the blog post. I had a similar thing happen in my experiment- despite explicitly saying people could drop out at any time and would still be paid, one person was extremely reluctant to do so despite bad side effects and needed a strong push from me. In some sense I don’t think this is fair to put on experimenters, people shouldn’t even need to be told “stop if you feel bad”. But this seems like a pattern, and therefore should be taken into account even if it’s not objectively fair.
“what would the cost/benefit analysis of address every single thing in this class of problems?”,
I mean, that is why I included “More research to shrink our unknown unknowns” as a general category. I do not think the research needs to be thorough, 3-4 very broad general area would suffice, but even if that does not fit into the points you mention, a statement along the lines you mentioned could work. In general, I do not think that more than one-two hours should be spent on writing this warning.
As for “If at any point you get sick or begin having side effects, stop the diet immediately”, that is indeed a good first step. My problem with it is that I am now understanding that you can develop side effects that indicate one should stop, that are not apparent unless you really track several variables intently. A general instruction to, in addition to doing one’s own research, decide what to look for in advance could have worked.
You note that you felt obliged to keep going in order to provide better data
I want to note three things:
I did not feel pressured by SMTM either in their public or private communications at any point
That is not quite correct. I wanted to keep going for the data to be included as a data point, especially if the diet turned out not to work, which I thought it would not.
I had not noticed any side effects or problems at that point. I was tired of potatoes, but it was manageable.
Maybe I should better emphasize on what points I feel SMTM is responsible and on what points I do not. I do not feel that SMTM is responsible for my safety and what could have happened, or the impact of the diet on me. In fact, I do not believe SMTM to be responsible for anything me-related: I made my own choices, and the fact that I did not stop the diet sooner was my own mistake I am owning up to.
What I do feel is that SMTM had a very loose methodology in how they conducted their studies, were more trying to confirm their hypothesis than really challenge it, and as a result the data is quite muddled and probably not that useful.
The safety part is related in the sense that:
It might be that many potato diet’s benefits are actually just coming from malnutrition. But if there were evidences of safety, that would not turn up to be a problem
It is relevant about what kind of preemptive research they made to decide whether the potato diet was worth it
For me at least, it goes with the general notion that it was very unclear what they were looking for
On one hand, I’ve been disappointed with SMTM’s handling of this study. I would rather have had more caveats, more pushes to make people more responsible for their own health, and more contextualization around crash diets and monodiets.
On the other hand, I want this data. And I want more experiments in general. When I tried to run my own study, for a much smaller ask than a monodiet, with what I view as acceptable amounts of caveats and contextualization, I really struggled with recruitment and subject follow through. That’s partially a function of audience size, except my smaller audience is in part downstream of me being more cautious and less sweeping in my claims in general. I’m just less fun to read than Slime Mold Time Mold.
[it didn’t help follow through that I was specifically recruiting people with ADHD, who are hell’s own subject group, but the issue with recruitment was audience size and excitement]
This doesn’t undo any of the damage done by SMTM’s lack of caution (it feels unfair judging them on your specific case because they did tell you not to do it, but I’d independently noticed less caution and caveats than I’d like). But if we want this kind of data at all we either need to accept those costs, or become more willing to participate in experiments with accurate caveats.
I notice I’m very confused. SMTM said not to do it because of bipolarity:
(The reason I chose to do it is that I am not under medication and that I am type 2 bipolar, not type 1)
I did end up with more pronounced bipolar symptoms—which I included here to give an exhaustive list of effects the diet has had on me—and that was fine for me because I was warned of the risks and that I had done a risk analysis based on this.
My concerns about refeeding syndrome and physical weakness, as far as I know, have nothing to do with bipolarity. Also, as far as I know, no permanent or drastic damage has been effected in me by all this. I am not writing this so much to say “this went very poorly” but much more that “this could have gone way worse”
Maybe I have not been explicit enough about this, but I do too. There is a reason I was excited to sign up, and I am still very enthusiastic about decentralized science. One of the concern I had in publishing this is that I do not want for the cost of such groups to run an experiment to be too high.
(I am a little unsure how exploitable the data are at the moment, besides the weight loss. I think this is in big part because of the exploration/exploitation uncertainty mentioned above, and that had they gone in either direction more clearly, the data would have been a lot more useful)
So to be clear: If SMTM had written in big capital letters “we have not researched into any of the following:
The best practices to stop a diet
Whether potatoes are safe to consume in big quantities for an extended period of time
What kind of signs to look for that are clear indications you should stop
More research to shrink our unknown unknowns
If you sign up for this diet, you understand that we do not know anything and have no ideas about any of these points or any others that are not mentioned in our post.”
Then I would have been completely fine with it. My issue is that there were little to no communication about either what was unsafe, or their level of knowledge about the safety of the diet.
Also, this would have been less problematic for me if they went fully into the “explore” route and were not doing this study as a mean of replication (no set period of time, full communication between participants, frequent updates about the diet, etc). This issue is more relevant for the “please sign up and fill this form” type of study
Part of me strongly agrees with you. At the same time… I don’t think “how much work would it have been to address [single bad thing]?” is the right unit of measurement. The actual question is “what would the cost/benefit analysis of address every single thing in this class of problems?”, and that’s a lot less clear cut.
Some things that make it difficult:
There are just a lot of potential problems.
There are an infinite number of things they didn’t research. Every time they add one to the explicit list of unresearched things they make the list as a whole harder to read (-> people are more likely to miss warnings relevant to them) while also conveying more safety and confidence. God forbid people decide anything not on the list of “things we didn’t check” was in fact checked.
You can say they should estimate the danger from each of these and only explicitly disavow research over a certain level of importance, but that’s an inherently noisy process people are likely to disagree on, and no matter what there has to be something just below the line. So again adding warnings can leave people with an inaccurately increased feeling of safety.
When I share medical info, I always preface it with “anything with a real effect can hurt you, think this through for yourself”. I think that’s a little better than what SMTM did. But I don’t see any easy additions to their statements of “If at any point you get sick or begin having side-effects, stop the diet immediately. We can still use your data up to that point, and we don’t want anything to happen to you” and “We are not doctors. We are 20 rats in a trenchcoat. eee! eee! eee!” that make things obviously better.
You note that you felt obliged to keep going in order to provide better data. Maybe there was pressure in private communication, but AFAICT they explicitly spoke against that in the blog post. I had a similar thing happen in my experiment- despite explicitly saying people could drop out at any time and would still be paid, one person was extremely reluctant to do so despite bad side effects and needed a strong push from me. In some sense I don’t think this is fair to put on experimenters, people shouldn’t even need to be told “stop if you feel bad”. But this seems like a pattern, and therefore should be taken into account even if it’s not objectively fair.
I mean, that is why I included “More research to shrink our unknown unknowns” as a general category. I do not think the research needs to be thorough, 3-4 very broad general area would suffice, but even if that does not fit into the points you mention, a statement along the lines you mentioned could work. In general, I do not think that more than one-two hours should be spent on writing this warning.
As for “If at any point you get sick or begin having side effects, stop the diet immediately”, that is indeed a good first step. My problem with it is that I am now understanding that you can develop side effects that indicate one should stop, that are not apparent unless you really track several variables intently. A general instruction to, in addition to doing one’s own research, decide what to look for in advance could have worked.
I want to note three things:
I did not feel pressured by SMTM either in their public or private communications at any point
That is not quite correct. I wanted to keep going for the data to be included as a data point, especially if the diet turned out not to work, which I thought it would not.
I had not noticed any side effects or problems at that point. I was tired of potatoes, but it was manageable.
Maybe I should better emphasize on what points I feel SMTM is responsible and on what points I do not. I do not feel that SMTM is responsible for my safety and what could have happened, or the impact of the diet on me. In fact, I do not believe SMTM to be responsible for anything me-related: I made my own choices, and the fact that I did not stop the diet sooner was my own mistake I am owning up to.
What I do feel is that SMTM had a very loose methodology in how they conducted their studies, were more trying to confirm their hypothesis than really challenge it, and as a result the data is quite muddled and probably not that useful.
The safety part is related in the sense that:
It might be that many potato diet’s benefits are actually just coming from malnutrition. But if there were evidences of safety, that would not turn up to be a problem
It is relevant about what kind of preemptive research they made to decide whether the potato diet was worth it
For me at least, it goes with the general notion that it was very unclear what they were looking for