For the weirdness around how they did this / didn’t preregister, I thought they were pretty upfront about how this was exploratory (trying to generate hypotheses and see if the effect is plausible, not confirm a hypothesis). The next step would be a more serious experiment like the Potato Camp they mentioned.
Doing it this way lets them figure out what to focus more expensive / complicated studies on without wasting time and money (like if they ran a big experiment without peeling the potatoes and it ended up being a complete waste).
I think doing this kind of thing is really important because trying to make everything safe means we frequently don’t do useful science (see: we let millions of people die to avoid letting volunteers take risks in vaccine studies). Doing more mad science is a useful correction. That said, I think it’s important to remind volunteers that they shouldn’t keep going if the mad science is hurting them. I get the impression SMTM was trying to tell people this but they probably should have been more direct about telling people saying they’re not feeling good to stop.
The next step would be a more serious experiment like the Potato Camp they mentioned.
This is puzzling to me. Randomizing people to different kinds of somewhat restrictive diets[1] seems like a way cheaper and more obvious experiment to test some of SMTM’s hypotheses, such that the potassium in potatoes clears out lithium or whatever.
It seems to me that they would have incurred little additional cost if they had randomized people in this study they already did, so I am somewhat confused about the choice not to have done that.
I say “somewhat restrictive” because I’m reluctant to advocate very restrictive diets, given the very low caloric intake reported by some people in SMTM’s blog post, and the increased risk of gallstones and refeeding syndrome that people incur by eating that little.
Then shouldn’t it be weirder? Like, having full open data, not trying a month specifically and varying this, etc?
I think doing this kind of thing is really important because trying to make everything safe means we frequently don’t do useful science
That is a very good point, and I agree.
Two things on this point:
It did not seem to me that SMTM marked this as clearly unsafe, but I am not neutral enough to know. Just: There is a difference between “quit if you feel unwell” and “This might have drastic negative impact on you you will not even notice or feel”
While mad unsafe science is cool, it is a good idea to make it safe_r_. For instance, SMTM did tell us to take vit A and B12, which was warranted. The same way, other precautions, specifically about how to quit the diet, could have been mentioned
What I mean is that if the goal was just to explore, and modifying the diet while doing it is not a problem, then it would have made more sense to have access to everyone’s data as they were doing it (at least between participants). Also have more ways to communicate between participants to share our experience, tips and tricks, etc...
I wouldn’t have made participation mandatory (and I expect you wouldn’t either) but a way for participants to opt-in to a community does seem valuable. Maybe they thought that was covered by twitter? A lot of people did seem to be discussing it there.
I thought they were pretty upfront about how this was exploratory (trying to generate hypotheses and see if the effect is plausible, not confirm a hypothesis).
Also, yes, that was my impression as well and what I signed up for. It’s just that a lot of features of the study directly clash with this exploration.
(I also disagree that their results are that remarkable.)
Edit: By remarkable, I mean “shows that it is specific to potatoes”
For the weirdness around how they did this / didn’t preregister, I thought they were pretty upfront about how this was exploratory (trying to generate hypotheses and see if the effect is plausible, not confirm a hypothesis). The next step would be a more serious experiment like the Potato Camp they mentioned.
Doing it this way lets them figure out what to focus more expensive / complicated studies on without wasting time and money (like if they ran a big experiment without peeling the potatoes and it ended up being a complete waste).
I think doing this kind of thing is really important because trying to make everything safe means we frequently don’t do useful science (see: we let millions of people die to avoid letting volunteers take risks in vaccine studies). Doing more mad science is a useful correction. That said, I think it’s important to remind volunteers that they shouldn’t keep going if the mad science is hurting them. I get the impression SMTM was trying to tell people this but they probably should have been more direct about telling people saying they’re not feeling good to stop.
This is puzzling to me. Randomizing people to different kinds of somewhat restrictive diets[1] seems like a way cheaper and more obvious experiment to test some of SMTM’s hypotheses, such that the potassium in potatoes clears out lithium or whatever.
It seems to me that they would have incurred little additional cost if they had randomized people in this study they already did, so I am somewhat confused about the choice not to have done that.
I say “somewhat restrictive” because I’m reluctant to advocate very restrictive diets, given the very low caloric intake reported by some people in SMTM’s blog post, and the increased risk of gallstones and refeeding syndrome that people incur by eating that little.
Then shouldn’t it be weirder? Like, having full open data, not trying a month specifically and varying this, etc?
That is a very good point, and I agree.
Two things on this point:
It did not seem to me that SMTM marked this as clearly unsafe, but I am not neutral enough to know. Just: There is a difference between “quit if you feel unwell” and “This might have drastic negative impact on you you will not even notice or feel”
While mad unsafe science is cool, it is a good idea to make it safe_r_. For instance, SMTM did tell us to take vit A and B12, which was warranted. The same way, other precautions, specifically about how to quit the diet, could have been mentioned
It might be that I over-updated though.
What data do you think they should be sharing that they aren’t? They did put a dataset up somewhere.
Right, I must have phrased it poorly.
What I mean is that if the goal was just to explore, and modifying the diet while doing it is not a problem, then it would have made more sense to have access to everyone’s data as they were doing it (at least between participants). Also have more ways to communicate between participants to share our experience, tips and tricks, etc...
I wouldn’t have made participation mandatory (and I expect you wouldn’t either) but a way for participants to opt-in to a community does seem valuable. Maybe they thought that was covered by twitter? A lot of people did seem to be discussing it there.
Also, yes, that was my impression as well and what I signed up for. It’s just that a lot of features of the study directly clash with this exploration.
(I also disagree that their results are that remarkable.)
Edit: By remarkable, I mean “shows that it is specific to potatoes”