I think there will be a range of issues from a few diehards hitting serious issues to people just having low-grade issues which they don’t notice because they won’t be randomizing blocks, effects similar to the hedonic treadmill will make it hard to compare over time, they’ll get initial benefits from the usual placebo/Hawthorne/overjustification effects, and subjective self-rating has many known loopholes where you can think you’re getting better even as you’re actually getting worse—but regardless of the exact distribution or what the worst-cases look like, we won’t know for the reasons I list above.
Instead, we’ll get another internet circle-jerk about how Soylent is awesome and the critics are wrong.
What would you like to see done differently? You mentioned the more thorough self-experimentation he could have done (really should have done), but there’s still someone else who could step up to the plate and do some self-testing.
Thorough studies? Those might also be done some time in the future, whether or not they’re funded by Rob (not sure about this point, there might not be an incentive to do so once it’s being sold).
Sure, Rob jumped the gun and hyped it up. But most of the internet is already a giant circle-jerk. Doesn’t stop people from generating real information, right?
Doesn’t stop people from generating real information, right?
That’s the damnable thing about these sorts of biases, it’s not clear to me whether one can compensate for the biases. If you pay attention to the ‘real information’, you may wind up learning what we might call anti-information—information that predictably and systematically makes your beliefs worse than your defaults.
This is the problem of the clever arguer: how do you, and can you, adjust for the fact that all the reports coming out about Soylent are so deeply error-prone (and now with the kickstarter, we get a delicious cherry on top of conflicts of interest)?
I would much rather have a handful of randomized self-experiments from some obscure blogger interested in a weird recipe he came up with than a forum of Soylent enthusiasts raving to each other that the latest formulation on sale is the greatest ever and telling each other that if you feel bad you should be eating your Soylent twice a day and not three times a day, don’t you know about intermittent fasting?, and also I ran 20 seconds faster today, so Soylent must be working for me!
Ok, I see what your concern is, with the hype around Soylent everyone’s opinion is skewed (even if they’re not among the fanboys).
You decided above that it wasn’t worth your time to try your own self-experiments with it. What if someone else were to take the time to do it? I like the concept but agree with the major troubles you listed above, and I have no experience with designing self-experiments. But maybe I’ll take the time to try and do it properly, long-term, with regular blood tests, noting what I’ve been eating for a couple months before starting, taking data about my fitness levels, etc. Of course, I would need to analyze the risk to myself beforehand.
What if someone else were to take the time to do it?
If they actually go through with it and write it up, that’s better than the status quo, yes. But if they don’t determine to go through with it and may give up, it’s another selection bias, specifically, publication bias (person A does a self-experiment but halfway through runs out of spare effort and abandons it; person B, by chance, gets better results and blogs about it etc).
I think there will be a range of issues from a few diehards hitting serious issues to people just having low-grade issues which they don’t notice because they won’t be randomizing blocks, effects similar to the hedonic treadmill will make it hard to compare over time, they’ll get initial benefits from the usual placebo/Hawthorne/overjustification effects, and subjective self-rating has many known loopholes where you can think you’re getting better even as you’re actually getting worse—but regardless of the exact distribution or what the worst-cases look like, we won’t know for the reasons I list above.
Instead, we’ll get another internet circle-jerk about how Soylent is awesome and the critics are wrong.
What would you like to see done differently? You mentioned the more thorough self-experimentation he could have done (really should have done), but there’s still someone else who could step up to the plate and do some self-testing.
Thorough studies? Those might also be done some time in the future, whether or not they’re funded by Rob (not sure about this point, there might not be an incentive to do so once it’s being sold).
Sure, Rob jumped the gun and hyped it up. But most of the internet is already a giant circle-jerk. Doesn’t stop people from generating real information, right?
That’s the damnable thing about these sorts of biases, it’s not clear to me whether one can compensate for the biases. If you pay attention to the ‘real information’, you may wind up learning what we might call anti-information—information that predictably and systematically makes your beliefs worse than your defaults.
This is the problem of the clever arguer: how do you, and can you, adjust for the fact that all the reports coming out about Soylent are so deeply error-prone (and now with the kickstarter, we get a delicious cherry on top of conflicts of interest)?
I would much rather have a handful of randomized self-experiments from some obscure blogger interested in a weird recipe he came up with than a forum of Soylent enthusiasts raving to each other that the latest formulation on sale is the greatest ever and telling each other that if you feel bad you should be eating your Soylent twice a day and not three times a day, don’t you know about intermittent fasting?, and also I ran 20 seconds faster today, so Soylent must be working for me!
Ok, I see what your concern is, with the hype around Soylent everyone’s opinion is skewed (even if they’re not among the fanboys).
You decided above that it wasn’t worth your time to try your own self-experiments with it. What if someone else were to take the time to do it? I like the concept but agree with the major troubles you listed above, and I have no experience with designing self-experiments. But maybe I’ll take the time to try and do it properly, long-term, with regular blood tests, noting what I’ve been eating for a couple months before starting, taking data about my fitness levels, etc. Of course, I would need to analyze the risk to myself beforehand.
If they actually go through with it and write it up, that’s better than the status quo, yes. But if they don’t determine to go through with it and may give up, it’s another selection bias, specifically, publication bias (person A does a self-experiment but halfway through runs out of spare effort and abandons it; person B, by chance, gets better results and blogs about it etc).