Doesn’t stop people from generating real information, right?
That’s the damnable thing about these sorts of biases, it’s not clear to me whether one can compensate for the biases. If you pay attention to the ‘real information’, you may wind up learning what we might call anti-information—information that predictably and systematically makes your beliefs worse than your defaults.
This is the problem of the clever arguer: how do you, and can you, adjust for the fact that all the reports coming out about Soylent are so deeply error-prone (and now with the kickstarter, we get a delicious cherry on top of conflicts of interest)?
I would much rather have a handful of randomized self-experiments from some obscure blogger interested in a weird recipe he came up with than a forum of Soylent enthusiasts raving to each other that the latest formulation on sale is the greatest ever and telling each other that if you feel bad you should be eating your Soylent twice a day and not three times a day, don’t you know about intermittent fasting?, and also I ran 20 seconds faster today, so Soylent must be working for me!
Ok, I see what your concern is, with the hype around Soylent everyone’s opinion is skewed (even if they’re not among the fanboys).
You decided above that it wasn’t worth your time to try your own self-experiments with it. What if someone else were to take the time to do it? I like the concept but agree with the major troubles you listed above, and I have no experience with designing self-experiments. But maybe I’ll take the time to try and do it properly, long-term, with regular blood tests, noting what I’ve been eating for a couple months before starting, taking data about my fitness levels, etc. Of course, I would need to analyze the risk to myself beforehand.
What if someone else were to take the time to do it?
If they actually go through with it and write it up, that’s better than the status quo, yes. But if they don’t determine to go through with it and may give up, it’s another selection bias, specifically, publication bias (person A does a self-experiment but halfway through runs out of spare effort and abandons it; person B, by chance, gets better results and blogs about it etc).
That’s the damnable thing about these sorts of biases, it’s not clear to me whether one can compensate for the biases. If you pay attention to the ‘real information’, you may wind up learning what we might call anti-information—information that predictably and systematically makes your beliefs worse than your defaults.
This is the problem of the clever arguer: how do you, and can you, adjust for the fact that all the reports coming out about Soylent are so deeply error-prone (and now with the kickstarter, we get a delicious cherry on top of conflicts of interest)?
I would much rather have a handful of randomized self-experiments from some obscure blogger interested in a weird recipe he came up with than a forum of Soylent enthusiasts raving to each other that the latest formulation on sale is the greatest ever and telling each other that if you feel bad you should be eating your Soylent twice a day and not three times a day, don’t you know about intermittent fasting?, and also I ran 20 seconds faster today, so Soylent must be working for me!
Ok, I see what your concern is, with the hype around Soylent everyone’s opinion is skewed (even if they’re not among the fanboys).
You decided above that it wasn’t worth your time to try your own self-experiments with it. What if someone else were to take the time to do it? I like the concept but agree with the major troubles you listed above, and I have no experience with designing self-experiments. But maybe I’ll take the time to try and do it properly, long-term, with regular blood tests, noting what I’ve been eating for a couple months before starting, taking data about my fitness levels, etc. Of course, I would need to analyze the risk to myself beforehand.
If they actually go through with it and write it up, that’s better than the status quo, yes. But if they don’t determine to go through with it and may give up, it’s another selection bias, specifically, publication bias (person A does a self-experiment but halfway through runs out of spare effort and abandons it; person B, by chance, gets better results and blogs about it etc).