Is there a way to help users collect and analyze the data without needing to be a statistics expert?
Collection is really just a matter of finding the right devices and taking the time to use them. Analysis outside of immediate obvious effect can become difficult. If the effect is subtle and drowned in other effects, or hard to measure. If the intervention is not something user can easily or wants to reproduce. If the effect take long time to build up, or is shifted in time from intervention. If the successful effect only happens under several conditions or several interventions together. If the spray and pray approach is dangerous. If the spray and pray approach only hits gold once in a while. Multiple comparison problem (see wikipedia). If user is bad at keeping records. There are probably more. There are many many apps that just do correlation and none that do anything more. Here is a list of both problems and apps.
I agree that the highest-leverage place to start is probably the paradigm of encouraging people with obvious long-lasting chronic problems to look for immediate obvious effects by doing maximal spray n pray. Equivalents of “have nausea from cancer/chemo daily → medical marijuana → no more problems.”
Once we get away from that, I think that systems for collection, analysis, and self-blinding become important. There are a lot of details and trivial inconveniences in any research project, and most people just aren’t equipped to work them out on their own. There’s a lot you can do to smooth the path.
For example, I can imagine a nootropics test kit. It would come with:
A standardized questionnaire that you fill out for every nootropic you try
A sample of nootropics to try along with placebos. Placebos would mimic the appearance of various drugs so it’s impossible to tell which is which without deliberately unblinding yourself. the supply would be large enough to give you adequate power given the number of drugs you’re trying.
An analytic framework that takes multiple comparisons etc. into account and lets you see if any correlations are statistically significant.
Perhaps packaging drugs in different ways so that you can order more of the things that work, but with a different appearance, to do a more focused experiment on the likeliest candidates.
There’s a lot of detail to work out in designing such a kit, but it’s easy for me to see that it could convert an intractable problem into a do-able puzzle for a motivated and reasonably intelligent user.
An analytic framework that takes multiple comparisons etc. into account and lets you see if any correlations are statistically significant.
Blinding.
Two issues, one of which I did not think of, out of like 20.
EDIT: I suspect, including from my own experience, that many problems can be solved without resorting to advanced statistics. Often by using through experimental procedure instead. Like eliminating a food type for a month then not doing an intervention for a month. Repeat. Trying out medications sounds like it should be done safely. This safety can only be achieved by monitoring vital signs and analyzing them using advanced statistics.
Collection is really just a matter of finding the right devices and taking the time to use them. Analysis outside of immediate obvious effect can become difficult. If the effect is subtle and drowned in other effects, or hard to measure. If the intervention is not something user can easily or wants to reproduce. If the effect take long time to build up, or is shifted in time from intervention. If the successful effect only happens under several conditions or several interventions together. If the spray and pray approach is dangerous. If the spray and pray approach only hits gold once in a while. Multiple comparison problem (see wikipedia). If user is bad at keeping records. There are probably more. There are many many apps that just do correlation and none that do anything more. Here is a list of both problems and apps.
Your link is broken, and while Wikipedia may be a guide to problems, generically, I’m curious about the apps, and the problems specifically relevant.
https://wiki.openhumans.org/wiki/Finding_relations_between_variables_in_time_series This is the link I meant to post.
I agree that the highest-leverage place to start is probably the paradigm of encouraging people with obvious long-lasting chronic problems to look for immediate obvious effects by doing maximal spray n pray. Equivalents of “have nausea from cancer/chemo daily → medical marijuana → no more problems.”
Once we get away from that, I think that systems for collection, analysis, and self-blinding become important. There are a lot of details and trivial inconveniences in any research project, and most people just aren’t equipped to work them out on their own. There’s a lot you can do to smooth the path.
For example, I can imagine a nootropics test kit. It would come with:
A standardized questionnaire that you fill out for every nootropic you try
A sample of nootropics to try along with placebos. Placebos would mimic the appearance of various drugs so it’s impossible to tell which is which without deliberately unblinding yourself. the supply would be large enough to give you adequate power given the number of drugs you’re trying.
An analytic framework that takes multiple comparisons etc. into account and lets you see if any correlations are statistically significant.
Perhaps packaging drugs in different ways so that you can order more of the things that work, but with a different appearance, to do a more focused experiment on the likeliest candidates.
There’s a lot of detail to work out in designing such a kit, but it’s easy for me to see that it could convert an intractable problem into a do-able puzzle for a motivated and reasonably intelligent user.
Two issues, one of which I did not think of, out of like 20.
EDIT: I suspect, including from my own experience, that many problems can be solved without resorting to advanced statistics. Often by using through experimental procedure instead. Like eliminating a food type for a month then not doing an intervention for a month. Repeat. Trying out medications sounds like it should be done safely. This safety can only be achieved by monitoring vital signs and analyzing them using advanced statistics.