Drug development and genomics already use luck-based approaches. I wonder if doctors just worry about not looking respectable. A psychiatrist I know always seems a little defensive when the problem of finding the right med for a patient comes up, as if he were afraid others might see it as a bit seedy that he just tries things until something works.
Geneticists will silence every single gene in C. elegans to see the effect on longevity, then construct mechanistic hypotheses based on the the genes that had an effect.
If you read descriptions of how a genome-wide association study (GWAS) works, the field’s very clear that the method has trouble with:
Traits heavily determined by a small number of rare but but powerful alleles
Traits determined by a very large number of common but individually weak alleles
In animal studies, GWAS can be conplemented with QTL mapping to address these problems. Otherwise, you’re stuck with doing massive, massive studies on genetically diverse populations to try and get a full picture of the genetic architecture of a given trait.
In nutrition, it would not surprise me at all if Boswelia occasionally has a powerful effect for a small number of patients, but that studies of it in humans are underpowered. Boswelia is an herb containing multiple potential active ingredients, and I fully expect that different production processes and batch-to-batch variability would tend to confound studies, not to mention selection of different conditions, measured in different ways, among different patient populations—if the studies are done at all.
At some point, I’d like to consider whether there’s value in developing a protocol + software + device for facilitating personalized randomly controlled trials beyond whatever is the Pareto frontier for current enthusiasts. For example, do devices exist to facilitate self-blinded pill administration? Is there a way to help users collect and analyze the data without needing to be a statistics expert? Is it possible to make a logical list of things to try that’s better than an alphabetized list of herbs, medications, and other supplements?
It’s unlikely that I’ll have time to do this in the next year, but I do think about it from time to time.
At some point, I’d like to consider whether there’s value in developing a protocol + software + device for facilitating personalized randomly controlled trials beyond whatever is the Pareto frontier for current enthusiasts
I know someone who’s working on this and I love the idea, although thus far his app is absolutely useless for something that doesn’t kick in and wear off the same day and I haven’t verified it for that use case either. I also have an entrepreneur client whose very interested in this and would plausibly provide support to someone working on it, but it’s not his own top priority.
Let’s say your problem is reliable and obvious. Get a sample pack of pills and remedies. Take 1 pill at a time. If it’s a miracle cure, keep taking it. If not, try the next one. No fancy stuff required. Does anyone sell a sample pack like this?
Tl;dr
If your mystery malady is obvious, reliable, and rare in the general population, then you can simply try pills one at a time, one pill at a time, to see if they provide a reliable treatment. If so, then this will become apparent right away, and you can keep taking it a few more times to confirm it works reliably. It should only take a few trials to show the pill works for you. If the pill doesn’t work, you can set it aside after one try and try the next one.
This would not help you if the problem is a deficiency or excess of some compound that takes time to build up. It would only help if the pill had an immediate, obvious effect for your condition—a “miracle cure.”
This approach requires no fancy statistics, devices, or complexities of taking multiple pills at once. The easiest way to overcome the bottleneck would be if a company sold “sample packs” containing 1 pill of eat type. You’d take them one at a time, see if they fixed the problem, and if they did, you’d buy a larger quantity of that specific pill and keep experimenting to confirm the result.
Long:
I could imagine a fruitful collaboration between a mechanical engineer, a statistician, a programmer, and a biomedical researcher/doctor. The approach could be Bayes’ law-based. Using your problem as an example, here’s how it might work:
Does eating meat (cooked, store-bought, fresh, 50g) make me nauseous?
P(H): You specify how common it is in the human population for eating meat to cause nausea
P(E): You also specify how common it is for a member of the human population to eat meat and get nauseous, whether or not there’s a causal relationship
P(E|H): Finally, you specify how likely it would be for you to get nauseous after eating meat if eating meat makes you nauseous.
If you don’t have a good way to estimate these likelihoods, the app would give you suggestions for ways you could find reasonable numbers to plug in.
Then, every time you ate meat and got nauseous, you could enter another data point and it would update the likelihood that eating meat causes nausea.
Once you’re adequately confident that eating meat causes nausea, you could then repeat the exercise for a range of herbs.
Given that eating meat causes nausea, does taking Boswelia (BosPro) before eating prevent nausea from meat consumption?
P(H): You specify how common it is for people who get nauseous after eating meat to not get nauseous if they take Boswelia first.
P(E): You also specify how common it is for people who get nauseous from eating meat to take Boswelia, then eat meat, and not get nauseous.
P(E|H): Finally, you specify how likely it would be for you to not get nauseous after taking Boswelia, then eating meat, if taking Boswelia prevents nausea from meat consumption.
With a very reliable and unusual effect, as in your case, it’s possible to update a very low prior to a very confident prior in just a few trials.
I think the statistics would be doing most of the heavy lifting here. Being able to do a blinded placebo-controlled trial is important if you’re testing a relatively subtle/sporadic effect or testing a single compound.
But if you’re testing an obvious, reliable effect and experimenting with a range of compounds, I don’t think the self-blinding aspect is as important.
Here’s my attempt (not trained in this, correct me if I’m going about it wrong) to apply Bayesian inference to the problem.
For example, let’s say:
P(H), the chance that meat gives you nausea, is one in a million (.000001)
P(E), the likelihood of you just randomly feeling nauseous after eating meat, is one in a thousand (.001)
P(E|H), the likelihood you feel nauseous after eating meat, given that the meat causes you nausea, is 0.98
Then P(H|E), the chance that eating meat causes your nausea, given you ate meat and felt nauseous afterward, is 0.00098 after one time eating meat and getting nauseous. Feeding this back in as the new P(H), the chance that meat causes your nausea, P(H|E) becomes .9604, or 96.04%, the second time this happens (assuming you don’t eat meat and not feel nauseous). Realistically, we’d want to check if there’s something about the way you cooked the meat, the animal it comes from, the store you bought it from, the knowledge that you’re consuming meat, etc, that is confounding the result.
To deal with some of the confounds, you could repeat the experiment using different types of meat from different stores, and figure out how robust the effect is. It seems difficult to self-blind eating meat, since 50 g is about a sandwich’s worth of deli meat. Possibly you could try incorporating it into another more flavorful dish and eating the food with your eyes closed. But this is just to rule out the possibility that the result is from the perception of eating meat, as opposed to something primarily physiological about meat-material being in your digestive tract. If you weren’t concerned about this possibility, you could skip it.
But if the user is self-testing to find a treatment for a very reliable, obvious effect (as with your meat->nausea problem), and if our prior on any random pill just randomly working to prevent the nausea is low, then these results show you can probably figure out whether any random supplement works or not in just a few tries. That sort of obviates the need for fancy statistics or an app. You might be able to get much of the same result by just directing the user to “Try taking this pill, then eating 50 g meat. If the pill doesn’t work on the first try, move on to the next pill. If it does work, continue taking it. If it helps but doesn’t entirely eliminate the problem, explore dosing, timing, and manufacturer, to see if you can achieve significant improvements.”
I’m not sure if it makes sense to try multiple pills at once. Balancing the possibility that you stumble upon an effective treatment faster is the possibility that an effective treatment is disguished by a drug interaction, as well as the possibility that the average user just struggles to manage the added complexity, gets confused, and quits.
But this then becomes very simple.
If you have a reliable, obvious mystery malady that no doctor has treated effectively, systematically take 1 pill at a time until you find one with an effect. Then keep taking it, exploring dosage, timing, and manufacturer, until you maximize the effect. If satisfied, problem solved. If not fully satisfied, continue exploring other pills (unless it’s possible to devise a mechanistic hypothesis as to why the effective pill works, then use that to find likely alternatives more efficiently).
TBC: the client is interested scaleable software that people intuitively find useful for experimenting on themselves and can combine selfishly motivated individual data into useful aggregate data. But there’s a point here I want to argue.
I’m not sure if it makes sense to try multiple pills at once. Balancing the possibility that you stumble upon an effective treatment faster is the possibility that an effective treatment is disguished by a drug interaction, as well as the possibility that the average user just struggles to manage the added complexity, gets confused, and quits.
It seems like you’re assuming all drug interactions are bad. What if you need three things in combination to see the effect, and then it works really well?
That’s entirely possible. However, it introduces a level of risk that isn’t as present when you take one at a time. It’s unlikely that a given supplement will kill you at normal doses. Such a severe side effect would have been detected and the supplement most likely wouldn’t be sold.
But it’s not impossible that certain rare combinations of drugs might have lethal side effects in combination, and that you might stumble across such a combination by chance. The risk of this happening seems in my judgment to outweigh the potential benefit of finding the right drug for a chronic condition faster.
I’m not certain if it’s plausible an interaction between supplements could kill you in a few days. Even the examples here seem to take a while to work. The only potentially instantly lethal drugs I’ve seen are things like alcohol and heroine. But I would shy away from taking combinations—that’s just where my risk budget is at. I could be persuaded otherwise.
If you’re concerned about supplement interactions, you should be concerned about supplements effects at all. People should seek based on possible luck, but should know (and be able to tell doctors) what they’re putting into their bodies, and take contextual advice from relatives, friends, and community.
Collecting individuals susceptible to effective treatment with intervention X and having them ready for researchers to talk to, test more directly—is this already done? I suppose around individual notable conditions: celiac disease and gluten intolerance, or n=1 genetic issue self-diagnosis.
Some conditions have an intermittency, that makes it hard to assess interventions with unknown timing.
Perhaps blinded timing studies, self-studies, after something is found to work. Perhaps helping people to log and journal symptoms and effects during the blinded periods, as well as analyze, interpret, and share them—especially in ways that make it easier for others to trust.
At some point, I’d like to consider whether there’s value in developing a protocol + software + device for facilitating personalized randomly controlled trials beyond whatever is the Pareto frontier for current enthusiasts
Wow, yes. I’ve thought about this before and it feels like this would be so valuable. It’s been a while since I’ve looked into this space but as far as I’ve been able to tell the Pareto frontier is terrible.
I’m going to try to spend some time on this over the next few days.
Is there a way to help users collect and analyze the data without needing to be a statistics expert?
Collection is really just a matter of finding the right devices and taking the time to use them. Analysis outside of immediate obvious effect can become difficult. If the effect is subtle and drowned in other effects, or hard to measure. If the intervention is not something user can easily or wants to reproduce. If the effect take long time to build up, or is shifted in time from intervention. If the successful effect only happens under several conditions or several interventions together. If the spray and pray approach is dangerous. If the spray and pray approach only hits gold once in a while. Multiple comparison problem (see wikipedia). If user is bad at keeping records. There are probably more. There are many many apps that just do correlation and none that do anything more. Here is a list of both problems and apps.
I agree that the highest-leverage place to start is probably the paradigm of encouraging people with obvious long-lasting chronic problems to look for immediate obvious effects by doing maximal spray n pray. Equivalents of “have nausea from cancer/chemo daily → medical marijuana → no more problems.”
Once we get away from that, I think that systems for collection, analysis, and self-blinding become important. There are a lot of details and trivial inconveniences in any research project, and most people just aren’t equipped to work them out on their own. There’s a lot you can do to smooth the path.
For example, I can imagine a nootropics test kit. It would come with:
A standardized questionnaire that you fill out for every nootropic you try
A sample of nootropics to try along with placebos. Placebos would mimic the appearance of various drugs so it’s impossible to tell which is which without deliberately unblinding yourself. the supply would be large enough to give you adequate power given the number of drugs you’re trying.
An analytic framework that takes multiple comparisons etc. into account and lets you see if any correlations are statistically significant.
Perhaps packaging drugs in different ways so that you can order more of the things that work, but with a different appearance, to do a more focused experiment on the likeliest candidates.
There’s a lot of detail to work out in designing such a kit, but it’s easy for me to see that it could convert an intractable problem into a do-able puzzle for a motivated and reasonably intelligent user.
An analytic framework that takes multiple comparisons etc. into account and lets you see if any correlations are statistically significant.
Blinding.
Two issues, one of which I did not think of, out of like 20.
EDIT: I suspect, including from my own experience, that many problems can be solved without resorting to advanced statistics. Often by using through experimental procedure instead. Like eliminating a food type for a month then not doing an intervention for a month. Repeat. Trying out medications sounds like it should be done safely. This safety can only be achieved by monitoring vital signs and analyzing them using advanced statistics.
Drug development and genomics already use luck-based approaches. I wonder if doctors just worry about not looking respectable. A psychiatrist I know always seems a little defensive when the problem of finding the right med for a patient comes up, as if he were afraid others might see it as a bit seedy that he just tries things until something works.
Geneticists will silence every single gene in C. elegans to see the effect on longevity, then construct mechanistic hypotheses based on the the genes that had an effect.
If you read descriptions of how a genome-wide association study (GWAS) works, the field’s very clear that the method has trouble with:
Traits heavily determined by a small number of rare but but powerful alleles
Traits determined by a very large number of common but individually weak alleles
In animal studies, GWAS can be conplemented with QTL mapping to address these problems. Otherwise, you’re stuck with doing massive, massive studies on genetically diverse populations to try and get a full picture of the genetic architecture of a given trait.
In nutrition, it would not surprise me at all if Boswelia occasionally has a powerful effect for a small number of patients, but that studies of it in humans are underpowered. Boswelia is an herb containing multiple potential active ingredients, and I fully expect that different production processes and batch-to-batch variability would tend to confound studies, not to mention selection of different conditions, measured in different ways, among different patient populations—if the studies are done at all.
At some point, I’d like to consider whether there’s value in developing a protocol + software + device for facilitating personalized randomly controlled trials beyond whatever is the Pareto frontier for current enthusiasts. For example, do devices exist to facilitate self-blinded pill administration? Is there a way to help users collect and analyze the data without needing to be a statistics expert? Is it possible to make a logical list of things to try that’s better than an alphabetized list of herbs, medications, and other supplements?
It’s unlikely that I’ll have time to do this in the next year, but I do think about it from time to time.
I know someone who’s working on this and I love the idea, although thus far his app is absolutely useless for something that doesn’t kick in and wear off the same day and I haven’t verified it for that use case either. I also have an entrepreneur client whose very interested in this and would plausibly provide support to someone working on it, but it’s not his own top priority.
Tl;dr is tl;dr
Let’s say your problem is reliable and obvious. Get a sample pack of pills and remedies. Take 1 pill at a time. If it’s a miracle cure, keep taking it. If not, try the next one. No fancy stuff required. Does anyone sell a sample pack like this?
Tl;dr
If your mystery malady is obvious, reliable, and rare in the general population, then you can simply try pills one at a time, one pill at a time, to see if they provide a reliable treatment. If so, then this will become apparent right away, and you can keep taking it a few more times to confirm it works reliably. It should only take a few trials to show the pill works for you. If the pill doesn’t work, you can set it aside after one try and try the next one.
This would not help you if the problem is a deficiency or excess of some compound that takes time to build up. It would only help if the pill had an immediate, obvious effect for your condition—a “miracle cure.”
This approach requires no fancy statistics, devices, or complexities of taking multiple pills at once. The easiest way to overcome the bottleneck would be if a company sold “sample packs” containing 1 pill of eat type. You’d take them one at a time, see if they fixed the problem, and if they did, you’d buy a larger quantity of that specific pill and keep experimenting to confirm the result.
Long:
I could imagine a fruitful collaboration between a mechanical engineer, a statistician, a programmer, and a biomedical researcher/doctor. The approach could be Bayes’ law-based. Using your problem as an example, here’s how it might work:
Does eating meat (cooked, store-bought, fresh, 50g) make me nauseous?
P(H): You specify how common it is in the human population for eating meat to cause nausea
P(E): You also specify how common it is for a member of the human population to eat meat and get nauseous, whether or not there’s a causal relationship
P(E|H): Finally, you specify how likely it would be for you to get nauseous after eating meat if eating meat makes you nauseous.
If you don’t have a good way to estimate these likelihoods, the app would give you suggestions for ways you could find reasonable numbers to plug in.
Then, every time you ate meat and got nauseous, you could enter another data point and it would update the likelihood that eating meat causes nausea.
Once you’re adequately confident that eating meat causes nausea, you could then repeat the exercise for a range of herbs.
Given that eating meat causes nausea, does taking Boswelia (BosPro) before eating prevent nausea from meat consumption?
P(H): You specify how common it is for people who get nauseous after eating meat to not get nauseous if they take Boswelia first.
P(E): You also specify how common it is for people who get nauseous from eating meat to take Boswelia, then eat meat, and not get nauseous.
P(E|H): Finally, you specify how likely it would be for you to not get nauseous after taking Boswelia, then eating meat, if taking Boswelia prevents nausea from meat consumption.
With a very reliable and unusual effect, as in your case, it’s possible to update a very low prior to a very confident prior in just a few trials.
I think the statistics would be doing most of the heavy lifting here. Being able to do a blinded placebo-controlled trial is important if you’re testing a relatively subtle/sporadic effect or testing a single compound.
But if you’re testing an obvious, reliable effect and experimenting with a range of compounds, I don’t think the self-blinding aspect is as important.
Here’s my attempt (not trained in this, correct me if I’m going about it wrong) to apply Bayesian inference to the problem.
For example, let’s say:
P(H), the chance that meat gives you nausea, is one in a million (.000001)
P(E), the likelihood of you just randomly feeling nauseous after eating meat, is one in a thousand (.001)
P(E|H), the likelihood you feel nauseous after eating meat, given that the meat causes you nausea, is 0.98
Then P(H|E), the chance that eating meat causes your nausea, given you ate meat and felt nauseous afterward, is 0.00098 after one time eating meat and getting nauseous. Feeding this back in as the new P(H), the chance that meat causes your nausea, P(H|E) becomes .9604, or 96.04%, the second time this happens (assuming you don’t eat meat and not feel nauseous). Realistically, we’d want to check if there’s something about the way you cooked the meat, the animal it comes from, the store you bought it from, the knowledge that you’re consuming meat, etc, that is confounding the result.
To deal with some of the confounds, you could repeat the experiment using different types of meat from different stores, and figure out how robust the effect is. It seems difficult to self-blind eating meat, since 50 g is about a sandwich’s worth of deli meat. Possibly you could try incorporating it into another more flavorful dish and eating the food with your eyes closed. But this is just to rule out the possibility that the result is from the perception of eating meat, as opposed to something primarily physiological about meat-material being in your digestive tract. If you weren’t concerned about this possibility, you could skip it.
But if the user is self-testing to find a treatment for a very reliable, obvious effect (as with your meat->nausea problem), and if our prior on any random pill just randomly working to prevent the nausea is low, then these results show you can probably figure out whether any random supplement works or not in just a few tries. That sort of obviates the need for fancy statistics or an app. You might be able to get much of the same result by just directing the user to “Try taking this pill, then eating 50 g meat. If the pill doesn’t work on the first try, move on to the next pill. If it does work, continue taking it. If it helps but doesn’t entirely eliminate the problem, explore dosing, timing, and manufacturer, to see if you can achieve significant improvements.”
I’m not sure if it makes sense to try multiple pills at once. Balancing the possibility that you stumble upon an effective treatment faster is the possibility that an effective treatment is disguished by a drug interaction, as well as the possibility that the average user just struggles to manage the added complexity, gets confused, and quits.
But this then becomes very simple.
If you have a reliable, obvious mystery malady that no doctor has treated effectively, systematically take 1 pill at a time until you find one with an effect. Then keep taking it, exploring dosage, timing, and manufacturer, until you maximize the effect. If satisfied, problem solved. If not fully satisfied, continue exploring other pills (unless it’s possible to devise a mechanistic hypothesis as to why the effective pill works, then use that to find likely alternatives more efficiently).
TBC: the client is interested scaleable software that people intuitively find useful for experimenting on themselves and can combine selfishly motivated individual data into useful aggregate data. But there’s a point here I want to argue.
It seems like you’re assuming all drug interactions are bad. What if you need three things in combination to see the effect, and then it works really well?
That’s entirely possible. However, it introduces a level of risk that isn’t as present when you take one at a time. It’s unlikely that a given supplement will kill you at normal doses. Such a severe side effect would have been detected and the supplement most likely wouldn’t be sold.
But it’s not impossible that certain rare combinations of drugs might have lethal side effects in combination, and that you might stumble across such a combination by chance. The risk of this happening seems in my judgment to outweigh the potential benefit of finding the right drug for a chronic condition faster.
Examples: https://www.google.com/amp/s/www.news24.com/amp/health24/medical/backache/news/7-medication-combinations-that-could-be-deadly-20160829
I’m not certain if it’s plausible an interaction between supplements could kill you in a few days. Even the examples here seem to take a while to work. The only potentially instantly lethal drugs I’ve seen are things like alcohol and heroine. But I would shy away from taking combinations—that’s just where my risk budget is at. I could be persuaded otherwise.
The blinded aspect is hard.
If you’re concerned about supplement interactions, you should be concerned about supplements effects at all. People should seek based on possible luck, but should know (and be able to tell doctors) what they’re putting into their bodies, and take contextual advice from relatives, friends, and community.
Collecting individuals susceptible to effective treatment with intervention X and having them ready for researchers to talk to, test more directly—is this already done? I suppose around individual notable conditions: celiac disease and gluten intolerance, or n=1 genetic issue self-diagnosis.
Some conditions have an intermittency, that makes it hard to assess interventions with unknown timing.
Perhaps blinded timing studies, self-studies, after something is found to work. Perhaps helping people to log and journal symptoms and effects during the blinded periods, as well as analyze, interpret, and share them—especially in ways that make it easier for others to trust.
Wow, yes. I’ve thought about this before and it feels like this would be so valuable. It’s been a while since I’ve looked into this space but as far as I’ve been able to tell the Pareto frontier is terrible.
I’m going to try to spend some time on this over the next few days.
Collection is really just a matter of finding the right devices and taking the time to use them. Analysis outside of immediate obvious effect can become difficult. If the effect is subtle and drowned in other effects, or hard to measure. If the intervention is not something user can easily or wants to reproduce. If the effect take long time to build up, or is shifted in time from intervention. If the successful effect only happens under several conditions or several interventions together. If the spray and pray approach is dangerous. If the spray and pray approach only hits gold once in a while. Multiple comparison problem (see wikipedia). If user is bad at keeping records. There are probably more. There are many many apps that just do correlation and none that do anything more. Here is a list of both problems and apps.
Your link is broken, and while Wikipedia may be a guide to problems, generically, I’m curious about the apps, and the problems specifically relevant.
https://wiki.openhumans.org/wiki/Finding_relations_between_variables_in_time_series This is the link I meant to post.
I agree that the highest-leverage place to start is probably the paradigm of encouraging people with obvious long-lasting chronic problems to look for immediate obvious effects by doing maximal spray n pray. Equivalents of “have nausea from cancer/chemo daily → medical marijuana → no more problems.”
Once we get away from that, I think that systems for collection, analysis, and self-blinding become important. There are a lot of details and trivial inconveniences in any research project, and most people just aren’t equipped to work them out on their own. There’s a lot you can do to smooth the path.
For example, I can imagine a nootropics test kit. It would come with:
A standardized questionnaire that you fill out for every nootropic you try
A sample of nootropics to try along with placebos. Placebos would mimic the appearance of various drugs so it’s impossible to tell which is which without deliberately unblinding yourself. the supply would be large enough to give you adequate power given the number of drugs you’re trying.
An analytic framework that takes multiple comparisons etc. into account and lets you see if any correlations are statistically significant.
Perhaps packaging drugs in different ways so that you can order more of the things that work, but with a different appearance, to do a more focused experiment on the likeliest candidates.
There’s a lot of detail to work out in designing such a kit, but it’s easy for me to see that it could convert an intractable problem into a do-able puzzle for a motivated and reasonably intelligent user.
Two issues, one of which I did not think of, out of like 20.
EDIT: I suspect, including from my own experience, that many problems can be solved without resorting to advanced statistics. Often by using through experimental procedure instead. Like eliminating a food type for a month then not doing an intervention for a month. Repeat. Trying out medications sounds like it should be done safely. This safety can only be achieved by monitoring vital signs and analyzing them using advanced statistics.