I disagree with your comment on “good stimulant hygiene”—with long term lose dose amphetamine use people often seem to develop a tolerance to the side effects (insomnia, wired feeling) while remaining sensitive to the effect of increased focus. This is what makes it an effective ADD drug. A good experiment on it’s effects would alternate use and non-use over long stable time periods (weeks or longer).
I’d be curious to see the effects of your melatonin experiment performed with a blind placebo control. One confusing thing is that you appear to get less sleep on melatonin, but you’re saying this is a good thing (reduced need for sleep). Is there some objective way to confirm that it’s not simply reducing your sleep (as a stimulant would) as opposed to improving sleep (and therefore reducing the time you need to spend sleeping)? Perhaps you could try to see if there’s a measurable rebound effect that would distinguish improved sleep from sleep deprivation. Is your need for sleep greater than normal after taking it for several days?
I’m a big fan of your self experiments, and have previously read most of them on your website… and have used many of your ideas to design my own. However, it seems surprisingly difficult to design a self experiment which rigorously tests only one hypothesis.
Is there some objective way to confirm that it’s not simply reducing your sleep (as a stimulant would) as opposed to improving sleep (and therefore reducing the time you need to spend sleeping)?
Sleep quality is typically measured by “number of times awakened” or “amount moved” or so on- my experience with melatonin (and I believe gwern’s as well, but I didn’t check) is that melatonin decreases the number of times I awaken during the night.
But even if you measure that, it’s just a proxy. A paralytic drug will reduce the amount I toss and turn at night, but may not improve how I feel the next day. What you would want to do is measure energy level / creativity, but that’s even more difficult.
I suppose measuring “blocks of time where I spent at least 30 minutes doing focused, productive work” would be an easy thing to correlate against having taken melatonin the night before, but I think that it would be very difficult to measure an effect with such low resolution data.
Interesting… I already use that exact method, but never heard that term for it. I first read about it in the “Now Habit” by Neil Fiore, a strategic system for overcoming procrastination. It works incredibly well!
Low-resolution doesn’t really matter; what matters is how variable the data is. If you have a binary variable—as crude resolution as possible—which rarely flips, then an intervention which occasionally flips it will still be noticeable.
(How much data would it take? Well, that’s hard to estimate without any data at all...)
That makes sense, I guess all of the other unknown variables would serve to dither the low resolution data and it would work for the same reason that CDs sound good with only 16 bits of resolution.
I am going to begin a random placebo controlled trial of melatonin use, quantified against my ZEO and work logs. I just need to find some opaque capsules.
Thanks. The experiment starts today. I made up 14 pills and randomized them with R code, under the assumption that I couldn’t possibly subconsciously track each pill and then remember their locations. In hindsight, this is probably more labor intensive and error prone than the method you used to randomize your adderall...
I start with a 14 tray pill box with treatment (0.75mg melatonin + parsely) in the first 7 boxes, and placebo (parsely pills only) in the last 7. I randomly reorder them 3 times, according to each of the lists outputted by my code, and it writes the final pill locations to a text file.
Here’s my R code for randomizing “single blind” placebo controlled self-trials:
Interesting procedure. I’d agree it’s probably much more work than some simple physical procedure. (I’d also point out that 14 pairs won’t give you much significance—my above power analysis suggested that for awakenings, anyway, you’d want more like 140 pairs. But I should be happy you’re actually doing the experiment.)
Hey gwern, as you predicted I didn’t have enough data to learn anything… and I didn’t have time to do it longer. I considered repeating it, but now I’m scared off melatonin until I learn more about how it works. Dr. Ray Peat theorizes that it might have some negative health effects by inhibiting oxidative metabolism:
Also, anecdotally I don’t really see a huge benefit to melatonin. Even small doses (0.75mg) seem to make me slightly groggy when I wake 7.5 hours later. I may have unusually slow melatonin metabolism, as I have the “slow caffeine metabolizer” P450 CYP1A2 variant, the same enzyme responsible for clearing melatonin.
For my melatonin experiment, number of awakenings was 2.86 vs 2.43, but the p-value was only 0.43. The problem is that the standard deviation is 2.25! (On many nights, I awaken zero or one times, but on one particularly bad night, I woke up 7 times.) I suspect more data would show a more reliable effect and maybe a greater effect size than d=0.19.
I disagree with your comment on “good stimulant hygiene”—with long term lose dose amphetamine use people often seem to develop a tolerance to the side effects (insomnia, wired feeling) while remaining sensitive to the effect of increased focus. This is what makes it an effective ADD drug. A good experiment on it’s effects would alternate use and non-use over long stable time periods (weeks or longer).
I’d be curious to see the effects of your melatonin experiment performed with a blind placebo control. One confusing thing is that you appear to get less sleep on melatonin, but you’re saying this is a good thing (reduced need for sleep). Is there some objective way to confirm that it’s not simply reducing your sleep (as a stimulant would) as opposed to improving sleep (and therefore reducing the time you need to spend sleeping)? Perhaps you could try to see if there’s a measurable rebound effect that would distinguish improved sleep from sleep deprivation. Is your need for sleep greater than normal after taking it for several days?
I’m a big fan of your self experiments, and have previously read most of them on your website… and have used many of your ideas to design my own. However, it seems surprisingly difficult to design a self experiment which rigorously tests only one hypothesis.
Sleep quality is typically measured by “number of times awakened” or “amount moved” or so on- my experience with melatonin (and I believe gwern’s as well, but I didn’t check) is that melatonin decreases the number of times I awaken during the night.
But even if you measure that, it’s just a proxy. A paralytic drug will reduce the amount I toss and turn at night, but may not improve how I feel the next day. What you would want to do is measure energy level / creativity, but that’s even more difficult.
I suppose measuring “blocks of time where I spent at least 30 minutes doing focused, productive work” would be an easy thing to correlate against having taken melatonin the night before, but I think that it would be very difficult to measure an effect with such low resolution data.
This is even easier if you use pomodoro, because measuring productive time and noticing unproductive time is much easier.
Interesting… I already use that exact method, but never heard that term for it. I first read about it in the “Now Habit” by Neil Fiore, a strategic system for overcoming procrastination. It works incredibly well!
Low-resolution doesn’t really matter; what matters is how variable the data is. If you have a binary variable—as crude resolution as possible—which rarely flips, then an intervention which occasionally flips it will still be noticeable.
(How much data would it take? Well, that’s hard to estimate without any data at all...)
That makes sense, I guess all of the other unknown variables would serve to dither the low resolution data and it would work for the same reason that CDs sound good with only 16 bits of resolution.
I am going to begin a random placebo controlled trial of melatonin use, quantified against my ZEO and work logs. I just need to find some opaque capsules.
I look forward to your results.
Thanks. The experiment starts today. I made up 14 pills and randomized them with R code, under the assumption that I couldn’t possibly subconsciously track each pill and then remember their locations. In hindsight, this is probably more labor intensive and error prone than the method you used to randomize your adderall...
I start with a 14 tray pill box with treatment (0.75mg melatonin + parsely) in the first 7 boxes, and placebo (parsely pills only) in the last 7. I randomly reorder them 3 times, according to each of the lists outputted by my code, and it writes the final pill locations to a text file.
Here’s my R code for randomizing “single blind” placebo controlled self-trials:
sampleSize ← 14
intialSetup ← c(rep(“treatment”, sampleSize/2), rep(“placebo”, sampleSize/2))
reorder1 ← sample(1:sampleSize, sampleSize, replace = FALSE)
reorder2 ← sample(1:sampleSize, sampleSize, replace = FALSE)
reorder3 ← sample(1:sampleSize, sampleSize, replace = FALSE)
final ← intialSetup[reorder1][reorder2][reorder3]
write.table(final, “final.txt”)
paste(reorder1, “->”, 1:sampleSize)
paste(reorder2, “->”, 1:sampleSize)
paste(reorder3, “->”, 1:sampleSize)
Interesting procedure. I’d agree it’s probably much more work than some simple physical procedure. (I’d also point out that 14 pairs won’t give you much significance—my above power analysis suggested that for awakenings, anyway, you’d want more like 140 pairs. But I should be happy you’re actually doing the experiment.)
I plan to do much more than 14, but it was very tedious to set up so I started with that. I need to streamline the procedure.
How has the experiment been going?
Hey gwern, as you predicted I didn’t have enough data to learn anything… and I didn’t have time to do it longer. I considered repeating it, but now I’m scared off melatonin until I learn more about how it works. Dr. Ray Peat theorizes that it might have some negative health effects by inhibiting oxidative metabolism:
http://www.google.com/cse?cx=005233684413389937395%3Ad5qfhqsz7oo&ie=UTF-8&q=melatonin#gsc.tab=0&gsc.q=melatonin&gsc.page=1
Also, anecdotally I don’t really see a huge benefit to melatonin. Even small doses (0.75mg) seem to make me slightly groggy when I wake 7.5 hours later. I may have unusually slow melatonin metabolism, as I have the “slow caffeine metabolizer” P450 CYP1A2 variant, the same enzyme responsible for clearing melatonin.
For my melatonin experiment, number of awakenings was 2.86 vs 2.43, but the p-value was only 0.43. The problem is that the standard deviation is 2.25! (On many nights, I awaken zero or one times, but on one particularly bad night, I woke up 7 times.) I suspect more data would show a more reliable effect and maybe a greater effect size than d=0.19.
To expand; if d<=0.19, to detect this effect at p<0.05 with 75% odds, we need ~75 pairs of nights or ~150 nights of data:
Doable, but not trivial.