Since I don’t like the commenting options on your blog, I’m going to respond here to your melatonin experiment.
Is there anything I can (easily) do to fix this?
Comparing two periods is statistically dangerous because so many other details might have changed between them. It is good that you are going to do two more periods, but I think you should do many trials of shorter periods, say, one week each. The drawback is that sleep has cumulative aspects.
The reason I am doing longish trial periods is to test the “addiction/withdrawal objection” to melatonin use. Although, now that you mention it, week long periods should work for this. Also, I was thinking about using the interquartile range to eliminate the outliers from each month for the final analysis. Do you think that would be an appropriate technique?
Gwern suggests that you try smaller doses. I suggest that you try much smaller doses: 0.3mg. Or maybe cut those up.
Yeah, I was planning on trying 3mg, 2mg, and 1mg. Based on your and Gwern’s feedback, I’m probably going to end up testing 3mg, 1mg, and 0.5mg instead.
You can probably turn on “anonymous” commenting, meaning without registration.
What is the limit on doing experiments? I got the impression was that if the next 2 month cycle showed an advantage to melatonin, you were going to take it regularly. But if you plan on doing month long experiments for the next two years, your will avoid my complaint, though I think you would be better off doing shorter experiments.
You don’t need to worry about tolerance until after you have established that the drug works in the short term, so if you can save time by not worrying about it, you should. If you decide you want to use it long term, you should do month long experiments at that point. Note that you have not yet done a withdrawal experiment, though you have done something of a tolerance experiment.
Yes, the interquartile range is a great metric. You should compute it. But you should also compute metrics that are sensitive to outliers. Improving outliers may well be more valuable than improving typical values.
You are anchoring on the doses available in stores. Clinical studies often use 0.01mg. Given what is easily available, I suggest 0.15, 0.6, and 3, an approximate geometric progression. If you can find something smaller, use it and tell me where you found it.
Is there anything I can (easily) do to fix this?
The reason I am doing longish trial periods is to test the “addiction/withdrawal objection” to melatonin use. Although, now that you mention it, week long periods should work for this. Also, I was thinking about using the interquartile range to eliminate the outliers from each month for the final analysis. Do you think that would be an appropriate technique?
Yeah, I was planning on trying 3mg, 2mg, and 1mg. Based on your and Gwern’s feedback, I’m probably going to end up testing 3mg, 1mg, and 0.5mg instead.
You can probably turn on “anonymous” commenting, meaning without registration.
What is the limit on doing experiments? I got the impression was that if the next 2 month cycle showed an advantage to melatonin, you were going to take it regularly. But if you plan on doing month long experiments for the next two years, your will avoid my complaint, though I think you would be better off doing shorter experiments.
You don’t need to worry about tolerance until after you have established that the drug works in the short term, so if you can save time by not worrying about it, you should. If you decide you want to use it long term, you should do month long experiments at that point. Note that you have not yet done a withdrawal experiment, though you have done something of a tolerance experiment.
Yes, the interquartile range is a great metric. You should compute it. But you should also compute metrics that are sensitive to outliers. Improving outliers may well be more valuable than improving typical values.
You are anchoring on the doses available in stores. Clinical studies often use 0.01mg. Given what is easily available, I suggest 0.15, 0.6, and 3, an approximate geometric progression. If you can find something smaller, use it and tell me where you found it.