This is something you can figure out from basic stats and your experimental design, and I strongly recommend actually running the numbers.
As it happens, I learned how to do basic power calculations not that long ago. I didn’t do an explicit calculation for the melatonin trial because I didn’t randomize selection, instead doing an alternating days design and not always following that, so I thought why bother doing one in retrospect?
But if we were to wave that away, the power seems fine. I have something like 141 days of data, of which around 90-100 is usable, giving me maybe <50 pairs? If I fire up R and load in the two means and the standard deviation (which I had left over from calculating the effect size), and then play with the numbers, then to get an 85% chance I could find an effect at p=0.01:
> pwr.t.test(d=(456.4783 - 407.5312) / 131.4656,power=0.85,sig.level=0.01,type="paired",alternative="greater")
Paired t test power calculation
n = 84.3067
d = 0.3723187
sig.level = 0.01
power = 0.85
alternative = greater
NOTE: n is number of *pairs*
If I drop the p=0.01 for 0.05, it looks like I should have had a good shot at detecting the effect:
> pwr.t.test(d=(456.4783 - 407.5312) / 131.4656,power=0.85,sig.level=0.05,type="paired",alternative="greater")
Paired t test power calculation
n = 53.24355
So, it’s not great, but it’s at least not terribly wrong?
EDIT: Just realized that I equivocated over days vs pairs in my existing power analyses; 1 was wrong, but I apparently avoided the error in another, phew.
As it happens, I learned how to do basic power calculations not that long ago. I didn’t do an explicit calculation for the melatonin trial because I didn’t randomize selection, instead doing an alternating days design and not always following that, so I thought why bother doing one in retrospect?
But if we were to wave that away, the power seems fine. I have something like 141 days of data, of which around 90-100 is usable, giving me maybe <50 pairs? If I fire up R and load in the two means and the standard deviation (which I had left over from calculating the effect size), and then play with the numbers, then to get an 85% chance I could find an effect at p=0.01:
If I drop the p=0.01 for 0.05, it looks like I should have had a good shot at detecting the effect:
So, it’s not great, but it’s at least not terribly wrong?
EDIT: Just realized that I equivocated over days vs pairs in my existing power analyses; 1 was wrong, but I apparently avoided the error in another, phew.