The measurement of lots of other things leads to the pathology of data mining rather than trying to find the correct causative variable. The better experimental technique is to sequentially investigate each confounding variable and try to ensure they are eliminated. Sometimes this can be hard, but that is no excuse not to do properly CONTROLLED experiments rather than reporting noise.
Data mining is so problematic that medical journals have insisted that the experiment hypothesis is defined in advance so that unexpected variables with significant p-values are not reported instead (p-hacking).
I would far rather experimenters do the 1-bit experiment and then if the result doesn’t falsify the hypothesis, think about other explanations for the result and check those variables in the same way. Good experimentation is not for the lazy.
Sometimes this can be hard, but that is no excuse not to do properly CONTROLLED experiments rather than reporting noise.
Sometimes, people accept the hard work of doing properly CONTROLLED experiments. Then they not only FAIL COMPLETELY, they think they succeeded—and everybody else is tricked into believing it too. That’s what happened in John’s A/B test.
For unsolved problems, you can only find the correct causative variable with a “firehose of information.” Then you can go on to prove you’re right via a properly controlled experiment.
A classic unreplicable p-hacked study is the one where they found relationships between a discount on an Italian buffet and what the diners ate.
If I was those researchers, I wouldn’t mind gathering the data. That’s the firehose. But I wouldn’t want to publish it. Instead, I’d scour the data, combine it with my qualitative observations of the event, and see if we could come up with a specific pre-registerable causal hypothesis we could actually believe in for a follow-up study. Only the follow-up would be published.
“For unsolved problems, you can only find the correct causative variable with a “firehose of information.” Then you can go on to prove you’re right via a properly controlled experiment.”
That second part often doesn’t happen. For [bio]medical experiments it is just too expensive. Datamning ensues and any significant p value variables are then published. The medical journals are rife with this which is one reason 30-50% of medical research proves unrepeatable.
Never underestimate human nature to do the easiest thing rather than the correct one. Science can be painstakingly hard to get right, but the pressures to publish are high. I’ve seen it first hand in biotech, where the obvious questions to ask of the “result” were ignored.
I also am in biotech, and I agree these problems exist.
One way of making use of the “firehose of information” in biotech would be to insist that researchers publish their raw datasets, and provide additional supplementary information along with their paper. Imagine, for example, if researchers doing animal work were required to film themselves doing it and post the videos online for others to review. I think it’s easy to see how that would be a helpful “firehose of information” and would do a lot to flesh out the picture given by the normally reported figures in a publication.
I think you’re worried about people switching from hard analysis to squishier qualitative data, perhaps because resources are already so constrained that it feels like “one or the other.” I think John’s saying “why not both?”
The measurement of lots of other things leads to the pathology of data mining rather than trying to find the correct causative variable. The better experimental technique is to sequentially investigate each confounding variable and try to ensure they are eliminated. Sometimes this can be hard, but that is no excuse not to do properly CONTROLLED experiments rather than reporting noise.
Data mining is so problematic that medical journals have insisted that the experiment hypothesis is defined in advance so that unexpected variables with significant p-values are not reported instead (p-hacking).
I would far rather experimenters do the 1-bit experiment and then if the result doesn’t falsify the hypothesis, think about other explanations for the result and check those variables in the same way. Good experimentation is not for the lazy.
Sometimes, people accept the hard work of doing properly CONTROLLED experiments. Then they not only FAIL COMPLETELY, they think they succeeded—and everybody else is tricked into believing it too. That’s what happened in John’s A/B test.
For unsolved problems, you can only find the correct causative variable with a “firehose of information.” Then you can go on to prove you’re right via a properly controlled experiment.
A classic unreplicable p-hacked study is the one where they found relationships between a discount on an Italian buffet and what the diners ate.
If I was those researchers, I wouldn’t mind gathering the data. That’s the firehose. But I wouldn’t want to publish it. Instead, I’d scour the data, combine it with my qualitative observations of the event, and see if we could come up with a specific pre-registerable causal hypothesis we could actually believe in for a follow-up study. Only the follow-up would be published.
“For unsolved problems, you can only find the correct causative variable with a “firehose of information.” Then you can go on to prove you’re right via a properly controlled experiment.”
That second part often doesn’t happen. For [bio]medical experiments it is just too expensive. Datamning ensues and any significant p value variables are then published. The medical journals are rife with this which is one reason 30-50% of medical research proves unrepeatable.
Never underestimate human nature to do the easiest thing rather than the correct one. Science can be painstakingly hard to get right, but the pressures to publish are high. I’ve seen it first hand in biotech, where the obvious questions to ask of the “result” were ignored.
I also am in biotech, and I agree these problems exist.
One way of making use of the “firehose of information” in biotech would be to insist that researchers publish their raw datasets, and provide additional supplementary information along with their paper. Imagine, for example, if researchers doing animal work were required to film themselves doing it and post the videos online for others to review. I think it’s easy to see how that would be a helpful “firehose of information” and would do a lot to flesh out the picture given by the normally reported figures in a publication.
I think you’re worried about people switching from hard analysis to squishier qualitative data, perhaps because resources are already so constrained that it feels like “one or the other.” I think John’s saying “why not both?”