How much money do you think it cost to run the experiments to come up with the concept of the file drawer problem and the concept pre-registration of studies? I don’t think that’s knowledge that got created by running expensive experiments. It came from people engaging in theoretical thinking.
After some background about NHST on page 1, Sterling immediately begins tallying tests of significance in a years’ worth of 4 psychology journals, on page 2, and discovers that eg of 106 tests, 105 rejected the null hypothesis. On page 3, he discusses how this bias could come about.
So at least in this very early discussion of publication bias, it was driven by people engaged in empirical thinking.
After some background about NHST on page 1, Sterling immediately begins tallying tests of significance in a years’ worth of 4 psychology journals, on page 2, and discovers that eg of 106 tests, 105 rejected the null hypothesis. On page 3, he discusses how this bias could come about.
I think doing a literature review is engaging in using other people data. For the sake of this discussion JoshuaZ claimed that Einstein was doing theoretical work when he worked with other people’s data.
If I want to draw information from a literature review to gather insights I don’t need expensive equipment. JoshuaZ claimed that you need expensive equipement to gather new insights in biology. I claim that’s not true.
I claim that there enough published information that’s not well organised into theories that you can make major advances in biology without needing to buy any equipment.
As far as I understand you don’t run experiments on participants to see whether Dual ‘n’ back works. You simply gather Dual ‘n’ back data from other people and tried doing it yourself to know how it feel like.
That’s not expensive. You don’t need to write large grants to get a lot of money to do that kind of work.
You do need some money to pay your bills. Einstein made that money through being a patent clerk. I don’t know how you make your money to live. Of course you don’t have to tell and I respect if that’s private information.
For all I know you could be making money by being a patent clerk like Einstein.
A scientists who can’t work on his grant projects because he of the government shutdown could use his free time to do the kind of work that you are doing.
If you don’t like the label “theoretic” that’s fine. If you want to propose a different label that distinguish your approach from the making fancy expensive experiments approach I’m open to use another label.
I think in the last decades we had an explosion in the amount of data in biology. I think that organising that data into theories lags behind. I think it takes less effort to advance biology by organising into theories and to do a bit of phenomenology than to push for further for expensive equipment produced knowledge.
I claim that there enough published information that’s not well organised into theories that you can make major advances in biology without needing to buy any equipment.
This can be true but also suboptimal. I’m sure that given enough cleverness and effort, we could extract a lot of genetic causes out of existing SNP databases—but why bother when we can wait a decade and sequence everyone for $100 a head? People aren’t free, and equipment both complements and substitutes for them.
As far as I understand you don’t run experiments on participants to see whether Dual ‘n’ back works. You simply gather Dual ‘n’ back data from other people and tried doing it yourself to know how it feel like. That’s not expensive. You don’t need to write large grants to get a lot of money to do that kind of work.
I assume you’re referring to my DNB meta-analysis? Yes, it’s not gathering primary data—I did think about doing that early on, which is why I carefully compiled all anecdotes mentioning IQ tests in my FAQ, but I realized that between the sheer heterogeneity, lack of a control group, massive selection effects, etc, the data was completely worthless.
But I can only gather the studies into a meta-analysis because people are running these studies. And I need a lot of data to draw any kind of conclusion. If n-back studies had stopped in 2010, I’d be out of luck, because with the studies up to 2010, I can exclude zero as the net effect, but I can’t make a rigorous statement about the effect of passive vs active control groups. (In fact, it’s only with the last 3 or 4 studies that the confidence intervals for the two groups stopped overlapping.) And these studies are expensive. I’m corresponding with one study author to correct the payment covariate, and it seems that on average participants were paid $600 - and there were 40, so they blew $24,000 just on paying the subjects, never mind paying for the MRI machine, the grad students, the professor time, publication, etc. At this point, the total cost of the research must be well into the millions of dollars.
It’s true that it’s a little irritating that no one has published a meta-analysis on DNB and that it’s not that difficult for a random person like myself to do it, it requires little in the way of resources—but that doesn’t change the fact that I still needed these dozens of professionals to run all these very expensive experiments to provide grist for the mill.
To go way up to Einstein, he was drawing on a lot of expensive data like that which showed the Mercury anomaly, and then was verified by very expensive data (I shudder to think how much those expeditions must have cost in constant dollars). Without that data, he would just be another… string theorist. Not Einstein.
You do need some money to pay your bills. Einstein made that money through being a patent clerk. I don’t know how you make your money to live. Of course you don’t have to tell and I respect if that’s private information. For all I know you could be making money by being a patent clerk like Einstein.
Not by being a patent clerk, no. :)
A scientists who can’t work on his grant projects because he of the government shutdown could use his free time to do the kind of work that you are doing.
To a very limited extent. There has to be enough studies to productively review, and there has to be no existing reviews you’re duplicating. To give another example: suppose I had been furloughed and wanted to work on a creatine meta-analysis. I get as far as I got now—not that hard, maybe 10 hours of work—and I realize there’s only 3 studies. Now what? Well, what I am doing is waiting a few months for 2 scientists to reply, and then I’ll wait another 5 or 10 years for governments to fund more psychology studies which happen to use creatine. But in no way can I possibly “finish” this even given months of government-shutdown-time.
I think in the last decades we had an explosion in the amount of data in biology. I think that organising that data into theories lags behind. I think it takes less effort to advance biology by organising into theories and to do a bit of phenomenology than to push for further for expensive equipment produced knowledge.
I don’t think that’s a stupid or obviously incorrect claim, but I don’t think it’s right. Equipment is advancing fast (if not always as fast as my first example of genotyping/sequencing), so it’d be surprising to me if you could do more work by ignoring potential new data and reprocessing old work, and more generally, even though stuff like meta-analysis is accessible to anyone for free (case in point: myself), we don’t see anyone producing any impressive discoveries. Case in point: more than a few researchers already believed n-back might be an artifact of the control groups before I started my meta-analysis—my results are a welcome confirmation, not a novel discovery; or to use your vitamin D example, yes, it’s cool that we found an effect of vitamin D on sleep (I certainly believe it), but the counterfactual of “QS does not exist” is not “vitamin D’s effect on sleep goes unknown” but “Gominak discovers the effect on her patients and publishes a review paper in 2012 arguing that vitamin D affects sleep”.
The earliest citation in the Rosenthal paper that coined the term ‘file drawer’ is to a 1959 paper by one Theodore Sterling; I jailbroke this to “Publication Decisions and Their Possible Effects on Inferences Drawn from tests of Significance—or Vice Versa”.
After some background about NHST on page 1, Sterling immediately begins tallying tests of significance in a years’ worth of 4 psychology journals, on page 2, and discovers that eg of 106 tests, 105 rejected the null hypothesis. On page 3, he discusses how this bias could come about.
So at least in this very early discussion of publication bias, it was driven by people engaged in empirical thinking.
I think doing a literature review is engaging in using other people data. For the sake of this discussion JoshuaZ claimed that Einstein was doing theoretical work when he worked with other people’s data.
If I want to draw information from a literature review to gather insights I don’t need expensive equipment. JoshuaZ claimed that you need expensive equipement to gather new insights in biology. I claim that’s not true. I claim that there enough published information that’s not well organised into theories that you can make major advances in biology without needing to buy any equipment.
As far as I understand you don’t run experiments on participants to see whether Dual ‘n’ back works. You simply gather Dual ‘n’ back data from other people and tried doing it yourself to know how it feel like. That’s not expensive. You don’t need to write large grants to get a lot of money to do that kind of work.
You do need some money to pay your bills. Einstein made that money through being a patent clerk. I don’t know how you make your money to live. Of course you don’t have to tell and I respect if that’s private information.
For all I know you could be making money by being a patent clerk like Einstein.
A scientists who can’t work on his grant projects because he of the government shutdown could use his free time to do the kind of work that you are doing.
If you don’t like the label “theoretic” that’s fine. If you want to propose a different label that distinguish your approach from the making fancy expensive experiments approach I’m open to use another label.
I think in the last decades we had an explosion in the amount of data in biology. I think that organising that data into theories lags behind. I think it takes less effort to advance biology by organising into theories and to do a bit of phenomenology than to push for further for expensive equipment produced knowledge.
If I phrase it that way, would you agree?
This can be true but also suboptimal. I’m sure that given enough cleverness and effort, we could extract a lot of genetic causes out of existing SNP databases—but why bother when we can wait a decade and sequence everyone for $100 a head? People aren’t free, and equipment both complements and substitutes for them.
I assume you’re referring to my DNB meta-analysis? Yes, it’s not gathering primary data—I did think about doing that early on, which is why I carefully compiled all anecdotes mentioning IQ tests in my FAQ, but I realized that between the sheer heterogeneity, lack of a control group, massive selection effects, etc, the data was completely worthless.
But I can only gather the studies into a meta-analysis because people are running these studies. And I need a lot of data to draw any kind of conclusion. If n-back studies had stopped in 2010, I’d be out of luck, because with the studies up to 2010, I can exclude zero as the net effect, but I can’t make a rigorous statement about the effect of passive vs active control groups. (In fact, it’s only with the last 3 or 4 studies that the confidence intervals for the two groups stopped overlapping.) And these studies are expensive. I’m corresponding with one study author to correct the payment covariate, and it seems that on average participants were paid $600 - and there were 40, so they blew $24,000 just on paying the subjects, never mind paying for the MRI machine, the grad students, the professor time, publication, etc. At this point, the total cost of the research must be well into the millions of dollars.
It’s true that it’s a little irritating that no one has published a meta-analysis on DNB and that it’s not that difficult for a random person like myself to do it, it requires little in the way of resources—but that doesn’t change the fact that I still needed these dozens of professionals to run all these very expensive experiments to provide grist for the mill.
To go way up to Einstein, he was drawing on a lot of expensive data like that which showed the Mercury anomaly, and then was verified by very expensive data (I shudder to think how much those expeditions must have cost in constant dollars). Without that data, he would just be another… string theorist. Not Einstein.
Not by being a patent clerk, no. :)
To a very limited extent. There has to be enough studies to productively review, and there has to be no existing reviews you’re duplicating. To give another example: suppose I had been furloughed and wanted to work on a creatine meta-analysis. I get as far as I got now—not that hard, maybe 10 hours of work—and I realize there’s only 3 studies. Now what? Well, what I am doing is waiting a few months for 2 scientists to reply, and then I’ll wait another 5 or 10 years for governments to fund more psychology studies which happen to use creatine. But in no way can I possibly “finish” this even given months of government-shutdown-time.
I don’t think that’s a stupid or obviously incorrect claim, but I don’t think it’s right. Equipment is advancing fast (if not always as fast as my first example of genotyping/sequencing), so it’d be surprising to me if you could do more work by ignoring potential new data and reprocessing old work, and more generally, even though stuff like meta-analysis is accessible to anyone for free (case in point: myself), we don’t see anyone producing any impressive discoveries. Case in point: more than a few researchers already believed n-back might be an artifact of the control groups before I started my meta-analysis—my results are a welcome confirmation, not a novel discovery; or to use your vitamin D example, yes, it’s cool that we found an effect of vitamin D on sleep (I certainly believe it), but the counterfactual of “QS does not exist” is not “vitamin D’s effect on sleep goes unknown” but “Gominak discovers the effect on her patients and publishes a review paper in 2012 arguing that vitamin D affects sleep”.