Legibility, transparency, and open science are generally considered positive attributes, while opacity, elitism, and obscurantism are viewed as negative. However, increased legibility in science is not always beneficial and can often be detrimental.
Scientific management, with some exceptions, likely underperforms compared to simpler heuristics such as giving money to smart people or implementing grant lotteries. Scientific legibility suffers from the classic “Seeing like a State” problems. It constrains endeavors to the least informed stakeholder, hinders exploration, inevitably biases research to be simple and myopic, and exposes researchers to constant political tug-of-war between different interest groups poisoning objectivity.
I think the above would be considered relatively uncontroversial in EA circles. But I posit there is something deeper going on:
Novel research is inherently illegible. If it were legible, someone else would have already pursued it. As science advances her concepts become increasingly counterintuitive and further from common sense. Most of the legible low-hanging fruit has already been picked, and novel research requires venturing higher into the tree, pursuing illegible paths with indirect and hard-to-foresee impacts.
I’m pretty skeptical of this and think we need data to back up such a claim. However there might be bias: when anyone makes a serendipitous discovery it’s a better story, so it gets more attention. Has anyone gone through, say, the list of all Nobel laureates and looked at whether their research would have seemed promising before it produced results?
Thanks for your skepticism, Thomas. Before we get into this, I’d like to make sure actually disagree.
My position is not that scientific progress is mostly due to plucky outsiders who are ignored for decades. (I feel something like this is a popular view on LW). Indeed, I think most scientific progress is made through pretty conventional (academic) routes.
I think one can predict that future scientific progress will likely be made by young smart people at prestigious universities and research labs specializing in fields that have good feedback loops and/or have historically made a lot of progress: physics, chemistry, medicine, etc
My contention is that beyond very broad predictive factors like this, judging whether a research direction is fruitful is hard & requires inside knowledge. Much of this knowledge is illegible, difficult to attain because it takes a lot of specialized knowledge etc.
Do you disagree with this ?
I do think that novel research is inherently illegible.
Here are some thoughts on your comment :
1.Before getting into your Nobel prize proposal I’d like to caution for Hindsight bias (obvious reasons).
And perhaps to some degree I’d like to argue the burden of proof should be on the converse: show me evidence that scientific progress is very legible.
In some sense, predicting what directions will be fruitful is a bet against the (efficiënt ?) scientific market.
I also agree the amount of prediction one can do will vary a lot. Indeed, it was itself an innovation (eg Thomas Edison and his lightbulbs !) that some kind of scientific and engineering progress could by systematized: the discovery of R&D.
I think this works much better for certain domains than for others and a to large degree the ‘harder’ & more ‘novel’ the problem is the more labs defer ‘illegibly’ to the inside knowledge of researchers.
I guess I’m not sure what you mean by “most scientific progress,” and I’m missing some of the history here, but my sense is that importance-weighted science happens proportionally more outside of academia. E.g., Einstein did his miracle year outside of academia (and later stated that he wouldn’t have been able to do it, had he succeeded at getting an academic position), Darwin figured out natural selection, and Carnot figured out the Carnot cycle, all mostly on their own, outside of academia. Those are three major scientists who arguably started entire fields (quantum mechanics, biology, and thermodynamics). I would anti-predict that future scientific progress, of the field-founding sort, comes primarily from people at prestigious universities, since they, imo, typically have some of the most intense gatekeeping dynamics which make it harder to have original thoughts.
I do wonder to what degree that may be biased by the fact that there were vastly less academic positions before WWI/WWII. In the time of Darwin and Carnot these positions virtually didn’t exist. In the time of Einstein they did exist but they were quite rare still.
How many examples do you know of this happening past WWII?
Shannon was at Bell Labs iirc
As counterexample of field-founding happening in academia: Godel, Church, Turing were all in academia.
Oh, I actually 70% agree with this. I think there’s an important distinction between legibility to laypeople vs legibility to other domain experts. Let me lay out my beliefs:
In the modern history of fields you mentioned, more than 70% of discoveries are made by people trying to discover the thing, rather than serendipitously.
Other experts in the field, if truth-seeking, are able to understand the theory of change behind the research direction without investing huge amounts of time.
In most fields, experts and superforecasters informed by expert commentary will have fairly strong beliefs about which approaches to a problem will succeed. The person working on something will usually have less than 1 bit advantage about whether their framework will be successful than the experts, unless they have private information (e.g. already did the crucial experiment). This is the weakest belief and I could probably be convinced otherwise just by anecdotes.
The successful researchers might be confident they will succeed, but unsuccessful ones could be almost as confident on average. So it’s not that the research is illegible, it’s just genuinely hard to predict who will succeed.
People often work on different approaches to the problem even if they can predict which ones will work. This could be due to irrationality, other incentives, diminishing returns to each approach, comparative advantage, etc.
If research were illegible to other domain experts, I think you would not really get Kuhnian paradigms, which I am pretty confident exist. Paradigm shifts mostly come from the track record of an approach, so maybe this doesn’t count as researchers having an inside view of others’ work though.
Thank you, Thomas. I believe we find ourselves in broad agreement.
The distinction you make between lay-legibility and expert-legibility is especially well-drawn.
One point: the confidence of researchers in their own approach may not be the right thing to look at. Perhaps a better measure is seeing who can predict not only their own approach will succed but explain in detail why other approaches won’t work. Anecdotally, very succesful researchers have a keen sense of what will work out and what won’t—in private conversation many are willing to share detailed models why other approaches will not work or are not as promising. I’d have to think about this more carefully but anecdotally the most succesful researchers have many bits of information over their competitors not just one or two.
(Note that one bit of information means that their entire advantage could be wiped out by answering a single Y/N question. Not impossible, but not typical for most cases)
What areas of science are you thinking of? I think the discussion varies dramatically.
I think allowing less legibility would help make science less plodding, and allow it to move in larger steps. But there’s also a question of what direction it’s plodding. The problem I saw with psych and neurosci was that it tended to plod in nearly random, not very useful directions.
And what definition of “smart”? I’m afraid that by a common definition, smart people tend to do dumb research, in that they’ll do galaxy brained projects that are interesting but unlikely to pay off. This is how you get new science, but not useful science.
In cognitive psychology and neuroscience, I want to see money given to people who are both creative and practical. They will do new science that is also useful.
In psychology and neuroscience, scientists pick the grantees, and they tend to give money to those whose research they understand. This produces an effect where research keeps following one direction that became popular long ago. I think a different method of granting would work better, but the particular method matters a lot.
Thinking about it a little more, having a mix of personality types involved would probably be useful. I always appreciated the contributions of the rare philospher who actually learned enough to join a discussion about psych or neurosci research.
I think the most important application of meta science theory is alignment research.
Novel research is inherently illegible. If it were legible, someone else would have already pursued it.
It might also be that a legible path would be low status to pursue in the existing scientific communities and thus nobody pursues it.
If you look at a low-hanging fruit that was unpicked for a long time, airborne transmission of many viruses like the common cold, is a good example. There’s nothing illegible about it.
The core reason for holding the belief is because the world does not look to me like there’s little low hanging fruit in a variety of domains of knowledge I have thought about over the years. Of course it’s generally not that easy to argue for the value of ideas that the mainstream does not care about publically.
I find it curious that none of my ideas have a following in academia or have been reinvented/rediscovered by academia (including the most influential ones so far UDT, UDASSA, b-money). Not really complaining, as they’re already more popular than I had expected (Holden Karnofsky talked extensively about UDASSA on an 80,000 Hour podcast, which surprised me), it just seems strange that the popularity stops right at academia’s door.
If you look at the broader field of rationality, the work of Judea Pearl and that of Tetlock both could have been done twenty years earlier. Conceptually, I think you can argue that their work was some of the most important work that was done in the last decades.
Judea Pearl writes about how allergic people were against the idea of factoring in counterfactuals and causality.
Novel Science is Inherently Illegible
Legibility, transparency, and open science are generally considered positive attributes, while opacity, elitism, and obscurantism are viewed as negative. However, increased legibility in science is not always beneficial and can often be detrimental.
Scientific management, with some exceptions, likely underperforms compared to simpler heuristics such as giving money to smart people or implementing grant lotteries. Scientific legibility suffers from the classic “Seeing like a State” problems. It constrains endeavors to the least informed stakeholder, hinders exploration, inevitably biases research to be simple and myopic, and exposes researchers to constant political tug-of-war between different interest groups poisoning objectivity.
I think the above would be considered relatively uncontroversial in EA circles. But I posit there is something deeper going on:
Novel research is inherently illegible. If it were legible, someone else would have already pursued it. As science advances her concepts become increasingly counterintuitive and further from common sense. Most of the legible low-hanging fruit has already been picked, and novel research requires venturing higher into the tree, pursuing illegible paths with indirect and hard-to-foresee impacts.
I’m pretty skeptical of this and think we need data to back up such a claim. However there might be bias: when anyone makes a serendipitous discovery it’s a better story, so it gets more attention. Has anyone gone through, say, the list of all Nobel laureates and looked at whether their research would have seemed promising before it produced results?
Thanks for your skepticism, Thomas. Before we get into this, I’d like to make sure actually disagree. My position is not that scientific progress is mostly due to plucky outsiders who are ignored for decades. (I feel something like this is a popular view on LW). Indeed, I think most scientific progress is made through pretty conventional (academic) routes.
I think one can predict that future scientific progress will likely be made by young smart people at prestigious universities and research labs specializing in fields that have good feedback loops and/or have historically made a lot of progress: physics, chemistry, medicine, etc
My contention is that beyond very broad predictive factors like this, judging whether a research direction is fruitful is hard & requires inside knowledge. Much of this knowledge is illegible, difficult to attain because it takes a lot of specialized knowledge etc.
Do you disagree with this ?
I do think that novel research is inherently illegible. Here are some thoughts on your comment :
1.Before getting into your Nobel prize proposal I’d like to caution for Hindsight bias (obvious reasons).
And perhaps to some degree I’d like to argue the burden of proof should be on the converse: show me evidence that scientific progress is very legible. In some sense, predicting what directions will be fruitful is a bet against the (efficiënt ?) scientific market.
I also agree the amount of prediction one can do will vary a lot. Indeed, it was itself an innovation (eg Thomas Edison and his lightbulbs !) that some kind of scientific and engineering progress could by systematized: the discovery of R&D.
I think this works much better for certain domains than for others and a to large degree the ‘harder’ & more ‘novel’ the problem is the more labs defer ‘illegibly’ to the inside knowledge of researchers.
I guess I’m not sure what you mean by “most scientific progress,” and I’m missing some of the history here, but my sense is that importance-weighted science happens proportionally more outside of academia. E.g., Einstein did his miracle year outside of academia (and later stated that he wouldn’t have been able to do it, had he succeeded at getting an academic position), Darwin figured out natural selection, and Carnot figured out the Carnot cycle, all mostly on their own, outside of academia. Those are three major scientists who arguably started entire fields (quantum mechanics, biology, and thermodynamics). I would anti-predict that future scientific progress, of the field-founding sort, comes primarily from people at prestigious universities, since they, imo, typically have some of the most intense gatekeeping dynamics which make it harder to have original thoughts.
Good point.
I do wonder to what degree that may be biased by the fact that there were vastly less academic positions before WWI/WWII. In the time of Darwin and Carnot these positions virtually didn’t exist. In the time of Einstein they did exist but they were quite rare still.
How many examples do you know of this happening past WWII?
Shannon was at Bell Labs iirc
As counterexample of field-founding happening in academia: Godel, Church, Turing were all in academia.
Oh, I actually 70% agree with this. I think there’s an important distinction between legibility to laypeople vs legibility to other domain experts. Let me lay out my beliefs:
In the modern history of fields you mentioned, more than 70% of discoveries are made by people trying to discover the thing, rather than serendipitously.
Other experts in the field, if truth-seeking, are able to understand the theory of change behind the research direction without investing huge amounts of time.
In most fields, experts and superforecasters informed by expert commentary will have fairly strong beliefs about which approaches to a problem will succeed. The person working on something will usually have less than 1 bit advantage about whether their framework will be successful than the experts, unless they have private information (e.g. already did the crucial experiment). This is the weakest belief and I could probably be convinced otherwise just by anecdotes.
The successful researchers might be confident they will succeed, but unsuccessful ones could be almost as confident on average. So it’s not that the research is illegible, it’s just genuinely hard to predict who will succeed.
People often work on different approaches to the problem even if they can predict which ones will work. This could be due to irrationality, other incentives, diminishing returns to each approach, comparative advantage, etc.
If research were illegible to other domain experts, I think you would not really get Kuhnian paradigms, which I am pretty confident exist. Paradigm shifts mostly come from the track record of an approach, so maybe this doesn’t count as researchers having an inside view of others’ work though.
Thank you, Thomas. I believe we find ourselves in broad agreement. The distinction you make between lay-legibility and expert-legibility is especially well-drawn.
One point: the confidence of researchers in their own approach may not be the right thing to look at. Perhaps a better measure is seeing who can predict not only their own approach will succed but explain in detail why other approaches won’t work. Anecdotally, very succesful researchers have a keen sense of what will work out and what won’t—in private conversation many are willing to share detailed models why other approaches will not work or are not as promising. I’d have to think about this more carefully but anecdotally the most succesful researchers have many bits of information over their competitors not just one or two. (Note that one bit of information means that their entire advantage could be wiped out by answering a single Y/N question. Not impossible, but not typical for most cases)
What areas of science are you thinking of? I think the discussion varies dramatically.
I think allowing less legibility would help make science less plodding, and allow it to move in larger steps. But there’s also a question of what direction it’s plodding. The problem I saw with psych and neurosci was that it tended to plod in nearly random, not very useful directions.
And what definition of “smart”? I’m afraid that by a common definition, smart people tend to do dumb research, in that they’ll do galaxy brained projects that are interesting but unlikely to pay off. This is how you get new science, but not useful science.
In cognitive psychology and neuroscience, I want to see money given to people who are both creative and practical. They will do new science that is also useful.
In psychology and neuroscience, scientists pick the grantees, and they tend to give money to those whose research they understand. This produces an effect where research keeps following one direction that became popular long ago. I think a different method of granting would work better, but the particular method matters a lot.
Thinking about it a little more, having a mix of personality types involved would probably be useful. I always appreciated the contributions of the rare philospher who actually learned enough to join a discussion about psych or neurosci research.
I think the most important application of meta science theory is alignment research.
It might also be that a legible path would be low status to pursue in the existing scientific communities and thus nobody pursues it.
If you look at a low-hanging fruit that was unpicked for a long time, airborne transmission of many viruses like the common cold, is a good example. There’s nothing illegible about it.
mmm Good point. Do you have more examples?
The core reason for holding the belief is because the world does not look to me like there’s little low hanging fruit in a variety of domains of knowledge I have thought about over the years. Of course it’s generally not that easy to argue for the value of ideas that the mainstream does not care about publically.
Wei Dei recently wrote:
If you look at the broader field of rationality, the work of Judea Pearl and that of Tetlock both could have been done twenty years earlier. Conceptually, I think you can argue that their work was some of the most important work that was done in the last decades.
Judea Pearl writes about how allergic people were against the idea of factoring in counterfactuals and causality.
I don’t think the application to EA itself would be uncontroversial.