It doesn’t look like a subset of fields to me. More like a subset of steps in each field that need to be done by outsiders, while both preceding and following steps can be done by academia. That’s what I’m seeing with decision theory: academia worked out many prerequisites but got stuck, then LW got the chance to make a key step, but it didn’t feed back into academia so we’ve stagnated since then.
In both digital money and decision theory, academia was going in the wrong direction, in the sense that you did worse if you tried to build on top of the state of the art in academia instead of starting over from a past point. In digital money you essentially had to go back to just basic cryptographic primitives. (I can’t think of anything in the hundreds of academic papers on digital money that Bitcoin uses.) In decision theory the state of the art was CDT, and one of the justifications for CDT was that, to be rational, one ought to two-box in Newcomb’s problem and CDT does that while EDT doesn’t. I recall trying to come up with a version of UDT that would two-box in Newcomb’s problem and play D in PD against a copy because (until reading Eliezer’s posts on LW) I just took for granted without thinking about it that CDT must be an improvement over EDT and the academic consensus on Newcomb’s problem must be correct. I think I posted the resulting decision theory (which ended up being a lot more complex than the UDT I posted LW) to the “everything-list” and I can try to find it if you’d like.
We also don’t have evidence that following steps can be done by academia. In both Bitcoin and UDT the following steps are all being done outside of academia, as far as I can tell. I don’t think “we’ve stagnated” is right since the obvious next step after UDT is trying to solve logical uncertainty so we can tell what UDT actually recommends, and progress is being made there.
Even if we put this dispute aside, and just assume that key steps in each field need to be done by outsiders, that seems enough to conclude that we wouldn’t be better off now if Eliezer had gone into academia. If that had happened, there wouldn’t be anyone to catalyze the crucial “outsider step” and we’d still be stuck with CDT (and the overly complex and wrong version of UDT) today.
On second thought, my previous reply was a bit too hasty and combative. What you said does make sense; it’s perhaps more likely that academia is unable to take certain steps rather than that it can’t make any progress in entire fields. What I wrote about academia going in wrong directions is consistent with the missing outsider steps having been missing for a while, and academia not recognizing it.
If I steelman your argument from “Eliezer should have gone into academia” to “Eliezer should have made a greater effort to push his results back into academia” it becomes harder to refute, but still not obviously right. It seems that if academia is unable to take certain steps, it’s also unable to recognize that it’s missing those steps and also unable to recognize when outsiders have taken those steps. It’s not clear what resources (in terms of, e.g., skills and connections) an outsider would need to wield in order to force academia past those missing steps and to continue to make progress, and also unclear how far academia can go before it encounters another step that only outsiders can take.
Both Satoshi and Eliezer did make nontrivial efforts to let academia recognize their results. Satoshi by posting an academic-looking paper and Eliezer by working with someone to try to polish his results into acceptable academic form. Given that they failed, I’m tempted to think that it’s just not realistic to expect that someone exists with the necessary combination of research skills need to make crucial progress, personality or circumstances causing them to stay outside of academia, and resources needed to force academia to recognize their progress.
Do you know if Satoshi’s bitcoin whitepaper or Eliezer’s TDT writeup were ever submitted for peer review? If not, then it seems premature to say that academia has rejected them.
I don’t have that information, but being published in a peer-reviewed venue is not a prerequisite for academia to recognize a result. See this example. Getting a paper through peer review is very costly in terms of time and effort (especially at a highly reputable journal which might only accept less than 10% of submissions and that’s from academics who know what they’re supposed to do to maximize the chances of being accepted), and may not buy all that much in terms of additional attention from the people who might be able to build upon the work. I tried submitting a paper to an academic crypto conference (for a crypto primitive, not b-money) so I have some experience with this myself. Satoshi and Eliezer aren’t the only very smart people who haven’t tried very hard to publish in academia (i.e., hard enough to get published in a reputable journal). Just from people I know, there’s also Gary Drescher, Paul Christiano (for his approval-directed agent ideas), and other researchers at MIRI who seem to publish most of their results as technical reports. I guess that also includes you, who haven’t followed your own advice?
Obviously whole fields of academia going in wrong directions represents a huge societal waste, and it would be great to have a solution to fix that; I’m just not sure what the solution is. (Note that among other things, you have to get academics to admit at least implicitly that for decades they’ve been wasting time and other people’s money on wrong approaches.) I haven’t been too concerned about this for decision theory since I’m not sure that further progress in decision theory is really crucial for (or even contributes positively to) AI alignment, but I have been thinking about how to get more academics to switch their focus from ML and AI capability in general to AI alignment, especially to AI alignment ideas that we think are promising and neglected (like Paul’s ideas). So far my idea is to try to influence funders (Open Phil, FLI, other philanthropists) to direct their grants to those specific ideas.
I guess that also includes you, who haven’t followed your own advice?
Yeah. When I was working actively on MIRI math, I picked up the idea that getting stuff peer reviewed is nice but not necessary for progress. My opinion changed in the last couple years, when I was already mostly away. The strategy I’d suggest now is to try to join the academic conversation on their terms, as I did at the Cambridge conference. Ideally, getting publications in journals should be part of that.
I haven’t thought much about talking to funders, good to hear you’re pursuing that.
The strategy I’d suggest now is to try to join the academic conversation on their terms, as I did at the Cambridge conference.
MIRI seems to be doing more of this as well, but I’m not seeing any noticeable results so far. Judging by citations in Google Scholar, in the 2 years since that conference, it doesn’t look like any academics have picked up on the ideas presented there by you and MIRI people or made further progress?
One other thing that worries me is, unless we can precisely diagnose what is causing academia to be unable to take the “outsider steps”, it seems dangerous to make ourselves more like academia. What if that causes us to lose that ability ourselves?
I haven’t thought much about talking to funders, good to hear you’re pursuing that.
Well I’m doing what I can but I’m not sure I’m the best person for this job, given that I’m not very social/outgoing and my opportunities for travel are limited so it’s hard to meet those funders and build up relationships.
One other thing that worries me is, unless we can precisely diagnose what is causing academia to be unable to take the “outsider steps”, it seems dangerous to make ourselves more like academia. What if that causes us to lose that ability ourselves?
Seems that academic motivations can be “value”, e.g discovering something of utility or “momentum”, sort of like a beauty contest, more applicable in abstract areas where utility is not obvious. Possible third is immediate enjoyment which probably contributed to millennia of number theory before it became useful.
Doing novel non-incremental things for non-value (like valuing AI safety) reasons is likely to be difficult until enough acceptability is built up for momentum type motivations. (which also suggests trying to explicitly build up momentum as an intervention)
It doesn’t look like a subset of fields to me. More like a subset of steps in each field that need to be done by outsiders, while both preceding and following steps can be done by academia. That’s what I’m seeing with decision theory: academia worked out many prerequisites but got stuck, then LW got the chance to make a key step, but it didn’t feed back into academia so we’ve stagnated since then.
In both digital money and decision theory, academia was going in the wrong direction, in the sense that you did worse if you tried to build on top of the state of the art in academia instead of starting over from a past point. In digital money you essentially had to go back to just basic cryptographic primitives. (I can’t think of anything in the hundreds of academic papers on digital money that Bitcoin uses.) In decision theory the state of the art was CDT, and one of the justifications for CDT was that, to be rational, one ought to two-box in Newcomb’s problem and CDT does that while EDT doesn’t. I recall trying to come up with a version of UDT that would two-box in Newcomb’s problem and play D in PD against a copy because (until reading Eliezer’s posts on LW) I just took for granted without thinking about it that CDT must be an improvement over EDT and the academic consensus on Newcomb’s problem must be correct. I think I posted the resulting decision theory (which ended up being a lot more complex than the UDT I posted LW) to the “everything-list” and I can try to find it if you’d like.
We also don’t have evidence that following steps can be done by academia. In both Bitcoin and UDT the following steps are all being done outside of academia, as far as I can tell. I don’t think “we’ve stagnated” is right since the obvious next step after UDT is trying to solve logical uncertainty so we can tell what UDT actually recommends, and progress is being made there.
Even if we put this dispute aside, and just assume that key steps in each field need to be done by outsiders, that seems enough to conclude that we wouldn’t be better off now if Eliezer had gone into academia. If that had happened, there wouldn’t be anyone to catalyze the crucial “outsider step” and we’d still be stuck with CDT (and the overly complex and wrong version of UDT) today.
On second thought, my previous reply was a bit too hasty and combative. What you said does make sense; it’s perhaps more likely that academia is unable to take certain steps rather than that it can’t make any progress in entire fields. What I wrote about academia going in wrong directions is consistent with the missing outsider steps having been missing for a while, and academia not recognizing it.
If I steelman your argument from “Eliezer should have gone into academia” to “Eliezer should have made a greater effort to push his results back into academia” it becomes harder to refute, but still not obviously right. It seems that if academia is unable to take certain steps, it’s also unable to recognize that it’s missing those steps and also unable to recognize when outsiders have taken those steps. It’s not clear what resources (in terms of, e.g., skills and connections) an outsider would need to wield in order to force academia past those missing steps and to continue to make progress, and also unclear how far academia can go before it encounters another step that only outsiders can take.
Both Satoshi and Eliezer did make nontrivial efforts to let academia recognize their results. Satoshi by posting an academic-looking paper and Eliezer by working with someone to try to polish his results into acceptable academic form. Given that they failed, I’m tempted to think that it’s just not realistic to expect that someone exists with the necessary combination of research skills need to make crucial progress, personality or circumstances causing them to stay outside of academia, and resources needed to force academia to recognize their progress.
Do you know if Satoshi’s bitcoin whitepaper or Eliezer’s TDT writeup were ever submitted for peer review? If not, then it seems premature to say that academia has rejected them.
I don’t have that information, but being published in a peer-reviewed venue is not a prerequisite for academia to recognize a result. See this example. Getting a paper through peer review is very costly in terms of time and effort (especially at a highly reputable journal which might only accept less than 10% of submissions and that’s from academics who know what they’re supposed to do to maximize the chances of being accepted), and may not buy all that much in terms of additional attention from the people who might be able to build upon the work. I tried submitting a paper to an academic crypto conference (for a crypto primitive, not b-money) so I have some experience with this myself. Satoshi and Eliezer aren’t the only very smart people who haven’t tried very hard to publish in academia (i.e., hard enough to get published in a reputable journal). Just from people I know, there’s also Gary Drescher, Paul Christiano (for his approval-directed agent ideas), and other researchers at MIRI who seem to publish most of their results as technical reports. I guess that also includes you, who haven’t followed your own advice?
Obviously whole fields of academia going in wrong directions represents a huge societal waste, and it would be great to have a solution to fix that; I’m just not sure what the solution is. (Note that among other things, you have to get academics to admit at least implicitly that for decades they’ve been wasting time and other people’s money on wrong approaches.) I haven’t been too concerned about this for decision theory since I’m not sure that further progress in decision theory is really crucial for (or even contributes positively to) AI alignment, but I have been thinking about how to get more academics to switch their focus from ML and AI capability in general to AI alignment, especially to AI alignment ideas that we think are promising and neglected (like Paul’s ideas). So far my idea is to try to influence funders (Open Phil, FLI, other philanthropists) to direct their grants to those specific ideas.
Yeah. When I was working actively on MIRI math, I picked up the idea that getting stuff peer reviewed is nice but not necessary for progress. My opinion changed in the last couple years, when I was already mostly away. The strategy I’d suggest now is to try to join the academic conversation on their terms, as I did at the Cambridge conference. Ideally, getting publications in journals should be part of that.
I haven’t thought much about talking to funders, good to hear you’re pursuing that.
What triggered this?
MIRI seems to be doing more of this as well, but I’m not seeing any noticeable results so far. Judging by citations in Google Scholar, in the 2 years since that conference, it doesn’t look like any academics have picked up on the ideas presented there by you and MIRI people or made further progress?
One other thing that worries me is, unless we can precisely diagnose what is causing academia to be unable to take the “outsider steps”, it seems dangerous to make ourselves more like academia. What if that causes us to lose that ability ourselves?
Well I’m doing what I can but I’m not sure I’m the best person for this job, given that I’m not very social/outgoing and my opportunities for travel are limited so it’s hard to meet those funders and build up relationships.
Seems that academic motivations can be “value”, e.g discovering something of utility or “momentum”, sort of like a beauty contest, more applicable in abstract areas where utility is not obvious. Possible third is immediate enjoyment which probably contributed to millennia of number theory before it became useful.
Doing novel non-incremental things for non-value (like valuing AI safety) reasons is likely to be difficult until enough acceptability is built up for momentum type motivations. (which also suggests trying to explicitly build up momentum as an intervention)
Did you mean “likely to be difficult”?
thanks, fixed!