I think he solved the college problem by other means (recruiting people who are credible to bridge to the academic community). Not sure this was a mistake, considering costs like “not writing HPMOR”. Yes I agree some of the attitudes have produced unnecessary social friction, (and maybe a lot of it) but its harder to claim the individual decision was obviously bad.
Granted. Do you think the lack of peer review is hurting via the papers not getting enough acceptance in the community that would add to the AI safety cause? Or is the quality of papers themselves lacking due to not aiming at the peer review standard?
Academia can be terribly inefficient though. There were hundreds of academic papers on electronic cash before Bitcoin and none of them went anywhere in terms of deployment and commercial success, nor did any of them propose something architecturally close to Bitcoin. Similarly there were dozens of academic papers on the Absent Minded Driver problem but all that work didn’t get very far. Since lots of very smart people go into academia, it seems that there’s something wrong with the incentives in academia that prevents people from making the kind of progress that we’ve seen on these topics outside of academia. If Eliezer went into academia he may have fallen prey to the same kinds of problems.
I think your argument would work better when flipped on its head. There’s something wrong with incentives outside academia that prevents people from coming up with the Absent Minded Driver problem, and many other things besides! Most of the prerequisites for both Bitcoin and UDT come from academia.
Very few people outside academia has resources to do research so if you look at the total sum of results academia is obviously going to look better. But if you compare results per unit of work or cost, it no longer looks so great. Consider that the few people who worked on digital cash outside of academia quickly converged to ideas resembling Bitcoin, while none of the many academics did even after spending many more years on the problem. Similarly with UDT. It seems to me that if Eliezer went into academia, there’s no reason to think he would be much more productive than the average academic decision theorist. Nor would he likely have created a forum for amateurs to discussion decision theory (since no actual academic decision theorist has), in which case I wouldn’t have found an outlet for to spread my own decision theory ideas.
And to clarify, in case this is why you’re disagreeing with me, obviously academia is very good at certain types of problems, for example those involving incremental technical progress (like making slightly stronger ciphers each year until they’re really efficient and practically unbreakable). So I’m only talking about a subset of fields that apparently includes digital money and decision theory.
It doesn’t look like a subset of fields to me. More like a subset of steps in each field that need to be done by outsiders, while both preceding and following steps can be done by academia. That’s what I’m seeing with decision theory: academia worked out many prerequisites but got stuck, then LW got the chance to make a key step, but it didn’t feed back into academia so we’ve stagnated since then.
In both digital money and decision theory, academia was going in the wrong direction, in the sense that you did worse if you tried to build on top of the state of the art in academia instead of starting over from a past point. In digital money you essentially had to go back to just basic cryptographic primitives. (I can’t think of anything in the hundreds of academic papers on digital money that Bitcoin uses.) In decision theory the state of the art was CDT, and one of the justifications for CDT was that, to be rational, one ought to two-box in Newcomb’s problem and CDT does that while EDT doesn’t. I recall trying to come up with a version of UDT that would two-box in Newcomb’s problem and play D in PD against a copy because (until reading Eliezer’s posts on LW) I just took for granted without thinking about it that CDT must be an improvement over EDT and the academic consensus on Newcomb’s problem must be correct. I think I posted the resulting decision theory (which ended up being a lot more complex than the UDT I posted LW) to the “everything-list” and I can try to find it if you’d like.
We also don’t have evidence that following steps can be done by academia. In both Bitcoin and UDT the following steps are all being done outside of academia, as far as I can tell. I don’t think “we’ve stagnated” is right since the obvious next step after UDT is trying to solve logical uncertainty so we can tell what UDT actually recommends, and progress is being made there.
Even if we put this dispute aside, and just assume that key steps in each field need to be done by outsiders, that seems enough to conclude that we wouldn’t be better off now if Eliezer had gone into academia. If that had happened, there wouldn’t be anyone to catalyze the crucial “outsider step” and we’d still be stuck with CDT (and the overly complex and wrong version of UDT) today.
On second thought, my previous reply was a bit too hasty and combative. What you said does make sense; it’s perhaps more likely that academia is unable to take certain steps rather than that it can’t make any progress in entire fields. What I wrote about academia going in wrong directions is consistent with the missing outsider steps having been missing for a while, and academia not recognizing it.
If I steelman your argument from “Eliezer should have gone into academia” to “Eliezer should have made a greater effort to push his results back into academia” it becomes harder to refute, but still not obviously right. It seems that if academia is unable to take certain steps, it’s also unable to recognize that it’s missing those steps and also unable to recognize when outsiders have taken those steps. It’s not clear what resources (in terms of, e.g., skills and connections) an outsider would need to wield in order to force academia past those missing steps and to continue to make progress, and also unclear how far academia can go before it encounters another step that only outsiders can take.
Both Satoshi and Eliezer did make nontrivial efforts to let academia recognize their results. Satoshi by posting an academic-looking paper and Eliezer by working with someone to try to polish his results into acceptable academic form. Given that they failed, I’m tempted to think that it’s just not realistic to expect that someone exists with the necessary combination of research skills need to make crucial progress, personality or circumstances causing them to stay outside of academia, and resources needed to force academia to recognize their progress.
Do you know if Satoshi’s bitcoin whitepaper or Eliezer’s TDT writeup were ever submitted for peer review? If not, then it seems premature to say that academia has rejected them.
I don’t have that information, but being published in a peer-reviewed venue is not a prerequisite for academia to recognize a result. See this example. Getting a paper through peer review is very costly in terms of time and effort (especially at a highly reputable journal which might only accept less than 10% of submissions and that’s from academics who know what they’re supposed to do to maximize the chances of being accepted), and may not buy all that much in terms of additional attention from the people who might be able to build upon the work. I tried submitting a paper to an academic crypto conference (for a crypto primitive, not b-money) so I have some experience with this myself. Satoshi and Eliezer aren’t the only very smart people who haven’t tried very hard to publish in academia (i.e., hard enough to get published in a reputable journal). Just from people I know, there’s also Gary Drescher, Paul Christiano (for his approval-directed agent ideas), and other researchers at MIRI who seem to publish most of their results as technical reports. I guess that also includes you, who haven’t followed your own advice?
Obviously whole fields of academia going in wrong directions represents a huge societal waste, and it would be great to have a solution to fix that; I’m just not sure what the solution is. (Note that among other things, you have to get academics to admit at least implicitly that for decades they’ve been wasting time and other people’s money on wrong approaches.) I haven’t been too concerned about this for decision theory since I’m not sure that further progress in decision theory is really crucial for (or even contributes positively to) AI alignment, but I have been thinking about how to get more academics to switch their focus from ML and AI capability in general to AI alignment, especially to AI alignment ideas that we think are promising and neglected (like Paul’s ideas). So far my idea is to try to influence funders (Open Phil, FLI, other philanthropists) to direct their grants to those specific ideas.
I guess that also includes you, who haven’t followed your own advice?
Yeah. When I was working actively on MIRI math, I picked up the idea that getting stuff peer reviewed is nice but not necessary for progress. My opinion changed in the last couple years, when I was already mostly away. The strategy I’d suggest now is to try to join the academic conversation on their terms, as I did at the Cambridge conference. Ideally, getting publications in journals should be part of that.
I haven’t thought much about talking to funders, good to hear you’re pursuing that.
The strategy I’d suggest now is to try to join the academic conversation on their terms, as I did at the Cambridge conference.
MIRI seems to be doing more of this as well, but I’m not seeing any noticeable results so far. Judging by citations in Google Scholar, in the 2 years since that conference, it doesn’t look like any academics have picked up on the ideas presented there by you and MIRI people or made further progress?
One other thing that worries me is, unless we can precisely diagnose what is causing academia to be unable to take the “outsider steps”, it seems dangerous to make ourselves more like academia. What if that causes us to lose that ability ourselves?
I haven’t thought much about talking to funders, good to hear you’re pursuing that.
Well I’m doing what I can but I’m not sure I’m the best person for this job, given that I’m not very social/outgoing and my opportunities for travel are limited so it’s hard to meet those funders and build up relationships.
One other thing that worries me is, unless we can precisely diagnose what is causing academia to be unable to take the “outsider steps”, it seems dangerous to make ourselves more like academia. What if that causes us to lose that ability ourselves?
Seems that academic motivations can be “value”, e.g discovering something of utility or “momentum”, sort of like a beauty contest, more applicable in abstract areas where utility is not obvious. Possible third is immediate enjoyment which probably contributed to millennia of number theory before it became useful.
Doing novel non-incremental things for non-value (like valuing AI safety) reasons is likely to be difficult until enough acceptability is built up for momentum type motivations. (which also suggests trying to explicitly build up momentum as an intervention)
You don’t need to hire an academic to get peer review. Just send it to a relevant journal, and because of the blind review system, they will review it. I did it myself and got many valuable comments on my articles. One was finally accepted and one was rejected.
I think it is trainable skillset, and EY also has some papers, so I think he is able to master it. But he decided to write texts which are more attention grabbing.
As far as I understand the papers with EY in the authors list are collaborations between EY and other people where a lot of ideas come from EY but the actual paper writing was done by someone else.
How about refusing to go to college, or refusing to get TDT peer reviewed, or this.
I think he solved the college problem by other means (recruiting people who are credible to bridge to the academic community). Not sure this was a mistake, considering costs like “not writing HPMOR”. Yes I agree some of the attitudes have produced unnecessary social friction, (and maybe a lot of it) but its harder to claim the individual decision was obviously bad.
MIRI publishes very few peer reviewed papers per researcher per year, so the mistake of betting against academia is still hurting it.
Granted. Do you think the lack of peer review is hurting via the papers not getting enough acceptance in the community that would add to the AI safety cause? Or is the quality of papers themselves lacking due to not aiming at the peer review standard?
I think everything would’ve been better (more work, more discussion, more credibility) if the whole thing had been in academia from the beginning.
Another unfortunate factor was Eliezer’s crazy emphasis on secrecy in the early days.
Academia can be terribly inefficient though. There were hundreds of academic papers on electronic cash before Bitcoin and none of them went anywhere in terms of deployment and commercial success, nor did any of them propose something architecturally close to Bitcoin. Similarly there were dozens of academic papers on the Absent Minded Driver problem but all that work didn’t get very far. Since lots of very smart people go into academia, it seems that there’s something wrong with the incentives in academia that prevents people from making the kind of progress that we’ve seen on these topics outside of academia. If Eliezer went into academia he may have fallen prey to the same kinds of problems.
I think your argument would work better when flipped on its head. There’s something wrong with incentives outside academia that prevents people from coming up with the Absent Minded Driver problem, and many other things besides! Most of the prerequisites for both Bitcoin and UDT come from academia.
Very few people outside academia has resources to do research so if you look at the total sum of results academia is obviously going to look better. But if you compare results per unit of work or cost, it no longer looks so great. Consider that the few people who worked on digital cash outside of academia quickly converged to ideas resembling Bitcoin, while none of the many academics did even after spending many more years on the problem. Similarly with UDT. It seems to me that if Eliezer went into academia, there’s no reason to think he would be much more productive than the average academic decision theorist. Nor would he likely have created a forum for amateurs to discussion decision theory (since no actual academic decision theorist has), in which case I wouldn’t have found an outlet for to spread my own decision theory ideas.
And to clarify, in case this is why you’re disagreeing with me, obviously academia is very good at certain types of problems, for example those involving incremental technical progress (like making slightly stronger ciphers each year until they’re really efficient and practically unbreakable). So I’m only talking about a subset of fields that apparently includes digital money and decision theory.
It doesn’t look like a subset of fields to me. More like a subset of steps in each field that need to be done by outsiders, while both preceding and following steps can be done by academia. That’s what I’m seeing with decision theory: academia worked out many prerequisites but got stuck, then LW got the chance to make a key step, but it didn’t feed back into academia so we’ve stagnated since then.
In both digital money and decision theory, academia was going in the wrong direction, in the sense that you did worse if you tried to build on top of the state of the art in academia instead of starting over from a past point. In digital money you essentially had to go back to just basic cryptographic primitives. (I can’t think of anything in the hundreds of academic papers on digital money that Bitcoin uses.) In decision theory the state of the art was CDT, and one of the justifications for CDT was that, to be rational, one ought to two-box in Newcomb’s problem and CDT does that while EDT doesn’t. I recall trying to come up with a version of UDT that would two-box in Newcomb’s problem and play D in PD against a copy because (until reading Eliezer’s posts on LW) I just took for granted without thinking about it that CDT must be an improvement over EDT and the academic consensus on Newcomb’s problem must be correct. I think I posted the resulting decision theory (which ended up being a lot more complex than the UDT I posted LW) to the “everything-list” and I can try to find it if you’d like.
We also don’t have evidence that following steps can be done by academia. In both Bitcoin and UDT the following steps are all being done outside of academia, as far as I can tell. I don’t think “we’ve stagnated” is right since the obvious next step after UDT is trying to solve logical uncertainty so we can tell what UDT actually recommends, and progress is being made there.
Even if we put this dispute aside, and just assume that key steps in each field need to be done by outsiders, that seems enough to conclude that we wouldn’t be better off now if Eliezer had gone into academia. If that had happened, there wouldn’t be anyone to catalyze the crucial “outsider step” and we’d still be stuck with CDT (and the overly complex and wrong version of UDT) today.
On second thought, my previous reply was a bit too hasty and combative. What you said does make sense; it’s perhaps more likely that academia is unable to take certain steps rather than that it can’t make any progress in entire fields. What I wrote about academia going in wrong directions is consistent with the missing outsider steps having been missing for a while, and academia not recognizing it.
If I steelman your argument from “Eliezer should have gone into academia” to “Eliezer should have made a greater effort to push his results back into academia” it becomes harder to refute, but still not obviously right. It seems that if academia is unable to take certain steps, it’s also unable to recognize that it’s missing those steps and also unable to recognize when outsiders have taken those steps. It’s not clear what resources (in terms of, e.g., skills and connections) an outsider would need to wield in order to force academia past those missing steps and to continue to make progress, and also unclear how far academia can go before it encounters another step that only outsiders can take.
Both Satoshi and Eliezer did make nontrivial efforts to let academia recognize their results. Satoshi by posting an academic-looking paper and Eliezer by working with someone to try to polish his results into acceptable academic form. Given that they failed, I’m tempted to think that it’s just not realistic to expect that someone exists with the necessary combination of research skills need to make crucial progress, personality or circumstances causing them to stay outside of academia, and resources needed to force academia to recognize their progress.
Do you know if Satoshi’s bitcoin whitepaper or Eliezer’s TDT writeup were ever submitted for peer review? If not, then it seems premature to say that academia has rejected them.
I don’t have that information, but being published in a peer-reviewed venue is not a prerequisite for academia to recognize a result. See this example. Getting a paper through peer review is very costly in terms of time and effort (especially at a highly reputable journal which might only accept less than 10% of submissions and that’s from academics who know what they’re supposed to do to maximize the chances of being accepted), and may not buy all that much in terms of additional attention from the people who might be able to build upon the work. I tried submitting a paper to an academic crypto conference (for a crypto primitive, not b-money) so I have some experience with this myself. Satoshi and Eliezer aren’t the only very smart people who haven’t tried very hard to publish in academia (i.e., hard enough to get published in a reputable journal). Just from people I know, there’s also Gary Drescher, Paul Christiano (for his approval-directed agent ideas), and other researchers at MIRI who seem to publish most of their results as technical reports. I guess that also includes you, who haven’t followed your own advice?
Obviously whole fields of academia going in wrong directions represents a huge societal waste, and it would be great to have a solution to fix that; I’m just not sure what the solution is. (Note that among other things, you have to get academics to admit at least implicitly that for decades they’ve been wasting time and other people’s money on wrong approaches.) I haven’t been too concerned about this for decision theory since I’m not sure that further progress in decision theory is really crucial for (or even contributes positively to) AI alignment, but I have been thinking about how to get more academics to switch their focus from ML and AI capability in general to AI alignment, especially to AI alignment ideas that we think are promising and neglected (like Paul’s ideas). So far my idea is to try to influence funders (Open Phil, FLI, other philanthropists) to direct their grants to those specific ideas.
Yeah. When I was working actively on MIRI math, I picked up the idea that getting stuff peer reviewed is nice but not necessary for progress. My opinion changed in the last couple years, when I was already mostly away. The strategy I’d suggest now is to try to join the academic conversation on their terms, as I did at the Cambridge conference. Ideally, getting publications in journals should be part of that.
I haven’t thought much about talking to funders, good to hear you’re pursuing that.
What triggered this?
MIRI seems to be doing more of this as well, but I’m not seeing any noticeable results so far. Judging by citations in Google Scholar, in the 2 years since that conference, it doesn’t look like any academics have picked up on the ideas presented there by you and MIRI people or made further progress?
One other thing that worries me is, unless we can precisely diagnose what is causing academia to be unable to take the “outsider steps”, it seems dangerous to make ourselves more like academia. What if that causes us to lose that ability ourselves?
Well I’m doing what I can but I’m not sure I’m the best person for this job, given that I’m not very social/outgoing and my opportunities for travel are limited so it’s hard to meet those funders and build up relationships.
Seems that academic motivations can be “value”, e.g discovering something of utility or “momentum”, sort of like a beauty contest, more applicable in abstract areas where utility is not obvious. Possible third is immediate enjoyment which probably contributed to millennia of number theory before it became useful.
Doing novel non-incremental things for non-value (like valuing AI safety) reasons is likely to be difficult until enough acceptability is built up for momentum type motivations. (which also suggests trying to explicitly build up momentum as an intervention)
Did you mean “likely to be difficult”?
thanks, fixed!
That assumes he had nothing to learn from college, and the only function it could have provided is signalling and social credibility.
Not necessarily, just that costs of going to college were outweighed by other things (not saying that was EYs reasoning)
MIRI did try to hire an academic to get TDT peer reviewed. It didn’t work out but I don’t think “refuse” is a good description.
You don’t need to hire an academic to get peer review. Just send it to a relevant journal, and because of the blind review system, they will review it. I did it myself and got many valuable comments on my articles. One was finally accepted and one was rejected.
Writing academic papers needs a certain skillset. That’s not a skillset that EY developed, so it makes sense to give that task to someone who has it.
I think it is trainable skillset, and EY also has some papers, so I think he is able to master it. But he decided to write texts which are more attention grabbing.
As far as I understand the papers with EY in the authors list are collaborations between EY and other people where a lot of ideas come from EY but the actual paper writing was done by someone else.