[ETA: In light of pushback from Rob: I really don’t want this to become a self-fulfilling prophecy. My hope in making this post was to make the prediction less likely to come true, not more! I’m glad that MIRI & Eliezer are publicly engaging with the rest of the community more again, I want that to continue, and I want to do my part to help everybody to understand each other.]
And I know, before anyone bothers to say, that all of this reply is not written in the calm way that is right and proper for such arguments. I am tired. I have lost a lot of hope. There are not obvious things I can do, let alone arguments I can make, which I expect to be actually useful in the sense that the world will not end once I do them. I don’t have the energy left for calm arguments. What’s left is despair that can be given voice.
I grimly predict that the effect of this dialogue on the community will be polarization: People who didn’t like Yudkowsky and/or his views will like him / his views less, and the gap between them and Yud-fans will grow (more than it shrinks due to the effect of increased dialogue). I say this because IMO Yudkowsky comes across as angry and uncharitable in various parts of this dialogue, and also I think it was kinda a slog to get through & it doesn’t seem like much intellectual progress was made here.
FWIW I continue to think that Yudkowskys model of how the future will go is basically right, at least more right than Christiano’s. This is a big source of sadness and stress for me too, because (for example) my beloved daughter probably won’t live to adulthood.
The best part IMO was the mini-essay at the end about Thielian secrets and different kinds of tech progress—a progression of scenarios adding up to Yudkowsky’s understanding of Paul’s model:
But we can imagine that doesn’t happen either, because instead of needing to build a whole huge manufacturing plant, there’s just lots and lots of little innovations adding up to every key AGI threshold, which lots of actors are investing $10 million in at a time, and everybody knows which direction to move in to get to more serious AGI and they’re right in this shared forecast.
It does seem to me that the AI industry will move more in this direction than it currently is, over the next decade or so. However I still do expect that we won’t get all the way there. I would love to hear from Paul whether he endorses the view Yudkowsky attributes to him in this final essay.
I grimly predict that the effect of this dialogue on the community will be polarization
Beware of self-fulfilling prophecies (and other premature meta)! If both sides in a dispute expect the other side to just entrench, then they’re less likely to invest the effort to try to bridge the gap.
This very comment section is one of the main things that will determine the community’s reaction, and diverting our focus to ‘what will our reaction be?’ before we’ve talked about the object-level claims can prematurely lock in a certain reaction.
(That said, I think you’re doing a useful anti-polarization thing here, by showing empathy for people you disagree with, and showing willingness to criticize people you agree with. I don’t at all dislike this comment overall; I just want to caution against giving up on something before we’ve really tried. This is the first proper MIRI-response to Paul’s takeoff post, and should be a pretty big update for a lot of people—I don’t think people were even universally aware that Eliezer endorses hard takeoff anymore, much less aware of his reasoning.)
Fair enough! I too dislike premature meta, and feel bad that I engaged in it. However… I do still feel like my comment probably did more to prevent polarization than cause it? That’s my independent impression at any rate. (For the reasons you mention).
I certainly don’t want to give up! In light of your pushback I’ll edit to add something at the top.
While this may not be the ideal format for it, I thought Eliezer’s voicing of despair was a useful update to publish to the LW community about the current state of his AI beliefs.
I grimly predict that the effect of this dialogue on the community will be polarization: People who didn’t like Yudkowsky and/or his views will like him / his views less, and the gap between them and Yud-fans will grow (more than it shrinks due to the effect of increased dialogue). I say this because IMO Yudkowsky comes across as angry and uncharitable in various parts of this dialogue, and also I think it was kinda a slog to get through & it doesn’t seem like much intellectual progress was made here.
Strongly agree with that.
Since you agree with Yudkowksy, do you think you could strongman his position?
Yes, though I’m much more comfortable explaining and arguing for my own position than EY’s. It’s just that my position turns out to be pretty similar. (Partly this is independent convergence, but of course partly this is causal influence since I’ve read a lot of his stuff.)
There’s a lot to talk about, I’m not sure where to begin, and also a proper response would be a whole research project in itself. Fortunately I’ve already written a bunch of it; see these twosequences.
Here are some quick high-level thoughts:
1. Begin with timelines. The best way to forecast timelines IMO is Ajeya’s model; it should be the starting point and everything else should be adjustments from it. The core part of Ajeya’s model is a probability distribution over how many OOMs of compute we’d need with today’s ideas to get to TAI / AGI / APS-AI / AI-PONR / etc. [Unfamiliar with these acronyms? See Robbo’s helpful comment below] For reasons which I’ve explained in my sequence (and summarized in a gdoc) my distribution has significantly more mass on the 0-6 OOM range than Paul does, and less on the 13+ range. The single post that conveys this intuition most is Fun with +12 OOMs.
Now consider how takeoff speed views interact with timelines views. Paul-slow takeoff and <10 year timelines are in tension with each other. If <7 OOMs of compute would be enough to get something crazy powerful with today’s ideas, then the AI industry is not an efficient market right now. If we get human-level AGI in 2030, then on Paul’s view that means the world economy should be doubling in 2029 and should have doubled over the course of 2025 − 2028 and should already be accelerating now probably. It doesn’t look like that’s happening or about to happen. I think Paul agrees with this; in various conversations he’s said things like “If AGI happens in 10 years or less then probably we get fast takeoff.” [Paul please correct me if I’m mischaracterizing your view!]
Ajeya (and Paul) mostly update against <10 year timelines for this reason. I, by contrast, mostly update against slow takeoff. (Obviously with both do a bit of both, like good Bayesians.)
2. I feel like the debate between EY and Paul (and the broader debate about fast vs. slow takeoff) has been frustratingly much reference class tennis and frustratingly little gears-level modelling. This includes my own writing on the subject—lots of historical analogies and whatnot. I’ve tentatively attempted some things sorta like gears-level modelling (arguably What 2026 Looks Like is an example of this) and so far it seems to be pushing my intuitions more towards “Yep, fast takeoff is more likely.” But I feel like my thinking on this is super inadequate and I think we all should be doing better. Shame! Shame on all of us!
3. I think the focus on GDP (especially GWP) is really off, for reasons mentioned here. I think AI-PONR will probably come before GWP accelerates, and at any rate what we care about for timelines and takeoff speeds is AI-PONR and so our arguments should be about e.g. whether there will be warning shots and powerful AI tools of the sort that are relevant to solving alignment for APS-AI systems.
I feel like the debate between EY and Paul (and the broader debate about fast vs. slow takeoff) has been frustratingly much reference class tennis and frustratingly little gears-level modelling.
So, there’s this inherent problem with deep gearsy models, where you have to convey a bunch of upstream gears (and the evidence supporting them) before talking about the downstream questions of interest, because if you work backwards then peoples’ brains run out of stack space and they lose track of the whole multi-step path. But if you just go explaining upstream gears first, then people won’t immediately see how they’re relevant to alignment or timelines or whatever, and then lots of people just wander off. Then you go try to explain something about alignment or timelines or whatever, using an argument which relies on those upstream gears, and it goes right over a bunch of peoples’ heads because they don’t have that upstream gear in their world-models.
For the sort of argument in this post, it’s even worse, because a lot of people aren’t even explicitly aware that the relevant type of gear is a thing, or how to think about it beyond a rough intuitive level.
I first ran into this problem in the context of takeoff arguments a couple years ago, and wrote up this sequence mainly to convey the relevant kinds of gears and how to think about them. I claim that this (i.e. constraint slackness/tautness) is usually a good model for gear-type in arguments about reference-classes in practice: typically an intuitively-natural reference class is a set of cases which share some common constraint, and the examples in the reference class then provide evidence for the tautness/slackness of the constraint. For instance, in this post, Paul often points to market efficiency as a taut constraint, and Eliezer argues that constraint is not very taut (at least not in the way needed for the slow takeoff argument). Paul’s intuitive estimates of tautness are presumably driven by things like e.g. financial markets. On the other side, Eliezer wrote Inadequate Equilibria to talk about how taut market efficiency is in general, including gears “further up” and more examples.
So, there’s this inherent problem with deep gearsy models, where you have to convey a bunch of upstream gears (and the evidence supporting them) before talking about the downstream questions of interest, because if you work backwards then peoples’ brains run out of stack space and they lose track of the whole multi-step path. But if you just go explaining upstream gears first, then people won’t immediately see how they’re relevant to alignment or timelines or whatever, and then lots of people just wander off. Then you go try to explain something about alignment or timelines or whatever, using an argument which relies on those upstream gears, and it goes right over a bunch of peoples’ heads because they don’t have that upstream gear in their world-models.
The solution might be to start with a concise, low-detail summery (not even one that argues the case, just states it), then start explaining in full detail from the start, knowing that your readers now know which way you’re going.
Wait, I think I just invented the Abstract (not meant as a snide remark. I really did realize it after writing the above, and found it funny).
The core part of Ajeya’s model is a probability distribution over how many OOMs of compute we’d need with today’s ideas to get to TAI / AGI / APS-AI / AI-PONR / etc.
I didn’t know the last two acronyms despite reading a decent amount of this literature, so thought I’d leave this note for other readers. Listing all of them for completeness (readers will of course know the first two):
TAI: transformative AI
AGI: artificial general intelligence
APS-AI: Advanced, Planning, Strategically aware AI [1]
[ETA: In light of pushback from Rob: I really don’t want this to become a self-fulfilling prophecy. My hope in making this post was to make the prediction less likely to come true, not more! I’m glad that MIRI & Eliezer are publicly engaging with the rest of the community more again, I want that to continue, and I want to do my part to help everybody to understand each other.]
I grimly predict that the effect of this dialogue on the community will be polarization: People who didn’t like Yudkowsky and/or his views will like him / his views less, and the gap between them and Yud-fans will grow (more than it shrinks due to the effect of increased dialogue). I say this because IMO Yudkowsky comes across as angry and uncharitable in various parts of this dialogue, and also I think it was kinda a slog to get through & it doesn’t seem like much intellectual progress was made here.
FWIW I continue to think that Yudkowskys model of how the future will go is basically right, at least more right than Christiano’s. This is a big source of sadness and stress for me too, because (for example) my beloved daughter probably won’t live to adulthood.
The best part IMO was the mini-essay at the end about Thielian secrets and different kinds of tech progress—a progression of scenarios adding up to Yudkowsky’s understanding of Paul’s model:
It does seem to me that the AI industry will move more in this direction than it currently is, over the next decade or so. However I still do expect that we won’t get all the way there. I would love to hear from Paul whether he endorses the view Yudkowsky attributes to him in this final essay.
Beware of self-fulfilling prophecies (and other premature meta)! If both sides in a dispute expect the other side to just entrench, then they’re less likely to invest the effort to try to bridge the gap.
This very comment section is one of the main things that will determine the community’s reaction, and diverting our focus to ‘what will our reaction be?’ before we’ve talked about the object-level claims can prematurely lock in a certain reaction.
(That said, I think you’re doing a useful anti-polarization thing here, by showing empathy for people you disagree with, and showing willingness to criticize people you agree with. I don’t at all dislike this comment overall; I just want to caution against giving up on something before we’ve really tried. This is the first proper MIRI-response to Paul’s takeoff post, and should be a pretty big update for a lot of people—I don’t think people were even universally aware that Eliezer endorses hard takeoff anymore, much less aware of his reasoning.)
Fair enough! I too dislike premature meta, and feel bad that I engaged in it. However… I do still feel like my comment probably did more to prevent polarization than cause it? That’s my independent impression at any rate. (For the reasons you mention).
I certainly don’t want to give up! In light of your pushback I’ll edit to add something at the top.
While this may not be the ideal format for it, I thought Eliezer’s voicing of despair was a useful update to publish to the LW community about the current state of his AI beliefs.
Strongly agree with that.
Since you agree with Yudkowksy, do you think you could strongman his position?
Yes, though I’m much more comfortable explaining and arguing for my own position than EY’s. It’s just that my position turns out to be pretty similar. (Partly this is independent convergence, but of course partly this is causal influence since I’ve read a lot of his stuff.)
There’s a lot to talk about, I’m not sure where to begin, and also a proper response would be a whole research project in itself. Fortunately I’ve already written a bunch of it; see these two sequences.
Here are some quick high-level thoughts:
1. Begin with timelines. The best way to forecast timelines IMO is Ajeya’s model; it should be the starting point and everything else should be adjustments from it. The core part of Ajeya’s model is a probability distribution over how many OOMs of compute we’d need with today’s ideas to get to TAI / AGI / APS-AI / AI-PONR / etc. [Unfamiliar with these acronyms? See Robbo’s helpful comment below] For reasons which I’ve explained in my sequence (and summarized in a gdoc) my distribution has significantly more mass on the 0-6 OOM range than Paul does, and less on the 13+ range. The single post that conveys this intuition most is Fun with +12 OOMs.
Now consider how takeoff speed views interact with timelines views. Paul-slow takeoff and <10 year timelines are in tension with each other. If <7 OOMs of compute would be enough to get something crazy powerful with today’s ideas, then the AI industry is not an efficient market right now. If we get human-level AGI in 2030, then on Paul’s view that means the world economy should be doubling in 2029 and should have doubled over the course of 2025 − 2028 and should already be accelerating now probably. It doesn’t look like that’s happening or about to happen. I think Paul agrees with this; in various conversations he’s said things like “If AGI happens in 10 years or less then probably we get fast takeoff.” [Paul please correct me if I’m mischaracterizing your view!]
Ajeya (and Paul) mostly update against <10 year timelines for this reason. I, by contrast, mostly update against slow takeoff. (Obviously with both do a bit of both, like good Bayesians.)
2. I feel like the debate between EY and Paul (and the broader debate about fast vs. slow takeoff) has been frustratingly much reference class tennis and frustratingly little gears-level modelling. This includes my own writing on the subject—lots of historical analogies and whatnot. I’ve tentatively attempted some things sorta like gears-level modelling (arguably What 2026 Looks Like is an example of this) and so far it seems to be pushing my intuitions more towards “Yep, fast takeoff is more likely.” But I feel like my thinking on this is super inadequate and I think we all should be doing better. Shame! Shame on all of us!
3. I think the focus on GDP (especially GWP) is really off, for reasons mentioned here. I think AI-PONR will probably come before GWP accelerates, and at any rate what we care about for timelines and takeoff speeds is AI-PONR and so our arguments should be about e.g. whether there will be warning shots and powerful AI tools of the sort that are relevant to solving alignment for APS-AI systems.
(Got to go now)
So, there’s this inherent problem with deep gearsy models, where you have to convey a bunch of upstream gears (and the evidence supporting them) before talking about the downstream questions of interest, because if you work backwards then peoples’ brains run out of stack space and they lose track of the whole multi-step path. But if you just go explaining upstream gears first, then people won’t immediately see how they’re relevant to alignment or timelines or whatever, and then lots of people just wander off. Then you go try to explain something about alignment or timelines or whatever, using an argument which relies on those upstream gears, and it goes right over a bunch of peoples’ heads because they don’t have that upstream gear in their world-models.
For the sort of argument in this post, it’s even worse, because a lot of people aren’t even explicitly aware that the relevant type of gear is a thing, or how to think about it beyond a rough intuitive level.
I first ran into this problem in the context of takeoff arguments a couple years ago, and wrote up this sequence mainly to convey the relevant kinds of gears and how to think about them. I claim that this (i.e. constraint slackness/tautness) is usually a good model for gear-type in arguments about reference-classes in practice: typically an intuitively-natural reference class is a set of cases which share some common constraint, and the examples in the reference class then provide evidence for the tautness/slackness of the constraint. For instance, in this post, Paul often points to market efficiency as a taut constraint, and Eliezer argues that constraint is not very taut (at least not in the way needed for the slow takeoff argument). Paul’s intuitive estimates of tautness are presumably driven by things like e.g. financial markets. On the other side, Eliezer wrote Inadequate Equilibria to talk about how taut market efficiency is in general, including gears “further up” and more examples.
If you click through the link in the post to Intelligence Explosion Microeconomics, there’s a lot of this sort of reasoning in it.
The solution might be to start with a concise, low-detail summery (not even one that argues the case, just states it), then start explaining in full detail from the start, knowing that your readers now know which way you’re going.
Wait, I think I just invented the Abstract (not meant as a snide remark. I really did realize it after writing the above, and found it funny).
I didn’t know the last two acronyms despite reading a decent amount of this literature, so thought I’d leave this note for other readers. Listing all of them for completeness (readers will of course know the first two):
TAI: transformative AI
AGI: artificial general intelligence
APS-AI: Advanced, Planning, Strategically aware AI [1]
AI-PONR: AI point of no return [2]
[1] from Carlsmith, which Daniel does link to
[2] from Daniel, which he also linked
Sorry! I’ll go back and insert links + reference your comment