This going to point about 87 degrees off from the main point of the post, so I’m fine with discussing this elsewhere or in DMs or something, but I do wonder how cruxy this is:
More broadly, I think AI Alignment ideas/the EA community/the rationality community played a pretty substantial role in the founding of the three leading AGI labs (Deepmind, OpenAI, Anthropic), and man, I sure would feel better about a world where none of these would exist, though I also feel quite uncertain here. But it does sure feel like we had a quite large counterfactual effect on AI timelines.
I missed the first chunk of your conversation with Dylan at the lurkshop about this, but at the time, it sounded like you suspected “quite large” wasn’t 6-48 months, but maybe more than a decade. I could have gotten the wrong impression, but I remember being confused enough that I resolved to hunt you down later to ask (which I promptly forgot to do).
I gather that this isn’t the issue, but it does seem load bearing. A model that suggests alignment/EA/rationality influences sped up AGI by >10 years has some pretty heavy implications which are consistent with the other things you’ve mentioned. If my understanding is correct, if you thought the actual speedup was more like 6 months (or even negative somehow), it would take some of the bite out.
My attempt at paraphrasing your model goes something like: 1. Alignment efforts influence an early start on some important companies like Deepmind. 2. These companies tend to have more “vision” with regard to AGI; they aren’t just solving narrow basic tasks for incremental economic gain, but instead are very explicitly focused on the possibility of AGI. 3. The founding difference in vision legitimizes certain kinds of research and investment, like spending millions of dollars on training runs for general models that might seem counterfactually silly (“what are you doing, trying to chase some sci-fi nonsense? no one respectable would do that!”) 4. The early influences more aggressively push algorithmic and hardware advancements by bringing demand forward in time.
I’m not sure how to get a decade or more out of this, though. I’d agree this dynamic did burn some time until AGI (and even 6 months is not ideal in context), but it seems like the largest counterfactual difference is in start date, not end date. I could see our universe getting into AGI-minded research 10 years earlier than others, but I strongly suspect the “boring” kind of applications would have ended up triggering basically the same dynamic later.
Those other worlds would still have video games encouraging a Close Enough hardware architecture and other enormous industries driving demand for improved chip manufacturing. In alt-2023, they’d still have a consumer grade video card like the 4090, just probably with less tensor core stuff. As the level of performance per dollar rises on hardware, the number of people who might try pushing scale increases until someone inevitably manages to convince VCs or other influential targets that hey, maybe there’s something here.
(Before pivoting to safety research, “doing somewhat fringe AI stuff for funsies” was exactly the kind of thing I did. The counterfactual version of me would have almost certainly noticed, and I can’t imagine I would have been alone or even close to it.)
At that point, I think the main difference between the worlds is that alt-world gets an even more absurdly shocking speedrun of capabilities. The hardware overhang makes it easy to pick similar low hanging fruit, just more quickly.
I think the only way for alt-world to take as long from Speedrun Start as us is if a surprisingly large amount of our progress came from sequential conceptual insights. That really doesn’t match my impression of progress so far. Given the disjunctive architectural possibilities for reaching transformer-ish performance, it seems more like a mix of “try random stuff” and “architectures enabled by hardware.”
Even assuming we had a 10 year head start, I’d expect the counterfactual world to reach AGI less than 48 months after us. That gives enough time for 1-2 extra hardware generations on top of the sequential efforts of engineers building the systems involved. And I’m not sure we even have a 10 year head start.
I’d also expect the counterfactual world to be even less concerned with safety. We’re not doing well, but there are almost always ways to do much worse! In this frame, we acquired multiple leading labs who at least ostensibly consider the problem for the price of a few years. Even with my short timelines, it’s hard for me to say this was a bad deal with high confidence.
I missed the first chunk of your conversation with Dylan at the lurkshop about this, but at the time, it sounded like you suspected “quite large” wasn’t 6-48 months, but maybe more than a decade.
A decade in-expectation seems quite extreme.
To be clear, I don’t think AGI happening soon is particularly overdetermined, so I do think this is a thing that does actually differ quite a bit depending on details, but I do think it’s very unlikely that actions that people adjacent to rationality took that seriously sped up timelines by more than a decade. I would currently give that maybe 3% probability or something.
People think the speed-up by rationalists is only ~5 years? I thought people were thinking 10-40. I do not think I would trade the entire history of LessWrong, including the articulation of the alignment problem, for 5 years of timelines. I mean, maybe it’s the right call, but it hardly seems obvious.
When LessWrong was ~dead (before we worked on the revival) I had this strong sense that being able to even consider that OpenAI could be bad for the world, or the notion that the alignment problem wasn’t going to go okay by-default, was being edged out of the overton window, and I felt enough pressure that I could barely think about it with anyone else. I think without the people on LessWrong writing to each other about it, I wouldn’t really be able to think clearly about the situation, and people would have collectively made like ~100x less of a direct effort on things.
(To be clear, I think the absolute probability is still dire, this has not been sufficient to solve things).
And of course that’s just since the revival, the true impact counts the sequences, and much of the articulation of the problem at all.
As bad as things are now, I think we all could’ve been a lot less sane in very nearby worlds.
I mean, I don’t see the argument for more than that. Unless you have some argument for hardware progress stopping, my sense is that things would get cheap enough that someone is going to try the AI stuff that is happening today within a decade.
This going to point about 87 degrees off from the main point of the post, so I’m fine with discussing this elsewhere or in DMs or something, but I do wonder how cruxy this is:
I missed the first chunk of your conversation with Dylan at the lurkshop about this, but at the time, it sounded like you suspected “quite large” wasn’t 6-48 months, but maybe more than a decade. I could have gotten the wrong impression, but I remember being confused enough that I resolved to hunt you down later to ask (which I promptly forgot to do).
I gather that this isn’t the issue, but it does seem load bearing. A model that suggests alignment/EA/rationality influences sped up AGI by >10 years has some pretty heavy implications which are consistent with the other things you’ve mentioned. If my understanding is correct, if you thought the actual speedup was more like 6 months (or even negative somehow), it would take some of the bite out.
My attempt at paraphrasing your model goes something like:
1. Alignment efforts influence an early start on some important companies like Deepmind.
2. These companies tend to have more “vision” with regard to AGI; they aren’t just solving narrow basic tasks for incremental economic gain, but instead are very explicitly focused on the possibility of AGI.
3. The founding difference in vision legitimizes certain kinds of research and investment, like spending millions of dollars on training runs for general models that might seem counterfactually silly (“what are you doing, trying to chase some sci-fi nonsense? no one respectable would do that!”)
4. The early influences more aggressively push algorithmic and hardware advancements by bringing demand forward in time.
I’m not sure how to get a decade or more out of this, though. I’d agree this dynamic did burn some time until AGI (and even 6 months is not ideal in context), but it seems like the largest counterfactual difference is in start date, not end date. I could see our universe getting into AGI-minded research 10 years earlier than others, but I strongly suspect the “boring” kind of applications would have ended up triggering basically the same dynamic later.
Those other worlds would still have video games encouraging a Close Enough hardware architecture and other enormous industries driving demand for improved chip manufacturing. In alt-2023, they’d still have a consumer grade video card like the 4090, just probably with less tensor core stuff. As the level of performance per dollar rises on hardware, the number of people who might try pushing scale increases until someone inevitably manages to convince VCs or other influential targets that hey, maybe there’s something here.
(Before pivoting to safety research, “doing somewhat fringe AI stuff for funsies” was exactly the kind of thing I did. The counterfactual version of me would have almost certainly noticed, and I can’t imagine I would have been alone or even close to it.)
At that point, I think the main difference between the worlds is that alt-world gets an even more absurdly shocking speedrun of capabilities. The hardware overhang makes it easy to pick similar low hanging fruit, just more quickly.
I think the only way for alt-world to take as long from Speedrun Start as us is if a surprisingly large amount of our progress came from sequential conceptual insights. That really doesn’t match my impression of progress so far. Given the disjunctive architectural possibilities for reaching transformer-ish performance, it seems more like a mix of “try random stuff” and “architectures enabled by hardware.”
Even assuming we had a 10 year head start, I’d expect the counterfactual world to reach AGI less than 48 months after us. That gives enough time for 1-2 extra hardware generations on top of the sequential efforts of engineers building the systems involved. And I’m not sure we even have a 10 year head start.
I’d also expect the counterfactual world to be even less concerned with safety. We’re not doing well, but there are almost always ways to do much worse! In this frame, we acquired multiple leading labs who at least ostensibly consider the problem for the price of a few years. Even with my short timelines, it’s hard for me to say this was a bad deal with high confidence.
FWIW Im very angry about what happened and I think the speedup was around five years in expectation.
A decade in-expectation seems quite extreme.
To be clear, I don’t think AGI happening soon is particularly overdetermined, so I do think this is a thing that does actually differ quite a bit depending on details, but I do think it’s very unlikely that actions that people adjacent to rationality took that seriously sped up timelines by more than a decade. I would currently give that maybe 3% probability or something.
People think the speed-up by rationalists is only ~5 years? I thought people were thinking 10-40. I do not think I would trade the entire history of LessWrong, including the articulation of the alignment problem, for 5 years of timelines. I mean, maybe it’s the right call, but it hardly seems obvious.
When LessWrong was ~dead (before we worked on the revival) I had this strong sense that being able to even consider that OpenAI could be bad for the world, or the notion that the alignment problem wasn’t going to go okay by-default, was being edged out of the overton window, and I felt enough pressure that I could barely think about it with anyone else. I think without the people on LessWrong writing to each other about it, I wouldn’t really be able to think clearly about the situation, and people would have collectively made like ~100x less of a direct effort on things.
(To be clear, I think the absolute probability is still dire, this has not been sufficient to solve things).
And of course that’s just since the revival, the true impact counts the sequences, and much of the articulation of the problem at all.
As bad as things are now, I think we all could’ve been a lot less sane in very nearby worlds.
I mean, I don’t see the argument for more than that. Unless you have some argument for hardware progress stopping, my sense is that things would get cheap enough that someone is going to try the AI stuff that is happening today within a decade.
some people who would have been working on ai without lesswrong: sutskever, graves, bengio, hinton, lecun, schmidhuber, hassabis,
“When LessWrong was ~dead”
Which year are you referring to here?
2016-17
Added: To give context, here’s a list of number of LW posts by year:
2009: 852
2010: 1143
2011: 3002
2012: 2583
2013: 1973
2014: 1797
2015: 2002 (<– This should be ~1880, as we added all ~120 HPMOR posts and backdated them to 2015)
2016: 1303 (<– This is the most ‘dead’ year according to me, and the year with the fewest posts)
2017: 1671 (<– LW 2.0 revived in the second half of this year)
2018: 1709
2019: 2121
2020: 3099
2021: 3230
2022: 4538
First quarter of 2023: 1436, if you 4x that it is 5744
(My, it’s getting to be quite a lot of posts these days.)
Makes sense! Less confused now.