What are some of the most impressive things you do expect to see AI do, such that if you didn’t see them within 3 or 5 years, you’d majorly update about time to the type of AGI that might kill everyone?
Consider tasks that quite good software engineers (maybe top 40% at Jane Street) typically do in 8 hours without substantial prior context on that exact task. (As in, 8 hour median completion time.) Now, we’ll aim to sample these tasks such that the distribution and characteristics of these tasks are close to the distribution of work tasks in actual software engineering jobs (we probably can’t get that close because of the limited context constraint, but we’ll try).
In short timelines, I expect AIs will be able to succeed at these tasks 70% of the time within 3-5 years and if they didn’t, I would update toward longer timelines. (This is potentially using huge amounts of inference compute and using strategies that substantially differ from how humans do these tasks.)
The quantitative update would depend on how far AIs are from being able to accomplish this. If AIs were quite far (e.g., at 2 hours on this metric which is pretty close to where they are now) and the trend on horizon length indicated N years until 64 hours, I would update to something like 3 N as my median for AGI.
(I think a reasonable interpretation of the current trend indicates like 4 month doubling times. We’re currently at like a bit less than 1 hour for this metric I think, though maybe more like 30 min? Maybe you need to get to 64 hours until stuff feels pretty close to getting crazy. So, this suggests 2.3 year, though I expect longer in practice. My actual median for “AGI” in a strong sense is like 7 years, so 3x longer than this.)
Edit: Note that I’m not responding to “most impressive”, just trying to operationalize something that would make me update.
Thanks… but wait, this is among the most impressive things you expect to see? (You know more than I do about that distribution of tasks, so you could justifiably find it more impressive than I do.)
No, sorry I was mostly focused on “such that if you didn’t see them within 3 or 5 years, you’d majorly update about time to the type of AGI that might kill everyone”. I didn’t actually pick up on “most impressive” and actually tried to focus on something that occurs substantially before things get crazy.
Most impressive would probably be stuff like “automate all of AI R&D and greatly accelerate the pace of research at AI companies”. (This seems about 35% likely to me within 5 years, so I’d update by at least that much.) But this hardly seems that interesting? I think we can agree that once the AIs are automating whole companies stuff is very near.
Ok. So I take it you’re very impressed with the difficulty of the research that is going on in AI R&D.
we can agree that once the AIs are automating whole companies stuff
(FWIW I don’t agree with that; I don’t know what companies are up to, some of them might not be doing much difficult stuff and/or the managers might not be able to or care to tell the difference.)
I mean, I don’t think AI R&D is a particularly hard field persay, but I do think it involves lots of tricky stuff and isn’t much easier than automating some other plausibly-important-to-takeover field (e.g., robotics). (I could imagine that the AIs have a harder time automating philosophy even if they were trying to work on this, but it’s more confusing to reason about because human work on this is so dysfunctional.) The main reason I focused on AI R&D is that I think it is much more likely to be fully automated first and seems like it is probably fully automated prior to AI takeover.
Ok, I think I see what you’re saying. To check part of my understanding: when you say “AI R&D is fully automated”, I think you mean something like:
Most major AI companies have fired almost all of their SWEs. They still have staff to physically build datacenters, do business, etc.; and they have a few overseers / coordinators / strategizers of the fleet of AI R&D research gippities; but the overseers are acknowledged to basically not be doing much, and not clearly be even helping; and the overall output of the research group is “as good or better” than in 2025--measured… somehow.
I could imagine the capability occurring but not playing out that way, because the SWEs won’t necessarily be fired even after becoming useless—so it won’t be completely obvious from the outside. But this is a sociological point about when companies fire people, not a prediction about AI capabilities.
SWEs won’t necessarily be fired even after becoming useless
I’m actually surprised at how eager/willing big tech is to fire SWEs once they’re sure they won’t be economically valuable. I think a lot of priors for them being stable come from the ZIRP era. Now, these companies have quite frequent layoffs, silent layoffs, and performance firings. Companies becoming leaner will be a good litmus test for a lot of these claims.
What are some of the most impressive things you do expect to see AI do, such that if you didn’t see them within 3 or 5 years, you’d majorly update about time to the type of AGI that might kill everyone?
Consider tasks that quite good software engineers (maybe top 40% at Jane Street) typically do in 8 hours without substantial prior context on that exact task. (As in, 8 hour median completion time.) Now, we’ll aim to sample these tasks such that the distribution and characteristics of these tasks are close to the distribution of work tasks in actual software engineering jobs (we probably can’t get that close because of the limited context constraint, but we’ll try).
In short timelines, I expect AIs will be able to succeed at these tasks 70% of the time within 3-5 years and if they didn’t, I would update toward longer timelines. (This is potentially using huge amounts of inference compute and using strategies that substantially differ from how humans do these tasks.)
The quantitative update would depend on how far AIs are from being able to accomplish this. If AIs were quite far (e.g., at 2 hours on this metric which is pretty close to where they are now) and the trend on horizon length indicated N years until 64 hours, I would update to something like 3 N as my median for AGI.
(I think a reasonable interpretation of the current trend indicates like 4 month doubling times. We’re currently at like a bit less than 1 hour for this metric I think, though maybe more like 30 min? Maybe you need to get to 64 hours until stuff feels pretty close to getting crazy. So, this suggests 2.3 year, though I expect longer in practice. My actual median for “AGI” in a strong sense is like 7 years, so 3x longer than this.)
Edit: Note that I’m not responding to “most impressive”, just trying to operationalize something that would make me update.
Thanks… but wait, this is among the most impressive things you expect to see? (You know more than I do about that distribution of tasks, so you could justifiably find it more impressive than I do.)
No, sorry I was mostly focused on “such that if you didn’t see them within 3 or 5 years, you’d majorly update about time to the type of AGI that might kill everyone”. I didn’t actually pick up on “most impressive” and actually tried to focus on something that occurs substantially before things get crazy.
Most impressive would probably be stuff like “automate all of AI R&D and greatly accelerate the pace of research at AI companies”. (This seems about 35% likely to me within 5 years, so I’d update by at least that much.) But this hardly seems that interesting? I think we can agree that once the AIs are automating whole companies stuff is very near.
Ok. So I take it you’re very impressed with the difficulty of the research that is going on in AI R&D.
(FWIW I don’t agree with that; I don’t know what companies are up to, some of them might not be doing much difficult stuff and/or the managers might not be able to or care to tell the difference.)
I mean, I don’t think AI R&D is a particularly hard field persay, but I do think it involves lots of tricky stuff and isn’t much easier than automating some other plausibly-important-to-takeover field (e.g., robotics). (I could imagine that the AIs have a harder time automating philosophy even if they were trying to work on this, but it’s more confusing to reason about because human work on this is so dysfunctional.) The main reason I focused on AI R&D is that I think it is much more likely to be fully automated first and seems like it is probably fully automated prior to AI takeover.
Ok, I think I see what you’re saying. To check part of my understanding: when you say “AI R&D is fully automated”, I think you mean something like:
I could imagine the capability occurring but not playing out that way, because the SWEs won’t necessarily be fired even after becoming useless—so it won’t be completely obvious from the outside. But this is a sociological point about when companies fire people, not a prediction about AI capabilities.
I’m actually surprised at how eager/willing big tech is to fire SWEs once they’re sure they won’t be economically valuable. I think a lot of priors for them being stable come from the ZIRP era. Now, these companies have quite frequent layoffs, silent layoffs, and performance firings. Companies becoming leaner will be a good litmus test for a lot of these claims.