I like the style of this post, thanks for writing it! Some thoughts:
model scaling stops working
Roughly what probability would you put on this? I see this as really unlikely (perhaps <5%) such that ‘scaling stops working’ isn’t part of my model over the next 1-2yrs.
I will be slightly surprised if by end of 2024 there are AI agents running around the internet that are meaningfully in control of their own existence, e.g., are renting their own cloud compute without a human being involved.
Only slightly surprised? IMO being able to autonomously rent cloud compute seems quite significant (technically and legally), and I’d be very surprised if something like this happened on a 1yr horizon. I’d be negatively surprised if the US government didn’t institute regulation on the operation of autonomous agents of this type by the end of 2024, basically due to their potential for misuse and their economic value. It may help to know how you’re operationalizing AIs that are ‘meaningfully aware of their own existence’.
I think it’s pretty unlikely that scaling literally stops working, maybe I’m 5-10% that we soon get to a point where there are only very small or negligible improvements to increasing compute. But I’m like 10-20% on some weaker version.
A weaker version could look like there are diminishing returns to performance from scaling compute (as is true), and this makes it very difficult for companies to continue scaling. One mechanism at play is that the marginal improvements from scaling may not be enough to produce the additional revenue needed to cover the scaling costs, this is especially true in a competitive market where it’s not clear scaling will put one ahead of their competitors.
In the context of the post, I think it’s quite unlikely that I see strong evidence in the next year indicating that scaling has stopped (if only because a year of no-progress is not sufficient evidence). I was more so trying to point to how there [sic] are contingencies which would make OpenAI’s adoption of an RSP less safety-critical. I stand by the statements that scaling no longer yielding returns would be such a contingency, but I agree that it’s pretty unlikely.
We are currently at ASL-2 in Anthropic’s RSP. Based on the categorization, ASL-3 is “low-level autonomous capabilities”. I think ASL-3 systems probably don’t meet the bar of “meaningfully in control of their own existence”, but they probably meet the thing I think is more likely:
I think it wouldn’t be crazy if there were AI agents doing stuff online by the end of 2024, e.g., running social media accounts, selling consulting services; I expect such agents would be largely human-facilitated like AutoGPT
I think it’s currently a good bet (>40%) that we will see ASL-3 systems in 2024.
I’m not sure how big of a jump if will be from that to “meaningfully in control of their own existence”. I would be surprised if it were a small jump, such that we saw AIs renting their own cloud compute in 2024, but this is quite plausible on my models.
I think the evidence indicates that this is a hard task, but not super hard. e.g., looking at ARC’s report on autonomous tasks, one model partially completes the task of setting up GPT-J via a cloud provider (with human help).
I’ll amend my position to just being “surprised” without the slightly, as I think this better captures my beliefs — thanks for the push to think about this more. Maybe I’m at 5-10%.
It may help to know how you’re operationalizing AIs that are ‘meaningfully aware of their own existence’.
I like the style of this post, thanks for writing it! Some thoughts:
Roughly what probability would you put on this? I see this as really unlikely (perhaps <5%) such that ‘scaling stops working’ isn’t part of my model over the next 1-2yrs.
Only slightly surprised? IMO being able to autonomously rent cloud compute seems quite significant (technically and legally), and I’d be very surprised if something like this happened on a 1yr horizon. I’d be negatively surprised if the US government didn’t institute regulation on the operation of autonomous agents of this type by the end of 2024, basically due to their potential for misuse and their economic value. It may help to know how you’re operationalizing AIs that are ‘meaningfully aware of their own existence’.
I think it’s pretty unlikely that scaling literally stops working, maybe I’m 5-10% that we soon get to a point where there are only very small or negligible improvements to increasing compute. But I’m like 10-20% on some weaker version.
A weaker version could look like there are diminishing returns to performance from scaling compute (as is true), and this makes it very difficult for companies to continue scaling. One mechanism at play is that the marginal improvements from scaling may not be enough to produce the additional revenue needed to cover the scaling costs, this is especially true in a competitive market where it’s not clear scaling will put one ahead of their competitors.
In the context of the post, I think it’s quite unlikely that I see strong evidence in the next year indicating that scaling has stopped (if only because a year of no-progress is not sufficient evidence). I was more so trying to point to how there [sic] are contingencies which would make OpenAI’s adoption of an RSP less safety-critical. I stand by the statements that scaling no longer yielding returns would be such a contingency, but I agree that it’s pretty unlikely.
We are currently at ASL-2 in Anthropic’s RSP. Based on the categorization, ASL-3 is “low-level autonomous capabilities”. I think ASL-3 systems probably don’t meet the bar of “meaningfully in control of their own existence”, but they probably meet the thing I think is more likely:
I think it’s currently a good bet (>40%) that we will see ASL-3 systems in 2024.
I’m not sure how big of a jump if will be from that to “meaningfully in control of their own existence”. I would be surprised if it were a small jump, such that we saw AIs renting their own cloud compute in 2024, but this is quite plausible on my models.
I think the evidence indicates that this is a hard task, but not super hard. e.g., looking at ARC’s report on autonomous tasks, one model partially completes the task of setting up GPT-J via a cloud provider (with human help).
I’ll amend my position to just being “surprised” without the slightly, as I think this better captures my beliefs — thanks for the push to think about this more. Maybe I’m at 5-10%.
shrug, I’m being vague