Oh huh, I hadn’t even considered that interpretation. Personally, I think Leopold’s key error is in underrating how soon we will get to AGI if we continue as we have been, and in not thinking that that is as dangerous an achievement as I think it is.
So, if your interpretation of ‘overrates chance of existential catastrophe’ is correct, I am of the opposite opinion. Seems like Leopold expects we can make good use of AGI without a bunch more alignment. I think we’ll just doom ourselves if we try to use it.
Yes, and my modal time-to-AGI is late 2025 / early 2026. I think we’re right on the brink of a pre-AGI recursive self-improvement loop which will quickly rocket us past AGI. I think we are already in a significant compute overhang and data overhang. In other words, that software improvements alone can be more than sufficient.
In other words, I am concerned.
The difference between these two estimates feels like it can be pretty well accounted for by reasonable expected development friction for prototype-humanish-level self-improvers, who will still be subject to many (minus some) of the same limitations that prevent “9 woman from growing a baby in a month”. You can predict they’ll be able to lubricate more or less of that, but we can’t currently strictly scale project speeds by throwing masses of software engineers and money at it.
I believe you are correct about the importance of taking these phenomena into account: indivisibility of certain serial tasks, coordination overhead of larger team sizes.
I do think that my model takes these into account.
It’s certainly possible that my model is wrong. I feel like there’s a lot of uncertainty in many key variables, and likely I have overlooked things. The phenomena you point out don’t happen to be things that I neglected to consider though.
I understand—my point is more that the difference between these two positions could be readily explained by you being slightly more optimistic in estimated task time when doing the accounting, and the voice of experience saying “take your best estimate of the task time, and double it, and that’s what it actually is”.
Oh huh, I hadn’t even considered that interpretation. Personally, I think Leopold’s key error is in underrating how soon we will get to AGI if we continue as we have been, and in not thinking that that is as dangerous an achievement as I think it is.
So, if your interpretation of ‘overrates chance of existential catastrophe’ is correct, I am of the opposite opinion. Seems like Leopold expects we can make good use of AGI without a bunch more alignment. I think we’ll just doom ourselves if we try to use it.
His modal time-to-AGI is like 2027, with a 2-3 year intelligence explosion afterwards before humanity is ~ irrelevant.
Yeah this seems likely.
Yes, and my modal time-to-AGI is late 2025 / early 2026. I think we’re right on the brink of a pre-AGI recursive self-improvement loop which will quickly rocket us past AGI. I think we are already in a significant compute overhang and data overhang. In other words, that software improvements alone can be more than sufficient. In other words, I am concerned.
The difference between these two estimates feels like it can be pretty well accounted for by reasonable expected development friction for prototype-humanish-level self-improvers, who will still be subject to many (minus some) of the same limitations that prevent “9 woman from growing a baby in a month”. You can predict they’ll be able to lubricate more or less of that, but we can’t currently strictly scale project speeds by throwing masses of software engineers and money at it.
I believe you are correct about the importance of taking these phenomena into account: indivisibility of certain serial tasks, coordination overhead of larger team sizes. I do think that my model takes these into account.
It’s certainly possible that my model is wrong. I feel like there’s a lot of uncertainty in many key variables, and likely I have overlooked things. The phenomena you point out don’t happen to be things that I neglected to consider though.
I understand—my point is more that the difference between these two positions could be readily explained by you being slightly more optimistic in estimated task time when doing the accounting, and the voice of experience saying “take your best estimate of the task time, and double it, and that’s what it actually is”.