I think I agree with at least the high level frame here – the details of how compute is going to get used has some strategic implications, and it seems at least plausible that chain-of-thought centric AI development will mean the first general AI will be fairly compute-bottlenecked, and that has some implications we should be modeling.
I don’t know that I buy the second-order implications. I agree that regulations that only focus on training compute are going to be insufficient (I already expected that, but I bought[1] the argument that the first AGI systems capable of RSI were likely to have training budgets over the 100M flops threshold, and be at least moderately likely to tripwire over SB 1047, which could give us an additional saving throw of policymakers thinking critically about it at a time that mattered)
I don’t think the solution is “don’t bother regulating at all”, it’s “actually try to followup legislation with better legislation as we learn more.”
I also think the actual calendar time of Recursive Self Improvement is still going to be short enough that it counts as “sharp takeoff” for most intents and purposes (i.e. it might be measured in months, but not years)
I think I agree with at least the high level frame here – the details of how compute is going to get used has some strategic implications, and it seems at least plausible that chain-of-thought centric AI development will mean the first general AI will be fairly compute-bottlenecked, and that has some implications we should be modeling.
I don’t know that I buy the second-order implications. I agree that regulations that only focus on training compute are going to be insufficient (I already expected that, but I bought[1] the argument that the first AGI systems capable of RSI were likely to have training budgets over the 100M flops threshold, and be at least moderately likely to tripwire over SB 1047, which could give us an additional saving throw of policymakers thinking critically about it at a time that mattered)
I don’t think the solution is “don’t bother regulating at all”, it’s “actually try to followup legislation with better legislation as we learn more.”
I also think the actual calendar time of Recursive Self Improvement is still going to be short enough that it counts as “sharp takeoff” for most intents and purposes (i.e. it might be measured in months, but not years)
and probably still buy