Ok, I think you might be lumping this under “international governance”, but I feel like you are severely underestimating the potential of open source AI. (Or off-brand secret projects based on open source AI).
Right now, the big labs have a lead, and that lead is expected to grow temporarily in the short term since they have been scaling their compute to ranges out of reach to B-class players.
But… What happens when algorithmic innovation gets accelerated? It’s already somewhat accelerated now. How secret are you going to manage to keep the findings about potential algorithmic improvements? It is a LOT easier for one untrustworthy employee to leak a concept about how to approach model architectures more efficiently than it is for them to steal the existing model weights.
Also, what do you do if existing open weights models at the time of international lockdowns on training runs are already sufficiently strong to be dangerous with only tweaks to the fine-tuning and RL protocols?
Is this so far outside your Overton window that you don’t feel it’s worth discussing?
Deepseek-V3 has 37B active params and 671 total params. Thats a ratio of about 18.14:1. What if someone made an MoE model using Llama 405B, with 405B active params and 7.4 trillion total params? What if a new pre-training run wasn’t needed, just RL fine-tuning?
What about Llama 4, that is in progress? Will your international governance prevent that from being released?
The realistic answer for decision making purposes (not epistemic purposes) is mostly hope that offense-defense balances are good enough to prevent the end of the world, combined with accepting somewhat larger risk of misalignment to prevent open source from being bad.
To be kind of blunt, any scenario where AI progress is algorithm dominated and a world where basically everyone can train superintelligence without nations being able to control it and a world where timelines are short is a world where governance of AI is more or less useless, and alignment becomes a little more dicey, so we should mostly ignore such worlds for utility purposes (they are valid for prediction/epistemic purposes).
But I’m approx 80% confident that that’s the world we’re in! I don’t want to just give up in 80% of cases!
I think we need to try to figure out some empirical measures for how much like this the world is, and at least come up with plans for what to do about it. I don’t think it’s hopeless, just that the plans would be rather differently shaped.
I think we need to try to figure out some empirical measures for how much like this the world is, and at least come up with plans for what to do about it.
I basically agree with this.
A crux here is that I believe that AI governance is basically worthless if you cannot control the ability to create powerful AIs, because they will soon be able to create states within a state, and thus resist any governance of AI plan you might wish to implement.
In other words, you need something close to a monopoly on violence in order for a government to continue existing, and under scenarios where AIs get very powerful very quickly because of algorithms, there is no good way to control the distribution of AI power, and it’s too easy for people to defect from governance on AI.
I’m not hopeless that such a world can survive. However, I do think AI governance breaks hard if we are in a future where AI power primarily is determined by algorithms, since I basically consider algorithmic advances mostly impossible to control.
I don’t think it’s hopeless, just that the plans would be rather differently shaped.
IMO, a plan for short timelines where algorithmic secrets are very powerful I would pivot hard towards technical safety/alignment, and would more or less abandon all but the most minimal governance/control plans, and the other big thing I’d focus on is on making the world’s offense-defense balance much better than it is now, which means we’d have to invest more in making bio-tech less risky, with the caveat that we cannot assume the government will successfully suppress bio-threats from civilian actors.
Ok, I think you might be lumping this under “international governance”, but I feel like you are severely underestimating the potential of open source AI. (Or off-brand secret projects based on open source AI). Right now, the big labs have a lead, and that lead is expected to grow temporarily in the short term since they have been scaling their compute to ranges out of reach to B-class players.
But… What happens when algorithmic innovation gets accelerated? It’s already somewhat accelerated now. How secret are you going to manage to keep the findings about potential algorithmic improvements? It is a LOT easier for one untrustworthy employee to leak a concept about how to approach model architectures more efficiently than it is for them to steal the existing model weights.
Also, what do you do if existing open weights models at the time of international lockdowns on training runs are already sufficiently strong to be dangerous with only tweaks to the fine-tuning and RL protocols?
Is this so far outside your Overton window that you don’t feel it’s worth discussing?
Deepseek-V3 has 37B active params and 671 total params. Thats a ratio of about 18.14:1. What if someone made an MoE model using Llama 405B, with 405B active params and 7.4 trillion total params? What if a new pre-training run wasn’t needed, just RL fine-tuning?
What about Llama 4, that is in progress? Will your international governance prevent that from being released?
The realistic answer for decision making purposes (not epistemic purposes) is mostly hope that offense-defense balances are good enough to prevent the end of the world, combined with accepting somewhat larger risk of misalignment to prevent open source from being bad.
To be kind of blunt, any scenario where AI progress is algorithm dominated and a world where basically everyone can train superintelligence without nations being able to control it and a world where timelines are short is a world where governance of AI is more or less useless, and alignment becomes a little more dicey, so we should mostly ignore such worlds for utility purposes (they are valid for prediction/epistemic purposes).
But I’m approx 80% confident that that’s the world we’re in! I don’t want to just give up in 80% of cases!
I think we need to try to figure out some empirical measures for how much like this the world is, and at least come up with plans for what to do about it. I don’t think it’s hopeless, just that the plans would be rather differently shaped.
I basically agree with this.
A crux here is that I believe that AI governance is basically worthless if you cannot control the ability to create powerful AIs, because they will soon be able to create states within a state, and thus resist any governance of AI plan you might wish to implement.
In other words, you need something close to a monopoly on violence in order for a government to continue existing, and under scenarios where AIs get very powerful very quickly because of algorithms, there is no good way to control the distribution of AI power, and it’s too easy for people to defect from governance on AI.
I’m not hopeless that such a world can survive. However, I do think AI governance breaks hard if we are in a future where AI power primarily is determined by algorithms, since I basically consider algorithmic advances mostly impossible to control.
IMO, a plan for short timelines where algorithmic secrets are very powerful I would pivot hard towards technical safety/alignment, and would more or less abandon all but the most minimal governance/control plans, and the other big thing I’d focus on is on making the world’s offense-defense balance much better than it is now, which means we’d have to invest more in making bio-tech less risky, with the caveat that we cannot assume the government will successfully suppress bio-threats from civilian actors.