Your infosecurity argument seems to involve fixing a point in time, and comparing a (more capable) centralized AI project against multiple (less capable) decentralized AI projects. However, almost all of the risks you’re considering depend much more on the capability of the AI project rather than the point in time at which they occur. So I think best practice here would be to fix a rough capability profile, and compare a (shorter timelines) centralized AI project against multiple (longer timelines) decentralized AI projects.
In more detail:
It’s not clear whether having one project would reduce the chance that the weights are stolen. We think that it would be harder to steal the weights of a single project, but the incentive to do so would also be stronger – it’s not clear how these balance out.
You don’t really spell out why the incentive to steal the weights is stronger, but my guess is that your argument here is “centralization --> more resources --> more capabilities --> more incentive to steal the weights”.
I would instead frame it as:
At a fixed capability level, the incentive to steal the weights will be the same, but the security practices of a centralized project will be improved. Therefore, holding capabilities fixed, having one project should reduce the chance that the weights are stolen.
Then separately I would also have a point that centralized AI projects get more resources and so should be expected to achieve a given capability profile sooner, which shortens timelines, the effects of which could then be considered separately (and which you presumably believe are less important, given that you don’t really consider them in the post).
(I get somewhat similar vibes from the section on racing, particularly about the point that China might also speed up, though it’s not quite as clear there.)
Fwiw, my own position is that for both infosec and racing it’s the brute fact that USG see fits to centralise all resources and develop AGI asap that would cause China to 1) try much harder to steal the weights than when private companies had developed the same capabilities themselves, 2) try much harder to race to AGI themselves.
So the argument here is either that China is more responsive to “social proof” of the importance of AI (rather than observations of AI capabilities), or that China wants to compete with USG for competition’s sake (e.g. showing they are as good as or better than USG)? I agree this is plausible.
It’s a bit weird to me to call this an “incentive”, since both of these arguments don’t seem to be making any sort of appeal to rational self-interest on China’s part. Maybe change it to “motivation”? I think that would have been clearer to me.
(Btw, you seem to be assuming that the core reason for centralization will be “beat China”, but it could also be “make this technology safe”. Presumably this would make a difference to this point as well as others in the post.)
I agree that centralising to make AI safe would make a difference. It seems a lot less likely to me than centralising to beat China (there’s already loads of beat China rhetoric, and it doesn’t seem very likely to go away).
Thanks, I expect you’re right that there’s some confusion in my thinking here.
Haven’t got to the bottom of it yet, but on more incentive to steal the weights: - partly I’m reasoning in the way that you guess, more resources → more capabilities → more incentives - I’m also thinking “stronger signal that the US is all in and thinks this is really important → raises p(China should also be all in) from a Chinese perspective → more likely China invests hard in stealing the weights” - these aren’t independent lines of reasoning, as the stronger signal is sent by spending more resources - but I tentatively think that it’s not the case that at a fixed capability level the incentives to steal the weights are the same. I think they’d be higher with a centralised project, as conditional on a centralised project there’s more reason for China to believe a) AGI is the one thing that matters, b) the US is out to dominate
While a centralized project would get more resources, it also has more ability to pause//investigate things.
So EG if the centralized project researchers see something concerning (perhaps early warning signs of scheming), it seems more able to do a bunch of research on that Concerning Thing before advancing.
I think makes the effect of centralization on timelines unclear, at least if one expects these kinds of “warning signs” on the pathway to very advanced capabilities.
(It’s plausible that you could get something like this from a decentralized model with sufficient oversight, but this seems much harder, especially if we expect most of the technical talent to stay with the decentralized projects as opposed to joining the oversight body.)
Tbc, I don’t want to strongly claim that centralization implies shorter timelines. Besides the point you raise there’s also things like bureaucracy and diseconomies of scale. I’m just trying to figure out what the authors of the post were saying.
That said, if I had to guess, I’d guess that centralization speeds up timelines.
Your infosecurity argument seems to involve fixing a point in time, and comparing a (more capable) centralized AI project against multiple (less capable) decentralized AI projects. However, almost all of the risks you’re considering depend much more on the capability of the AI project rather than the point in time at which they occur. So I think best practice here would be to fix a rough capability profile, and compare a (shorter timelines) centralized AI project against multiple (longer timelines) decentralized AI projects.
In more detail:
You don’t really spell out why the incentive to steal the weights is stronger, but my guess is that your argument here is “centralization --> more resources --> more capabilities --> more incentive to steal the weights”.
I would instead frame it as:
At a fixed capability level, the incentive to steal the weights will be the same, but the security practices of a centralized project will be improved. Therefore, holding capabilities fixed, having one project should reduce the chance that the weights are stolen.
Then separately I would also have a point that centralized AI projects get more resources and so should be expected to achieve a given capability profile sooner, which shortens timelines, the effects of which could then be considered separately (and which you presumably believe are less important, given that you don’t really consider them in the post).
(I get somewhat similar vibes from the section on racing, particularly about the point that China might also speed up, though it’s not quite as clear there.)
Fwiw, my own position is that for both infosec and racing it’s the brute fact that USG see fits to centralise all resources and develop AGI asap that would cause China to 1) try much harder to steal the weights than when private companies had developed the same capabilities themselves, 2) try much harder to race to AGI themselves.
So the argument here is either that China is more responsive to “social proof” of the importance of AI (rather than observations of AI capabilities), or that China wants to compete with USG for competition’s sake (e.g. showing they are as good as or better than USG)? I agree this is plausible.
It’s a bit weird to me to call this an “incentive”, since both of these arguments don’t seem to be making any sort of appeal to rational self-interest on China’s part. Maybe change it to “motivation”? I think that would have been clearer to me.
(Btw, you seem to be assuming that the core reason for centralization will be “beat China”, but it could also be “make this technology safe”. Presumably this would make a difference to this point as well as others in the post.)
Changed to motivation, thanks for the suggestion.
I agree that centralising to make AI safe would make a difference. It seems a lot less likely to me than centralising to beat China (there’s already loads of beat China rhetoric, and it doesn’t seem very likely to go away).
Thanks, I expect you’re right that there’s some confusion in my thinking here.
Haven’t got to the bottom of it yet, but on more incentive to steal the weights:
- partly I’m reasoning in the way that you guess, more resources → more capabilities → more incentives
- I’m also thinking “stronger signal that the US is all in and thinks this is really important → raises p(China should also be all in) from a Chinese perspective → more likely China invests hard in stealing the weights”
- these aren’t independent lines of reasoning, as the stronger signal is sent by spending more resources
- but I tentatively think that it’s not the case that at a fixed capability level the incentives to steal the weights are the same. I think they’d be higher with a centralised project, as conditional on a centralised project there’s more reason for China to believe a) AGI is the one thing that matters, b) the US is out to dominate
(Replied to Tom above)
While a centralized project would get more resources, it also has more ability to pause//investigate things.
So EG if the centralized project researchers see something concerning (perhaps early warning signs of scheming), it seems more able to do a bunch of research on that Concerning Thing before advancing.
I think makes the effect of centralization on timelines unclear, at least if one expects these kinds of “warning signs” on the pathway to very advanced capabilities.
(It’s plausible that you could get something like this from a decentralized model with sufficient oversight, but this seems much harder, especially if we expect most of the technical talent to stay with the decentralized projects as opposed to joining the oversight body.)
Tbc, I don’t want to strongly claim that centralization implies shorter timelines. Besides the point you raise there’s also things like bureaucracy and diseconomies of scale. I’m just trying to figure out what the authors of the post were saying.
That said, if I had to guess, I’d guess that centralization speeds up timelines.