Superintelligence Strategy paper seems to hold as a basic assumption that major state actors can’t be prevented from creating or stealing frontier AI, and the only moat is the amount of inference compute (if centralized, frontier training compute is only a small fraction of inference, and technical expertise is sufficiently widespread). Open weights models make it trivial for weaker rogue actors to gain access, but don’t help with inference compute.
If what the bad actor is trying to do with the AI is just get a clear set of instructions for a dangerous weapon, and a bit of help debugging lab errors… that costs only a trivial amount of inference compute.
In the paper, not letting weaker actors get access to frontier models and too much compute is the focus of Nonproliferation chapter. The framing in the paper suggests that in certain respects open weights models don’t make nearly as much of a difference. This is useful for distinguishing between various problems that open weights models can cause, as opposed to equally associating all possible problems with them.
Resources both closed and open must be overwhelmingly devoted to defense (vs offense) with respect to possible CBRN and other catastrophic risks from both open and closed models[1]. Otherwise the risk of easy offense, hard defense weapons (like bioweapons) puts civilization at dire risk. Competition and the race to AGI could be seen as a significant detractor from the impetus to devote these necessarily overwhelming resources[2].
So how can we reduce possible recklessness from competition without centralized and therefore most likely corrupt control? To me transparency and open source provide an alternative: Transparency into what the closed hyper-scalers are doing with their billions of dollars worth of inference+training compute[3]; And open source + open science to promote healthy competition and innovation along with public insight into safety and security implications.
With such openness, we must assume there will be a degree of malicious misuse. Again, knowing this upfront, we need to devote both inference and training compute now to heading off such threats[2]. Yes it’s easier to destroy than to create & protect; this is why we must devote overwhelmingly more resources to the latter.
This as controlling and closing CBRN capable models, like you mention, is not likely to happen and bad actors should be assumed to have access already.
Since CBRN defense is an advanced capability and requires complex reasoning, it could actually provide an alignment bonus (vs being an alignment tax) to frontier models. So we should not necessarily equate defense and capability as mutually exclusive.
Superintelligence Strategy paper seems to hold as a basic assumption that major state actors can’t be prevented from creating or stealing frontier AI, and the only moat is the amount of inference compute (if centralized, frontier training compute is only a small fraction of inference, and technical expertise is sufficiently widespread). Open weights models make it trivial for weaker rogue actors to gain access, but don’t help with inference compute.
If what the bad actor is trying to do with the AI is just get a clear set of instructions for a dangerous weapon, and a bit of help debugging lab errors… that costs only a trivial amount of inference compute.
In the paper, not letting weaker actors get access to frontier models and too much compute is the focus of Nonproliferation chapter. The framing in the paper suggests that in certain respects open weights models don’t make nearly as much of a difference. This is useful for distinguishing between various problems that open weights models can cause, as opposed to equally associating all possible problems with them.
Resources both closed and open must be overwhelmingly devoted to defense (vs offense) with respect to possible CBRN and other catastrophic risks from both open and closed models[1]. Otherwise the risk of easy offense, hard defense weapons (like bioweapons) puts civilization at dire risk. Competition and the race to AGI could be seen as a significant detractor from the impetus to devote these necessarily overwhelming resources[2].
So how can we reduce possible recklessness from competition without centralized and therefore most likely corrupt control? To me transparency and open source provide an alternative: Transparency into what the closed hyper-scalers are doing with their billions of dollars worth of inference+training compute[3]; And open source + open science to promote healthy competition and innovation along with public insight into safety and security implications.
With such openness, we must assume there will be a degree of malicious misuse. Again, knowing this upfront, we need to devote both inference and training compute now to heading off such threats[2]. Yes it’s easier to destroy than to create & protect; this is why we must devote overwhelmingly more resources to the latter.
This as controlling and closing CBRN capable models, like you mention, is not likely to happen and bad actors should be assumed to have access already.
Since CBRN defense is an advanced capability and requires complex reasoning, it could actually provide an alignment bonus (vs being an alignment tax) to frontier models. So we should not necessarily equate defense and capability as mutually exclusive.
E.g. there should be sufficient compute dedicated to advancing CBRN defensive capability