Indeed. We are in trouble, and there is no plan as of today. We are soon going to blow past autonomous replication, and then adaptation and R&D. There are almost no remaining clear red lines.
In light of this, we appeal to the AI research and policy communities to quickly increase research into and funding around this difficult topic.
hum, unsure, honestly I don’t think we need much more research on this. What kind of research are you proposing? like I think that the only sensible policy that I see for open-source AI is that we should avoid models that are able to do AI R&D in the wild, and a clear Shelling point for this is stopping before full ARA. But we definitely need more advocacy.
the only sensible policy that I see for open-source AI is that we should avoid models that are able to do AI R&D in the wild, and a clear Shelling point for this is stopping before full ARA
This is insufficient, because capabilities latent in an open weights model can be elicited later, possibly much later, after frontier models acquire them. Llama-3-405B can now be extremely cheaply fine-tuned on mere 1K reasoning traces of the Feb 2025 s1 dataset (paper) to become a thinking model (with long reasoning traces). This wasn’t possible at the time of its release in Jul 2024.
This is not as salient currently because DeepSeek-R1 is open weights anyway and much cheaper to inference, but if it wasn’t, then Llama-3-405B would’ve become the most capable open weights reasoning model. When frontier models gain R&D and full ARA capabilities, it’ll likely become possible to finetune (and scaffold) Llama 4 to gain them as well, even as in the next few weeks (before its release) these capabilities remain completely inaccessible.
Proliferation for open weights models must be measured in perplexity and training compute, not in capabilities that are currently present or possible to elicit, because what’s possible to elicit will change, while proliferation is immediately irreversible.
Indeed. We are in trouble, and there is no plan as of today. We are soon going to blow past autonomous replication, and then adaptation and R&D. There are almost no remaining clear red lines.
hum, unsure, honestly I don’t think we need much more research on this. What kind of research are you proposing? like I think that the only sensible policy that I see for open-source AI is that we should avoid models that are able to do AI R&D in the wild, and a clear Shelling point for this is stopping before full ARA. But we definitely need more advocacy.
This is insufficient, because capabilities latent in an open weights model can be elicited later, possibly much later, after frontier models acquire them. Llama-3-405B can now be extremely cheaply fine-tuned on mere 1K reasoning traces of the Feb 2025 s1 dataset (paper) to become a thinking model (with long reasoning traces). This wasn’t possible at the time of its release in Jul 2024.
This is not as salient currently because DeepSeek-R1 is open weights anyway and much cheaper to inference, but if it wasn’t, then Llama-3-405B would’ve become the most capable open weights reasoning model. When frontier models gain R&D and full ARA capabilities, it’ll likely become possible to finetune (and scaffold) Llama 4 to gain them as well, even as in the next few weeks (before its release) these capabilities remain completely inaccessible.
Proliferation for open weights models must be measured in perplexity and training compute, not in capabilities that are currently present or possible to elicit, because what’s possible to elicit will change, while proliferation is immediately irreversible.
I think, in the policy world, perplexity will never be fashionable.
Training compute maybe, but if so, how to ban llama3? This is already too late
If so, the only policy that is see is red lines at full ARA.
And we need to pray that this is sufficient, and that the buffer between ara and takeover is sufficient. I think it is.