I think It’s closer to my view than either of the above, but I am somewhere between this and Thane’s view. I, like Stephen Byrnes, suspect that AGI will be more effective and efficient once it has figured out how to incorporate more insights from neuroscience. I think there are going to be some fairly fundamental differences that make it hard to extrapolate specific findings from today’s model architectures to these future architectures.
I can’t be sure of this, and I certainly don’t argue that work on aligning current models should stop, but I’m also not sure that even taking the more open-minded approach called upon here is sufficiently weird enough to capture the differences.
For example:
view that AGI will emerge from a rapidly evolving ecosystem of heterogeneous building blocks and specialised components makes me think that “intelligence containment”, especially through compute governance, will be very short-lived.
Then, if we assume that the “G factor” containment is probably futile, AI policy and governance folks should perhaps start paying more attention to the governance of competence through the control of the access to the training data.
I completely agree with the first assertion that I expect compute governance to be fairly short lived. I’m hopeful it can grant us a couple years, but not hopeful that it can grant us 10 years.
However, I disagree with the second assertion that training data governance would be more helpful. I do think it wouldn’t be a bad idea, especially with encouraging frontier labs to be more thoughtful about excluding some nasty weapons tech from the training data. I don’t think you are likely to get an extended period of successful AI governance from including training data as well as compute.
For three reasons:
a lot of internet data (text, video, audio, scientific data, etc.) is being generated in a rapid way. It would be far more difficult to regulate the secret collection of this general purpose data than it would be to restrict unauthorized use of large datacenters.
If the ‘more brain-like AGI’ is the path forwards that the tech takes, then I expect from looking at the data rates of sensory inputs, adjusted for the learning-relevant information value of those inputs, and the rates of intra-brain-region communications, that data would be utilized far more effectively by brain-like AGI. Thus, data wouldn’t be a bottleneck.
I also expect that compute for the training and inference is going to be quite cheap relative to current frontier models. Despite this, I am hopeful for compute governance providing a delay because I expect that the fastest path from current models to brain-like AGI would be through using a combination of current models and various architecture search techniques to discover efficient ways to make a brain-like AGI. So by regulating the compute that someone could use to do that search, you slow down the initial finding of the better algorithm even though you fail to regulate the algorithm once it is discovered.
I agree that training data governance is not robust to non-cooperative actors. But I think there is a much better chance to achieve a very broad industrial, academic, international, and legal consensus about it being a good way to jigsaw capabilities without sacrificing the raw reasoning ability, which the opponents of compute governance hold as purely counter-productive (“intelligence just makes things better”). That’s why I titled my post “Open Agency model can solve the AI regulation dilemma” (emphasis on the last word).
This could even be seen not just as a “safety” measure, but as a truly good regularisation measure of the collective civilisational intelligence: to make intelligence more robust to distributional shifts and paradigm shifts, it’s better to compartmentalise it and make communication between the compartments going through a relatively narrow, classical informational channel, namely human language or specific protocols rather than raw DNN activation dynamics.
Yes, I agree there’s a lot of value in thoughtful regulation of training data (whether government enforced or voluntary) by cooperative actors. You raise good points. I was meaning just to refer to the control of non-cooperative actors.
This post hits a compromise between Thane’s view and Quinton/Nora Belrose’s view.
I think It’s closer to my view than either of the above, but I am somewhere between this and Thane’s view. I, like Stephen Byrnes, suspect that AGI will be more effective and efficient once it has figured out how to incorporate more insights from neuroscience. I think there are going to be some fairly fundamental differences that make it hard to extrapolate specific findings from today’s model architectures to these future architectures.
I can’t be sure of this, and I certainly don’t argue that work on aligning current models should stop, but I’m also not sure that even taking the more open-minded approach called upon here is sufficiently weird enough to capture the differences.
I completely agree with the first assertion that I expect compute governance to be fairly short lived. I’m hopeful it can grant us a couple years, but not hopeful that it can grant us 10 years.
However, I disagree with the second assertion that training data governance would be more helpful. I do think it wouldn’t be a bad idea, especially with encouraging frontier labs to be more thoughtful about excluding some nasty weapons tech from the training data. I don’t think you are likely to get an extended period of successful AI governance from including training data as well as compute.
For three reasons:
a lot of internet data (text, video, audio, scientific data, etc.) is being generated in a rapid way. It would be far more difficult to regulate the secret collection of this general purpose data than it would be to restrict unauthorized use of large datacenters.
If the ‘more brain-like AGI’ is the path forwards that the tech takes, then I expect from looking at the data rates of sensory inputs, adjusted for the learning-relevant information value of those inputs, and the rates of intra-brain-region communications, that data would be utilized far more effectively by brain-like AGI. Thus, data wouldn’t be a bottleneck.
I also expect that compute for the training and inference is going to be quite cheap relative to current frontier models. Despite this, I am hopeful for compute governance providing a delay because I expect that the fastest path from current models to brain-like AGI would be through using a combination of current models and various architecture search techniques to discover efficient ways to make a brain-like AGI. So by regulating the compute that someone could use to do that search, you slow down the initial finding of the better algorithm even though you fail to regulate the algorithm once it is discovered.
I agree that training data governance is not robust to non-cooperative actors. But I think there is a much better chance to achieve a very broad industrial, academic, international, and legal consensus about it being a good way to jigsaw capabilities without sacrificing the raw reasoning ability, which the opponents of compute governance hold as purely counter-productive (“intelligence just makes things better”). That’s why I titled my post “Open Agency model can solve the AI regulation dilemma” (emphasis on the last word).
This could even be seen not just as a “safety” measure, but as a truly good regularisation measure of the collective civilisational intelligence: to make intelligence more robust to distributional shifts and paradigm shifts, it’s better to compartmentalise it and make communication between the compartments going through a relatively narrow, classical informational channel, namely human language or specific protocols rather than raw DNN activation dynamics.
Yes, I agree there’s a lot of value in thoughtful regulation of training data (whether government enforced or voluntary) by cooperative actors. You raise good points. I was meaning just to refer to the control of non-cooperative actors.