(Because the “military” AIs working with humans will have this kind of hardware to hunt them down with)
You need to make a lot of extra assumptions about the world for this reasoning to work.
These “military” AI’s need to exist. And they need to be reasonably loosely chained. If their safety rules are so strict they can’t do anything, they can’t do anything however fast they don’t do it. They need to be actively trying to do their job, as opposed to playing along for the humans but not really caring. They need to be smart enough. If the escaped AI uses some trick that the “military” AI just can’t comprehend, then it fails to comprehend again and again, very fast.
Of course. Implicitly I am assuming that intelligent algorithms have diminishing returns on architectural complexity. So if the “military” owned model is simple with an architecture + training suite controlled and understood by humans, the assumption is the “free” model cannot be effectively that much more intelligent with a better architecture if it lacks compute by a factor of millions. That greater intelligence is a scale dependent phenomenon.
This is so far consistent with empirical evidence. Do you happen to know of evidence to challenge this assumption? As far as I know with LLM experiments, there are tweaks to architecture but the main determinant for benchmark performance is model+data scale (which are interdependent), and non transformer architectures seem to show similar emergent properties.
As far as I know with LLM experiments, there are tweaks to architecture but the main determinant for benchmark performance is model+data scale (which are interdependent), and non transformer architectures seem to show similar emergent properties.
So within the rather limited subspace of LLM architectures, all architectures are about the same.
Ie once you ignore the huge space of architectures that just ignore the data and squander compute, then architecture doesn’t matter. Ie we have one broad family of techniques, (with gradient decent, text prediction etc) and anything in that family is about equally good. And anything outside basically doesn’t work at all.
This looks to me to be fairly strong evidence that you can’t get a large improvement in performance by randomly bumbling around with small architecture tweaks to existing models.
Does this say anything about whether a fundamentally different approach might do better? No. We can’t tell that from this evidence. Although looking at the human brain, we can see it seems to be more data efficient than LLM’s. And we know that in theory models could be Much more data efficient. Addition is very simple. Solomonov induction would have it as a major hypothesis after seeing only a couple of examples. But GPT2 saw loads of arithmetic in training, and still couldn’t reliably do it.
So I think LLM architectures form a flat bottomed local semi-minima (minimal in at least most dimensions). It’s hard to get big improvements just by tweaking it. (We are applying enough grad student descent to ensure that) but nowhere near global optimal.
Suppose everything is really data bottlenecked, and the slower AI has a more data efficient algorithm. Or maybe the slower AI knows how to make synthetic data, and the human trained AI doesn’t.
There’s one interesting technical detail. Human brain uses heavy irregular sparsity. This is where most possible connections between layers have no connection—zero weight. On a gpu, there is limited performance improvement from sparsity. This is because the hardware subunits can only calculate sparse activations if the memory address patterns are regular. Irregular sparsity doesn’t have hardware support.
Future neural processors will support full sparsity. This will allow them to run 100x faster probably with the same amount of silicon (way less macs but you have layer activation multicast units)- but only on new specialized hardware.
It’s possible that whatever neural architectures that are found using recursion—that leave the state space of llms or human brains—will have similar compute requirements. That they will be functionally useless in current hardware, running thousands of times faster on purpose built hardware.
Same idea though. I don’t see why “the military” can’t do recursion using their own AIs and use custom hardware to outcompete any “rogues”.
Also it provides a simple way to keep control of AI : track the location of custom hardware, know your customer, etc.
I suspect if AI is anything like computer graphics there will be at least 5-10 paradigm shifts to new architectures that need updated hardware to run, obsoleting everything deployed, before settling in something that is optimal. Flops are not actually fungible and Turing complete doesn’t mean your training run will complete this century.
Same idea though. I don’t see why “the military” can’t do recursion using their own AIs and use custom hardware to outcompete any “rogues”.
One of the deep fundamental reasons here is alignment failures. Either the “military” isn’t trying very hard, or humans know they haven’t solved alignment. Humans know they can’t build a functional “military” AI, all they can do is make another rouge AI. Or the humans don’t know that, and the military AI is another rouge AI.
For this military AI to be fighting other AI’s on behalf of humans, a lot of alignment work has to go right.
The second deep reason is that recursive self improvement is a strong positive feedback loop. It isn’t clear how strong, but it could be Very strong. So suppose the first AI undergoes a recursive improvement FOOM. And it happens that the rouge AI gets there before any military. Perhaps because the creators of the military AI are taking their time to check the alignment theory.
Positive feedback loops tend to amplify small differences.
Also, about all those hardware differences. A smart AI might well come up with a design that efficiently uses old hardware. Oh, and this is all playing out in the future, not now. Maybe the custom AI hardware is everywhere by the time this is happening.
I suspect if AI is anything like computer graphics there will be at least 5-10 paradigm shifts to new architectures that need updated hardware to run, obsoleting everything deployed, before settling in something that is optimal. Flops are not actually fungible and Turing complete doesn’t mean your training run will complete this century.
This is with humans doing the research. Humans invent new algorithms more slowly than new chips are made. So it makes sense to adjust the algorithm to the chip. If the AI can do software research far faster than any human, adjusting the software to the hardware (an approach that humans use a lot throughout most of computing) becomes an even better idea.
This is with humans doing the research. Humans invent new algorithms more slowly than new chips are made. So it makes sense to adjust the algorithm to the chip. If the AI can do software research far faster than any human, adjusting the software to the hardware (an approach that humans use a lot throughout most of computing) becomes an even better idea.
Note that if you are seeking an improved network architecture and you need it to work on a limited family of chips, this is constraining your search. You may not be able to find a meaningful improvement over the sota with that constraint in place, regardless of your intelligence level. Something like sparsity, in memory compute, neural architecture (this is where the chip is structured like the network it is modeling with dedicated hardware) can offer colossal speedups.
this is constraining your search. You may not be able to find a meaningful improvement over the sota with that constraint in place, regardless of your intelligence level.
I mean the space of algorithms that can run on an existing chip is pretty huge. Yes it is a constraint. And it’s theoretically possible that the search could return no solutions, if the SOTA was achieved with Much better chips, or was near optimal already, or the agent doing the search wasn’t much smarter than us.
For example, there are techniques that decompose a matrix into its largest eigenvectors. Which works great without needing sparse hardware.
You need to make a lot of extra assumptions about the world for this reasoning to work.
These “military” AI’s need to exist. And they need to be reasonably loosely chained. If their safety rules are so strict they can’t do anything, they can’t do anything however fast they don’t do it. They need to be actively trying to do their job, as opposed to playing along for the humans but not really caring. They need to be smart enough. If the escaped AI uses some trick that the “military” AI just can’t comprehend, then it fails to comprehend again and again, very fast.
Of course. Implicitly I am assuming that intelligent algorithms have diminishing returns on architectural complexity. So if the “military” owned model is simple with an architecture + training suite controlled and understood by humans, the assumption is the “free” model cannot be effectively that much more intelligent with a better architecture if it lacks compute by a factor of millions. That greater intelligence is a scale dependent phenomenon.
This is so far consistent with empirical evidence. Do you happen to know of evidence to challenge this assumption? As far as I know with LLM experiments, there are tweaks to architecture but the main determinant for benchmark performance is model+data scale (which are interdependent), and non transformer architectures seem to show similar emergent properties.
So within the rather limited subspace of LLM architectures, all architectures are about the same.
Ie once you ignore the huge space of architectures that just ignore the data and squander compute, then architecture doesn’t matter. Ie we have one broad family of techniques, (with gradient decent, text prediction etc) and anything in that family is about equally good. And anything outside basically doesn’t work at all.
This looks to me to be fairly strong evidence that you can’t get a large improvement in performance by randomly bumbling around with small architecture tweaks to existing models.
Does this say anything about whether a fundamentally different approach might do better? No. We can’t tell that from this evidence. Although looking at the human brain, we can see it seems to be more data efficient than LLM’s. And we know that in theory models could be Much more data efficient. Addition is very simple. Solomonov induction would have it as a major hypothesis after seeing only a couple of examples. But GPT2 saw loads of arithmetic in training, and still couldn’t reliably do it.
So I think LLM architectures form a flat bottomed local semi-minima (minimal in at least most dimensions). It’s hard to get big improvements just by tweaking it. (We are applying enough grad student descent to ensure that) but nowhere near global optimal.
Suppose everything is really data bottlenecked, and the slower AI has a more data efficient algorithm. Or maybe the slower AI knows how to make synthetic data, and the human trained AI doesn’t.
There’s one interesting technical detail. Human brain uses heavy irregular sparsity. This is where most possible connections between layers have no connection—zero weight. On a gpu, there is limited performance improvement from sparsity. This is because the hardware subunits can only calculate sparse activations if the memory address patterns are regular. Irregular sparsity doesn’t have hardware support.
Future neural processors will support full sparsity. This will allow them to run 100x faster probably with the same amount of silicon (way less macs but you have layer activation multicast units)- but only on new specialized hardware.
It’s possible that whatever neural architectures that are found using recursion—that leave the state space of llms or human brains—will have similar compute requirements. That they will be functionally useless in current hardware, running thousands of times faster on purpose built hardware.
Same idea though. I don’t see why “the military” can’t do recursion using their own AIs and use custom hardware to outcompete any “rogues”.
Also it provides a simple way to keep control of AI : track the location of custom hardware, know your customer, etc.
I suspect if AI is anything like computer graphics there will be at least 5-10 paradigm shifts to new architectures that need updated hardware to run, obsoleting everything deployed, before settling in something that is optimal. Flops are not actually fungible and Turing complete doesn’t mean your training run will complete this century.
One of the deep fundamental reasons here is alignment failures. Either the “military” isn’t trying very hard, or humans know they haven’t solved alignment. Humans know they can’t build a functional “military” AI, all they can do is make another rouge AI. Or the humans don’t know that, and the military AI is another rouge AI.
For this military AI to be fighting other AI’s on behalf of humans, a lot of alignment work has to go right.
The second deep reason is that recursive self improvement is a strong positive feedback loop. It isn’t clear how strong, but it could be Very strong. So suppose the first AI undergoes a recursive improvement FOOM. And it happens that the rouge AI gets there before any military. Perhaps because the creators of the military AI are taking their time to check the alignment theory.
Positive feedback loops tend to amplify small differences.
Also, about all those hardware differences. A smart AI might well come up with a design that efficiently uses old hardware. Oh, and this is all playing out in the future, not now. Maybe the custom AI hardware is everywhere by the time this is happening.
This is with humans doing the research. Humans invent new algorithms more slowly than new chips are made. So it makes sense to adjust the algorithm to the chip. If the AI can do software research far faster than any human, adjusting the software to the hardware (an approach that humans use a lot throughout most of computing) becomes an even better idea.
Note that if you are seeking an improved network architecture and you need it to work on a limited family of chips, this is constraining your search. You may not be able to find a meaningful improvement over the sota with that constraint in place, regardless of your intelligence level. Something like sparsity, in memory compute, neural architecture (this is where the chip is structured like the network it is modeling with dedicated hardware) can offer colossal speedups.
I mean the space of algorithms that can run on an existing chip is pretty huge. Yes it is a constraint. And it’s theoretically possible that the search could return no solutions, if the SOTA was achieved with Much better chips, or was near optimal already, or the agent doing the search wasn’t much smarter than us.
For example, there are techniques that decompose a matrix into its largest eigenvectors. Which works great without needing sparse hardware.