Same idea though. I don’t see why “the military” can’t do recursion using their own AIs and use custom hardware to outcompete any “rogues”.
One of the deep fundamental reasons here is alignment failures. Either the “military” isn’t trying very hard, or humans know they haven’t solved alignment. Humans know they can’t build a functional “military” AI, all they can do is make another rouge AI. Or the humans don’t know that, and the military AI is another rouge AI.
For this military AI to be fighting other AI’s on behalf of humans, a lot of alignment work has to go right.
The second deep reason is that recursive self improvement is a strong positive feedback loop. It isn’t clear how strong, but it could be Very strong. So suppose the first AI undergoes a recursive improvement FOOM. And it happens that the rouge AI gets there before any military. Perhaps because the creators of the military AI are taking their time to check the alignment theory.
Positive feedback loops tend to amplify small differences.
Also, about all those hardware differences. A smart AI might well come up with a design that efficiently uses old hardware. Oh, and this is all playing out in the future, not now. Maybe the custom AI hardware is everywhere by the time this is happening.
I suspect if AI is anything like computer graphics there will be at least 5-10 paradigm shifts to new architectures that need updated hardware to run, obsoleting everything deployed, before settling in something that is optimal. Flops are not actually fungible and Turing complete doesn’t mean your training run will complete this century.
This is with humans doing the research. Humans invent new algorithms more slowly than new chips are made. So it makes sense to adjust the algorithm to the chip. If the AI can do software research far faster than any human, adjusting the software to the hardware (an approach that humans use a lot throughout most of computing) becomes an even better idea.
This is with humans doing the research. Humans invent new algorithms more slowly than new chips are made. So it makes sense to adjust the algorithm to the chip. If the AI can do software research far faster than any human, adjusting the software to the hardware (an approach that humans use a lot throughout most of computing) becomes an even better idea.
Note that if you are seeking an improved network architecture and you need it to work on a limited family of chips, this is constraining your search. You may not be able to find a meaningful improvement over the sota with that constraint in place, regardless of your intelligence level. Something like sparsity, in memory compute, neural architecture (this is where the chip is structured like the network it is modeling with dedicated hardware) can offer colossal speedups.
this is constraining your search. You may not be able to find a meaningful improvement over the sota with that constraint in place, regardless of your intelligence level.
I mean the space of algorithms that can run on an existing chip is pretty huge. Yes it is a constraint. And it’s theoretically possible that the search could return no solutions, if the SOTA was achieved with Much better chips, or was near optimal already, or the agent doing the search wasn’t much smarter than us.
For example, there are techniques that decompose a matrix into its largest eigenvectors. Which works great without needing sparse hardware.
One of the deep fundamental reasons here is alignment failures. Either the “military” isn’t trying very hard, or humans know they haven’t solved alignment. Humans know they can’t build a functional “military” AI, all they can do is make another rouge AI. Or the humans don’t know that, and the military AI is another rouge AI.
For this military AI to be fighting other AI’s on behalf of humans, a lot of alignment work has to go right.
The second deep reason is that recursive self improvement is a strong positive feedback loop. It isn’t clear how strong, but it could be Very strong. So suppose the first AI undergoes a recursive improvement FOOM. And it happens that the rouge AI gets there before any military. Perhaps because the creators of the military AI are taking their time to check the alignment theory.
Positive feedback loops tend to amplify small differences.
Also, about all those hardware differences. A smart AI might well come up with a design that efficiently uses old hardware. Oh, and this is all playing out in the future, not now. Maybe the custom AI hardware is everywhere by the time this is happening.
This is with humans doing the research. Humans invent new algorithms more slowly than new chips are made. So it makes sense to adjust the algorithm to the chip. If the AI can do software research far faster than any human, adjusting the software to the hardware (an approach that humans use a lot throughout most of computing) becomes an even better idea.
Note that if you are seeking an improved network architecture and you need it to work on a limited family of chips, this is constraining your search. You may not be able to find a meaningful improvement over the sota with that constraint in place, regardless of your intelligence level. Something like sparsity, in memory compute, neural architecture (this is where the chip is structured like the network it is modeling with dedicated hardware) can offer colossal speedups.
I mean the space of algorithms that can run on an existing chip is pretty huge. Yes it is a constraint. And it’s theoretically possible that the search could return no solutions, if the SOTA was achieved with Much better chips, or was near optimal already, or the agent doing the search wasn’t much smarter than us.
For example, there are techniques that decompose a matrix into its largest eigenvectors. Which works great without needing sparse hardware.