The way I would put it is: there is a wide range of competence and organizational efficacy in any government, and in most of the ways in which you could slice up any government. They include many of the most and least competent people in the world. And governments have access to extraordinary levels of resources and powers. The richest billionaire in the world can scarcely spend annually what an obscure city you’ve never heard of might spend annually on healthcare or pensions.
But governments (and all large organizations, such as corporations) suffer from an extreme deficit of coordination and executive function (pun very intended). Highly-functional organizations cannot replicate themselves, cannot grow without losing functionality, cannot evolve etc. There are permanent productivity differences in businesses doing the exact same thing, because the net selection is too weak—organizations are barely able to maintain their status quo. Oversight and accountability are difficult, there are only a few competent managers or troubleshooters to go around, only a few isolated islands of competence which can be trusted, there’s always a crisis elsewhere as a distraction… A large organization cannot walk and chew bubblegum; leadership like CEOs exist mostly to (1) decide when to stop walking and start chewing bubblegum instead, and (2) yell at everyone that the goal is now ‘chewing bubblegum’ for as long as it takes for the vast lumbering giant to halt. They are much like humans in having a spotlight of conscious attention and a working memory with just a few slots, except that instead of the ‘magic number 7±2’, it’s a little more like ‘the not-so-magic number 0–2’; and then everything else carries on autonomously. Attempts to pay attention to more things just leads to accomplishing nothing, or disaster. (Imagine if you had to consciously control your breathing and heart rate and all the other things your autonomous nervous system does! not to mention the 99% of cognition which is inaccessible to consciousness.)
So, large organizations can often accomplish anything they want, but they cannot accomplish everything. They often fail on the meta-level of wanting to accomplish the right thing, and go off and accomplish the wrong things.
Whereas cases of successful projects like the Manhattan Project/Project Nobska/Skunkworks/Apollo Project/Operation Warp Speed are cases where the large organization decides to accomplish the right thing: it makes the project genuinely a top priority, assigning the top people to it, and ensuring it is not held hostage to autonomous routine concerns or nice-to-have things like being environmentally-friendly. (The Eye of Sauron turns its baleful gaze away from the broader war and towards any sources of slowdown, burning through them.) When the Manhattan Project needed tons of highly-conductive metal like copper, they didn’t waste time requesting it or even seizing it from other military needs; they simply borrowed tons of silver from Fort Knox. (More balefully, they didn’t give a toss about ‘pollution’ or ‘safety’, and so the cleanup goes on to this day—costing OOMs more than the Manhattan Project itself ever did!)
When people talk about ‘We need a Manhattan Project’, what they are really saying is not ‘we need to spend $ $ $ on this’ (often $ $ $ has already been spent) but that the large-organization should burn 1 slot of attention on that goal, setting a hard endpoint with Consequences™.
When the Eye of Sauron does turn its gaze onto a specific project, the results are shocking.
In an intelligence community context, the American spy satellites like the KH program achieved astonishing things in photography, physics, and rocketry—things like handling ultra-high-resolution photography in space (with its unique problems like disposing of hundreds of gallons of water in space) or scooping up landing satellites in helicopters were just the start. (I was skimming a book the other day which included some hilarious anecdotes—like American spies would go take tourist photos of themselves in places like Red Square just to assist trigonometry for photo analysis.) American presidents obsessed over the daily spy satellite reports, and this helped ensure that the spy satellite footage was worth obsessing over. (Amateurs fear the CIA, but pros fear NRO.)
Or consider the NSA. Specifically, the Snowden leaks: systematic, well-maintained, engineered, comprehensive, extensive, utterly successfully kept secret hacks of everyone. At this point, probably a lot of LW2 readers weren’t ‘around’ for that, but as humdrum as they may seem now—‘oh sure, of course the NSA was spying on and hacking everyone’—the leaks were massively traumatic to the infosec sector because people just couldn’t imagine that basically all the medium-case scenarios were true simultaneously—the NSA really had backdoored your cipher, really was piggybacking on other state actors, really was keeping all the hacks successfully secret by extreme levels of OPSEC which is why hardly anyone had ever seen a NSA hack, really was recording all Internet traffic at some points, really was storing much traffic to crack later with quantum computers, really was intercepting your computer in the mail to add hardware backdoors, etc. They might’ve speculated idly about this or that and everyone knew about a few things like the Dual_EC backdoor, but there is a world of difference between some fun speculation and, say, being a Google datacenter architecture being shown a NSA PowerPoint presentation literally making fun of you for not encrypting your fiberoptic lines & letting the NSA harvest all your data. (It’s a bit like the horror people feel as they go from idle speculation in 201x about ‘hey, what if deep learning just kept going and began approaching AGI’ to playing with GPT-4 and realizing ‘oh my god, it’s actually happening’. They thought they lived in one world and not the other world, until the illusion shattered.) And this was, of course, in part because this let the NSA provide tons of intelligence that made it directly to the President’s daily briefing, which then gave the NSA budget and powers to get more intelligence, and so on, in a virtuous circle for them.
Meanwhile, of course, in that same American intelligence community at the same time, you have many, shall we say, less impressive things happening. (There is a wide range of competence.)
There are not many slots of attention to spend, and I strongly suspect that little attention has been spared for AI until recently. After all, how could the intelligence agencies take it seriously at the orders of their masters, when their masters still do not*, and even most AI people continue to strenuously downplay its importance? Stealing models of stochastic parrots is not a priority for anyone except ornithophiles. (It’s just a fad, it’s not real intelligence, there have been so many false alarms before, it’ll hit the wall soon, it’ll run out of data, I saw some messed-up hands in an image the other day, it’s a distraction from real issues like hardware, and aren’t OA user numbers falling anyway...?)
It’s become fashionable to claim that the NSA and/or CCP has surely already long ago hacked OpenAI (or Anthropic, or DeepMind—the claim is salted according to one’s taste) and stolen this or that (GPT-3, or GPT-4). I doubt these claims, as they are made without any evidence and have explicit ulterior motives for hyping up threats & justifying acceleration (and we’d see more impressive things out of Chinese DL if the CCP really done so). I’ve been hearing for many years now the notion that if the publicly-known DL is X% then there must be a secret government AGI project which is way better and does X+Y%—needless to say, at no point have we ever learned of such a thing having been true. (This also holds for various topics in genetics and a few other disruptive technologies; if one had a nickel for every time some commenter claimed that there was a government Manhattan Project for which there was zero evidence because it was secret, one could almost fund one’s own Manhattan Project.) If there have been serious hacking campaigns or serious ‘AGI Manhattan Projects’, they have probably started relatively recently and may be completely unsuccessful. But most of all, I doubt the idea that the NSA would’ve blasted through OA security, in the way it definitely could if they took off the gloves, because there is no sign that the Eye of Sauron has truly turned to AI and any intelligence agencies begun truly prioritizing AI-related industrial espionage, as opposed to continuing on the usual autonomous autopilot of various low-priority efforts and plucking low-hanging fruit. (Note that even the China chip ban was justified in considerable part by hypersonic missile & advanced aerospace military R&D, which are topics the Eye has been staring at.)
However, just because they have not yet doesn’t mean they won’t at some point. And when they do, the situation will alter radically—and outsiders may have no idea.
* See anything from Xi Jingping lately that indicates he gives a s—t about AI rather than doubling down on his reign’s classic-Marxist-style monomania of supply-side heavy industry & construction?
In an intelligence community context, the American spy satellites like the KH program achieved astonishing things in photography, physics, and rocketry—things like handling ultra-high-resolution photography in space (with its unique problems like disposing of hundreds of gallons of water in space) or scooping up landing satellites in helicopters were just the start. (I was skimming a book the other day which included some hilarious anecdotes—like American spies would go take tourist photos of themselves in places like Red Square just to assist trigonometry for photo analysis.) American presidents obsessed over the daily spy satellite reports, and this helped ensure that the spy satellite footage was worth obsessing over. (Amateurs fear the CIA, but pros fear NRO.)
The way I would put it is: there is a wide range of competence and organizational efficacy in any government, and in most of the ways in which you could slice up any government. They include many of the most and least competent people in the world. And governments have access to extraordinary levels of resources and powers. The richest billionaire in the world can scarcely spend annually what an obscure city you’ve never heard of might spend annually on healthcare or pensions.
But governments (and all large organizations, such as corporations) suffer from an extreme deficit of coordination and executive function (pun very intended). Highly-functional organizations cannot replicate themselves, cannot grow without losing functionality, cannot evolve etc. There are permanent productivity differences in businesses doing the exact same thing, because the net selection is too weak—organizations are barely able to maintain their status quo. Oversight and accountability are difficult, there are only a few competent managers or troubleshooters to go around, only a few isolated islands of competence which can be trusted, there’s always a crisis elsewhere as a distraction… A large organization cannot walk and chew bubblegum; leadership like CEOs exist mostly to (1) decide when to stop walking and start chewing bubblegum instead, and (2) yell at everyone that the goal is now ‘chewing bubblegum’ for as long as it takes for the vast lumbering giant to halt. They are much like humans in having a spotlight of conscious attention and a working memory with just a few slots, except that instead of the ‘magic number 7±2’, it’s a little more like ‘the not-so-magic number 0–2’; and then everything else carries on autonomously. Attempts to pay attention to more things just leads to accomplishing nothing, or disaster. (Imagine if you had to consciously control your breathing and heart rate and all the other things your autonomous nervous system does! not to mention the 99% of cognition which is inaccessible to consciousness.)
So, large organizations can often accomplish anything they want, but they cannot accomplish everything. They often fail on the meta-level of wanting to accomplish the right thing, and go off and accomplish the wrong things. Whereas cases of successful projects like the Manhattan Project/Project Nobska/Skunkworks/Apollo Project/Operation Warp Speed are cases where the large organization decides to accomplish the right thing: it makes the project genuinely a top priority, assigning the top people to it, and ensuring it is not held hostage to autonomous routine concerns or nice-to-have things like being environmentally-friendly. (The Eye of Sauron turns its baleful gaze away from the broader war and towards any sources of slowdown, burning through them.) When the Manhattan Project needed tons of highly-conductive metal like copper, they didn’t waste time requesting it or even seizing it from other military needs; they simply borrowed tons of silver from Fort Knox. (More balefully, they didn’t give a toss about ‘pollution’ or ‘safety’, and so the cleanup goes on to this day—costing OOMs more than the Manhattan Project itself ever did!)
When people talk about ‘We need a Manhattan Project’, what they are really saying is not ‘we need to spend $ $ $ on this’ (often $ $ $ has already been spent) but that the large-organization should burn 1 slot of attention on that goal, setting a hard endpoint with Consequences™.
When the Eye of Sauron does turn its gaze onto a specific project, the results are shocking.
In an intelligence community context, the American spy satellites like the KH program achieved astonishing things in photography, physics, and rocketry—things like handling ultra-high-resolution photography in space (with its unique problems like disposing of hundreds of gallons of water in space) or scooping up landing satellites in helicopters were just the start. (I was skimming a book the other day which included some hilarious anecdotes—like American spies would go take tourist photos of themselves in places like Red Square just to assist trigonometry for photo analysis.) American presidents obsessed over the daily spy satellite reports, and this helped ensure that the spy satellite footage was worth obsessing over. (Amateurs fear the CIA, but pros fear NRO.)
Or consider the NSA. Specifically, the Snowden leaks: systematic, well-maintained, engineered, comprehensive, extensive, utterly successfully kept secret hacks of everyone. At this point, probably a lot of LW2 readers weren’t ‘around’ for that, but as humdrum as they may seem now—‘oh sure, of course the NSA was spying on and hacking everyone’—the leaks were massively traumatic to the infosec sector because people just couldn’t imagine that basically all the medium-case scenarios were true simultaneously—the NSA really had backdoored your cipher, really was piggybacking on other state actors, really was keeping all the hacks successfully secret by extreme levels of OPSEC which is why hardly anyone had ever seen a NSA hack, really was recording all Internet traffic at some points, really was storing much traffic to crack later with quantum computers, really was intercepting your computer in the mail to add hardware backdoors, etc. They might’ve speculated idly about this or that and everyone knew about a few things like the Dual_EC backdoor, but there is a world of difference between some fun speculation and, say, being a Google datacenter architecture being shown a NSA PowerPoint presentation literally making fun of you for not encrypting your fiberoptic lines & letting the NSA harvest all your data. (It’s a bit like the horror people feel as they go from idle speculation in 201x about ‘hey, what if deep learning just kept going and began approaching AGI’ to playing with GPT-4 and realizing ‘oh my god, it’s actually happening’. They thought they lived in one world and not the other world, until the illusion shattered.) And this was, of course, in part because this let the NSA provide tons of intelligence that made it directly to the President’s daily briefing, which then gave the NSA budget and powers to get more intelligence, and so on, in a virtuous circle for them.
Meanwhile, of course, in that same American intelligence community at the same time, you have many, shall we say, less impressive things happening. (There is a wide range of competence.)
There are not many slots of attention to spend, and I strongly suspect that little attention has been spared for AI until recently. After all, how could the intelligence agencies take it seriously at the orders of their masters, when their masters still do not*, and even most AI people continue to strenuously downplay its importance? Stealing models of stochastic parrots is not a priority for anyone except ornithophiles. (It’s just a fad, it’s not real intelligence, there have been so many false alarms before, it’ll hit the wall soon, it’ll run out of data, I saw some messed-up hands in an image the other day, it’s a distraction from real issues like hardware, and aren’t OA user numbers falling anyway...?)
It’s become fashionable to claim that the NSA and/or CCP has surely already long ago hacked OpenAI (or Anthropic, or DeepMind—the claim is salted according to one’s taste) and stolen this or that (GPT-3, or GPT-4). I doubt these claims, as they are made without any evidence and have explicit ulterior motives for hyping up threats & justifying acceleration (and we’d see more impressive things out of Chinese DL if the CCP really done so). I’ve been hearing for many years now the notion that if the publicly-known DL is X% then there must be a secret government AGI project which is way better and does X+Y%—needless to say, at no point have we ever learned of such a thing having been true. (This also holds for various topics in genetics and a few other disruptive technologies; if one had a nickel for every time some commenter claimed that there was a government Manhattan Project for which there was zero evidence because it was secret, one could almost fund one’s own Manhattan Project.) If there have been serious hacking campaigns or serious ‘AGI Manhattan Projects’, they have probably started relatively recently and may be completely unsuccessful. But most of all, I doubt the idea that the NSA would’ve blasted through OA security, in the way it definitely could if they took off the gloves, because there is no sign that the Eye of Sauron has truly turned to AI and any intelligence agencies begun truly prioritizing AI-related industrial espionage, as opposed to continuing on the usual autonomous autopilot of various low-priority efforts and plucking low-hanging fruit. (Note that even the China chip ban was justified in considerable part by hypersonic missile & advanced aerospace military R&D, which are topics the Eye has been staring at.)
However, just because they have not yet doesn’t mean they won’t at some point. And when they do, the situation will alter radically—and outsiders may have no idea.
* See anything from Xi Jingping lately that indicates he gives a s—t about AI rather than doubling down on his reign’s classic-Marxist-style monomania of supply-side heavy industry & construction?
What is that book with the fun anecdotes?