There seems to be a fair amount of motivated reasoning with denying China’s AI capabilities when they’re basically neck and neck with the U.S. (their chatbots, video bots, social media algorithms, and self driving cars are as roughly good as or better than ours).
I think a lot of policy approaches fail within an AGI singleton race paradigm. It’s also clear a lot of EA policy efforts are basically in denial that this is already starting to happen.
I’m glad Leopold Aschenbrenner spelled out the uncomfortable but remarkably obvious truth for us. China is gearing to wage war for silicon. The U.S. is slowly tightening their control over silicon. This reads exactly like a race.
I personally think it’s futile to try to stop this and our best bet is to solve alignment as fast as possible with a Manhattan project scale effort.
Without access to hardware, further scaling will be a problem. GPT-4 level models don’t need that much hardware, but this changes when you scale by another 30 or 1000 times. Fabs take a long time to get into production even when the tools they need are available. With whatever fabs there are, you still need chip designs. And a few years is forever in AI time.
I think Leopold addresses this but 5% of our compute will be used to make a hypothetical AGI while China can direct 100% of their compute. They can make up in quality with quantity and they also happen to have far more energy than us which is probably the more salient variable in the AGI equation.
Also I’m of the opinion that the GPU bans are largely symbolic. There is little incentive to respect them, especially when China realizes stakes are higher than they seem now. In fact they are largely symbolic now.
That shows the opposite. Purchases of 1 or 6 A100s, in an era where the SOTA is going to take 100,000+ B100s, 2 generations later, are totally irrelevant and downright symbolic.
I mean are you sure Singapore’s sudden large increase in GPU purchases is organic? GPU bans have very obviously not stopped Chinese AI progress, so I think we should build conclusions starting from there instead of the reverse order.
I also think US GPU superiority is short lived. China can skip engineering milestones we’ve had to pass, exploit the fact that they have far more energy than us, skip the general purpose computing/gaming tech debt that may exist in current GPUs, etc.
EDIT: This is selective rationalism. If you sought any evidence in this issue, it would become extremely obvious that Singapore’s orders of H100s magically increased by many magnitudes after they were banned in China.
Just want to register that I agree that—regardless of US GPU superiority right now—the US AI superiority is pretty small, and decreasing. Yi-Large beats a bunch of GPT-4 versions—even in English—on lmsys; it scores just above stuff like Gemini. Their open source releases like DeepSeekV2 look like ~Llama 3 70b level. And so on and so forth.
Maybe whatever OpenAI is training now will destroy whatever China has, and establish OpenAI as firmly in the lead.… or maybe not. Yi says they’re training their next model as well, so it isn’t like they’ve stopped doing things.
I think some chunk of “China is so far behind” is fueled by the desire to be able to stop US labs while not just letting China catch up, but that is what it would actually do.
Manhattan project was primarily an engineering effort, with all necessary science established before. Trying to solve alignment now with such project is like starting Manhattan project in 1900.
There was more theory laid out and theory discovered in the process but I think more importantly there were just a lot of approaches to try. I don’t think your analogy fits best. The alignment Manhattan project to me would be to scale up existing mech-interp work 1000x and try every single alignment idea under the sun simultaneously with the goal of automating it once we’re confident of human level systems. Can you explain more of where your analogy works and what would break the above?
Manhattan project was based on information with 99%+ of certainty that fission chain reaction for uranium is possible and it is producing large amount of energy. The problem was to cause simultaneously large number of fission chain reactions so amount of energy produced is enough to cause large explosion. If you have this definition of the problem, you have nice possible solution space which you can explore more-or-less methodically and expect result.
I don’t think you can present the same nice definition for alignment.
I think the real analogy for alignment is not Manhattan project but “how to successfuly make first nuclear strike given that the enemy has detection system and nuclear ICBM too”.
The Manhattan project had elements where they were worried they’d end the world through atmospheric chain reactions (but this wasn’t taken too seriously). The scientists on this project considered MAD and nuclear catastrophes were considered as plausible outcomes. Many had existential dread. I think it actually maps out well, since you are uncertain how likely a nuclear exchange is, but you could easily say there is a high chance of it happening, just like you can easily now say with some level of uncertainty that p(doom) is high.
I think the real analogy for alignment is not Manhattan project but “how to successfully make first nuclear strike given that the enemy has detection system and nuclear ICBM too”.
This requires the planners to be completely convinced that p(doom) is high (as in self immolation and not Russian roulette where 5⁄6 bullets lead to eternal prosperity). The odds of a retaliatory strike or war vs the USSR on the other hand at any given point was 100%. The US’s nuclear advantage at no point was overwhelming enough outside of Japan where we did use it. The fact that a first-strike against the USSR was never pursued is evidence of this. Think of the USSR instead being in Iran’s relative position today. If Iran tried to build thousands of nukes today and it looked like they would succeed, we’d definitely see a first strike or a hot war.
So alignment isn’t like this, there is a non trivial chance that even RLHF just happens to scale to super intelligence. After 20 years, MIRI, nor anyone can prove nor disprove this, and that’s enough reason to try to do it anyways, just like how nuclear might inevitably lead to the nations with the nukes to engage in an exchange, but they were built anyways. And unlike nuclear, the upside of ASI being aligned is practically infinite. In the first strike scenario, it’s a definite severe downside to preventing a potentially more severe downside in the future.
Centralized organizations don’t tend to be able to “try every single idea” if you have resources spread out over different organizations, more different kind of ideas are usually tried.
Don’t see how this is relevant to my broader point. But the Manhattan project was essentially try every research direction instead of picking and choosing to reduce experimentation time.
There seems to be a fair amount of motivated reasoning with denying China’s AI capabilities when they’re basically neck and neck with the U.S. (their chatbots, video bots, social media algorithms, and self driving cars are as roughly good as or better than ours).
I think a lot of policy approaches fail within an AGI singleton race paradigm. It’s also clear a lot of EA policy efforts are basically in denial that this is already starting to happen.
I’m glad Leopold Aschenbrenner spelled out the uncomfortable but remarkably obvious truth for us. China is gearing to wage war for silicon. The U.S. is slowly tightening their control over silicon. This reads exactly like a race.
I personally think it’s futile to try to stop this and our best bet is to solve alignment as fast as possible with a Manhattan project scale effort.
Without access to hardware, further scaling will be a problem. GPT-4 level models don’t need that much hardware, but this changes when you scale by another 30 or 1000 times. Fabs take a long time to get into production even when the tools they need are available. With whatever fabs there are, you still need chip designs. And a few years is forever in AI time.
I think Leopold addresses this but 5% of our compute will be used to make a hypothetical AGI while China can direct 100% of their compute. They can make up in quality with quantity and they also happen to have far more energy than us which is probably the more salient variable in the AGI equation.
Also I’m of the opinion that the GPU bans are largely symbolic. There is little incentive to respect them, especially when China realizes stakes are higher than they seem now. In fact they are largely symbolic now.
That shows the opposite. Purchases of 1 or 6 A100s, in an era where the SOTA is going to take 100,000+ B100s, 2 generations later, are totally irrelevant and downright symbolic.
I mean are you sure Singapore’s sudden large increase in GPU purchases is organic? GPU bans have very obviously not stopped Chinese AI progress, so I think we should build conclusions starting from there instead of the reverse order.
I also think US GPU superiority is short lived. China can skip engineering milestones we’ve had to pass, exploit the fact that they have far more energy than us, skip the general purpose computing/gaming tech debt that may exist in current GPUs, etc.
EDIT: This is selective rationalism. If you sought any evidence in this issue, it would become extremely obvious that Singapore’s orders of H100s magically increased by many magnitudes after they were banned in China.
Just want to register that I agree that—regardless of US GPU superiority right now—the US AI superiority is pretty small, and decreasing. Yi-Large beats a bunch of GPT-4 versions—even in English—on lmsys; it scores just above stuff like Gemini. Their open source releases like DeepSeekV2 look like ~Llama 3 70b level. And so on and so forth.
Maybe whatever OpenAI is training now will destroy whatever China has, and establish OpenAI as firmly in the lead.… or maybe not. Yi says they’re training their next model as well, so it isn’t like they’ve stopped doing things.
I think some chunk of “China is so far behind” is fueled by the desire to be able to stop US labs while not just letting China catch up, but that is what it would actually do.
Manhattan project was primarily an engineering effort, with all necessary science established before. Trying to solve alignment now with such project is like starting Manhattan project in 1900.
There was more theory laid out and theory discovered in the process but I think more importantly there were just a lot of approaches to try. I don’t think your analogy fits best. The alignment Manhattan project to me would be to scale up existing mech-interp work 1000x and try every single alignment idea under the sun simultaneously with the goal of automating it once we’re confident of human level systems. Can you explain more of where your analogy works and what would break the above?
Manhattan project was based on information with 99%+ of certainty that fission chain reaction for uranium is possible and it is producing large amount of energy. The problem was to cause simultaneously large number of fission chain reactions so amount of energy produced is enough to cause large explosion. If you have this definition of the problem, you have nice possible solution space which you can explore more-or-less methodically and expect result.
I don’t think you can present the same nice definition for alignment.
I think the real analogy for alignment is not Manhattan project but “how to successfuly make first nuclear strike given that the enemy has detection system and nuclear ICBM too”.
The Manhattan project had elements where they were worried they’d end the world through atmospheric chain reactions (but this wasn’t taken too seriously). The scientists on this project considered MAD and nuclear catastrophes were considered as plausible outcomes. Many had existential dread. I think it actually maps out well, since you are uncertain how likely a nuclear exchange is, but you could easily say there is a high chance of it happening, just like you can easily now say with some level of uncertainty that p(doom) is high.
This requires the planners to be completely convinced that p(doom) is high (as in self immolation and not Russian roulette where 5⁄6 bullets lead to eternal prosperity). The odds of a retaliatory strike or war vs the USSR on the other hand at any given point was 100%. The US’s nuclear advantage at no point was overwhelming enough outside of Japan where we did use it. The fact that a first-strike against the USSR was never pursued is evidence of this. Think of the USSR instead being in Iran’s relative position today. If Iran tried to build thousands of nukes today and it looked like they would succeed, we’d definitely see a first strike or a hot war.
So alignment isn’t like this, there is a non trivial chance that even RLHF just happens to scale to super intelligence. After 20 years, MIRI, nor anyone can prove nor disprove this, and that’s enough reason to try to do it anyways, just like how nuclear might inevitably lead to the nations with the nukes to engage in an exchange, but they were built anyways. And unlike nuclear, the upside of ASI being aligned is practically infinite. In the first strike scenario, it’s a definite severe downside to preventing a potentially more severe downside in the future.
Centralized organizations don’t tend to be able to “try every single idea” if you have resources spread out over different organizations, more different kind of ideas are usually tried.
Don’t see how this is relevant to my broader point. But the Manhattan project was essentially try every research direction instead of picking and choosing to reduce experimentation time.