Iran is an agent, with a constrained amount of critical resources like nuclear engineers, centrifuges, etc.
AI development is a robust, agent-agnostic process that has an unlimited number of researchers working in adjacent areas who could easily cross-train to fill a deficit, an unlimited number of labs which would hire researchers from DeepMind and OpenAI if they closed, and an unlimited amount of GPUs to apply to the problem.
Probably efforts at getting the second-tier AI labs to take safety more seriously, in order to give the top tier more slack, will move back AI timelines a little? But most of the activities that my brain labels “nonviolent resistance” are the type that will be counterproductive unless there’s already a large social movement behind them.
Iran is a country, not an agent. Important distinction and I’m not being pedantic here. Iran’s aggressive military stance towards Israel is not quite the result of a robust, agent-agnostic process but it’s not the result of a single person optimizing for some goal either.
with a constrained amount of critical resources like nuclear engineers, centrifuges, etc. AI development is a robust, agent-agnostic process that has an unlimited number of researchers working in adjacent areas who could easily cross-train to fill a deficit...and an unlimited amount of GPUs to apply to the problem.
It’s important to recognize there are not actually an unlimited amount of GPUs, labs, dollars, or talented enough people available to allocate toward AGI development. People in particular is the sticking point for me here; there’s literally a highly constrained amount with the intelligence necessary to do and publish high level productive work in this kind of arena. It’s this reason that the second-tier labs are already an order of magnitude less effective than DeepMind. Maybe Oppenheimer or John von Neumann were unimportant to atomic bomb development timelines because they would just be replaced by someone else in the event that they found out it was going to cause the atmosphere to light up, but I doubt it.
This model treats the “product” DeepMind, Meta, and OpenAI generate in publishing research as having some static underlying driving propulsion like “market demand”. I don’t think there is actual market demand, in the monetary sense. From an shareholder’s point of view spending on arxiv research is mostly wasted money. There might be “demand” for AGI research in the wider sense that there are “charitable” people who like to fund long term scientific research, but that kind of demand is mostly driven by the starry eyed and often waning passions of higher ups at megacorps. A large number of people defecting or leaving for ethical reasons would affect that demand and start to encourage them to see their funding differently.
...an unlimited number of labs which would hire researchers from DeepMind and OpenAI if they closed...
The point of this strategy is to convince people not to work in AIG-adjacent research labs at all, or retrofit existing organizations into doing something else, not to get them to quit some specific organization.
Google, Microsoft, and Facebook are all developing AI. So are Tencent, Sberbank, and the NSA. So are thousands of startups.
The most important thing is to solve the real problem—to design AI which, even when autonomous and smarter than human, has “friendly” values. (Just to be concrete, 22:30 here gives an example of what such a design could look like, albeit while still at a very theoretical and schematic stage.)
So you can try to speed up AI alignment research, and you can also try to slow down unaligned AI research. But there are thousands of independent research centers doing unaligned AI. OK, in the present, maybe there’s just a handful which are the ones at immediate risk of creating unaligned, autonomous, smarter-than-human AI. They’re mostly in huge powerful organizations, and there would be secret projects at the same level that you don’t know about—but OK, maybe you can make an impact somehow, by sabotaging TPU production, or by getting the UNSC to work with NeurIPS on a global regulatory regime, or…
You know, it’s going to be hard to stop these organizations from reaching for ultimate power. Putin already said a few years back, the nation that controls AI, controls the world. But also, the level of computing power and software knowledge required to become an immediate risk, gets lower over time. Maybe you can achieve a temporary tactical success through these preventive interventions, but for a primarily preventive strategy to work, you would have to change the entire direction of technological culture in this multipolar world. Also, the work that we already know about, from the big organizations we know about, is already alarmingly close to the point of no return. Proliferation will keep happening, but maybe we already know the name of the organization that will be the architect of our fate, we just don’t know which it shall be.
On the other hand: if you can solve the problem of friendliness, the problem of alignment, in practice and not just in theory (meaning you have enough computational resources to actually run your actually aligned AI program), then that’s it. Thus the focus on solving alignment rather than preventing nonalignment.
If we’re in a situation where it’s an open secret that a certain specific research area leads to general artificial intelligence, we’re doomed. If we get into a position where compute is the only limiting factor, we’re doomed. There’s no arguing there. The goal is to prevent us from getting into that situation.
As it stands now, certainly lots of companies are practicing machine learning. I have secondhand descriptions of a lot of the “really advanced” NSA programs and they fit that bill. Not a lot of organizations I know of however are actually pushing the clock hand forward on AGI and meta. Even fewer are doing that consistently, or getting over the massive intellectual hurdles that require a team like DeepMind’s. Completing AGI will probably be the result of a marriage of increased computing power, which we can’t really control, and insights pioneered and published by top labs, which I legitimately think we could to some degree by modifying their goals and talking to their members. OpenAI is a nonprofit. At the absolute bare minimum, none of these companies’ publish their meta research for money. The worst things they seem to do at this stage aren’t achieved when they’re reaching for power so much as playing an intellectual & status game amongst themselves, and fulfilling their science fiction protagonist syndromes.
I don’t doubt that it would be better for us to have AI alignment solved than to rely on these speculations about how AI will be engineered, but I do not see any good argument as to why it’s a bad strategy.
If we were “doomed” in this way, would you agree that the thing to do—for those who could do it—is to keep trying to solve the problem of alignment? i.e. trying to identify an AI design that could be autonomous, and smarter than human, and yet still safe?
Let me articulate my intuitions in a little bit more of a refined way: “If we ever get to a point where there are few secrets left, or that it’s common knowledge one can solve AGI with ~1000-10,000 million dollars, then delaying tactics probably wouldn’t work, because there’s nothing left for DeepMind to publish that speeds up the timeline.”
Inside those bounds, yes. I still think that people should keep working on alignment today, I just think other dumber people like me should try the delaying tactics I articulated in addition to funding alignment research.
Iran is an agent, with a constrained amount of critical resources like nuclear engineers, centrifuges, etc.
AI development is a robust, agent-agnostic process that has an unlimited number of researchers working in adjacent areas who could easily cross-train to fill a deficit, an unlimited number of labs which would hire researchers from DeepMind and OpenAI if they closed, and an unlimited amount of GPUs to apply to the problem.
Probably efforts at getting the second-tier AI labs to take safety more seriously, in order to give the top tier more slack, will move back AI timelines a little? But most of the activities that my brain labels “nonviolent resistance” are the type that will be counterproductive unless there’s already a large social movement behind them.
Iran is a country, not an agent. Important distinction and I’m not being pedantic here. Iran’s aggressive military stance towards Israel is not quite the result of a robust, agent-agnostic process but it’s not the result of a single person optimizing for some goal either.
It’s important to recognize there are not actually an unlimited amount of GPUs, labs, dollars, or talented enough people available to allocate toward AGI development. People in particular is the sticking point for me here; there’s literally a highly constrained amount with the intelligence necessary to do and publish high level productive work in this kind of arena. It’s this reason that the second-tier labs are already an order of magnitude less effective than DeepMind. Maybe Oppenheimer or John von Neumann were unimportant to atomic bomb development timelines because they would just be replaced by someone else in the event that they found out it was going to cause the atmosphere to light up, but I doubt it.
This model treats the “product” DeepMind, Meta, and OpenAI generate in publishing research as having some static underlying driving propulsion like “market demand”. I don’t think there is actual market demand, in the monetary sense. From an shareholder’s point of view spending on arxiv research is mostly wasted money. There might be “demand” for AGI research in the wider sense that there are “charitable” people who like to fund long term scientific research, but that kind of demand is mostly driven by the starry eyed and often waning passions of higher ups at megacorps. A large number of people defecting or leaving for ethical reasons would affect that demand and start to encourage them to see their funding differently.
The point of this strategy is to convince people not to work in AIG-adjacent research labs at all, or retrofit existing organizations into doing something else, not to get them to quit some specific organization.
Google, Microsoft, and Facebook are all developing AI. So are Tencent, Sberbank, and the NSA. So are thousands of startups.
The most important thing is to solve the real problem—to design AI which, even when autonomous and smarter than human, has “friendly” values. (Just to be concrete, 22:30 here gives an example of what such a design could look like, albeit while still at a very theoretical and schematic stage.)
So you can try to speed up AI alignment research, and you can also try to slow down unaligned AI research. But there are thousands of independent research centers doing unaligned AI. OK, in the present, maybe there’s just a handful which are the ones at immediate risk of creating unaligned, autonomous, smarter-than-human AI. They’re mostly in huge powerful organizations, and there would be secret projects at the same level that you don’t know about—but OK, maybe you can make an impact somehow, by sabotaging TPU production, or by getting the UNSC to work with NeurIPS on a global regulatory regime, or…
You know, it’s going to be hard to stop these organizations from reaching for ultimate power. Putin already said a few years back, the nation that controls AI, controls the world. But also, the level of computing power and software knowledge required to become an immediate risk, gets lower over time. Maybe you can achieve a temporary tactical success through these preventive interventions, but for a primarily preventive strategy to work, you would have to change the entire direction of technological culture in this multipolar world. Also, the work that we already know about, from the big organizations we know about, is already alarmingly close to the point of no return. Proliferation will keep happening, but maybe we already know the name of the organization that will be the architect of our fate, we just don’t know which it shall be.
On the other hand: if you can solve the problem of friendliness, the problem of alignment, in practice and not just in theory (meaning you have enough computational resources to actually run your actually aligned AI program), then that’s it. Thus the focus on solving alignment rather than preventing nonalignment.
If we’re in a situation where it’s an open secret that a certain specific research area leads to general artificial intelligence, we’re doomed. If we get into a position where compute is the only limiting factor, we’re doomed. There’s no arguing there. The goal is to prevent us from getting into that situation.
As it stands now, certainly lots of companies are practicing machine learning. I have secondhand descriptions of a lot of the “really advanced” NSA programs and they fit that bill. Not a lot of organizations I know of however are actually pushing the clock hand forward on AGI and meta. Even fewer are doing that consistently, or getting over the massive intellectual hurdles that require a team like DeepMind’s. Completing AGI will probably be the result of a marriage of increased computing power, which we can’t really control, and insights pioneered and published by top labs, which I legitimately think we could to some degree by modifying their goals and talking to their members. OpenAI is a nonprofit. At the absolute bare minimum, none of these companies’ publish their meta research for money. The worst things they seem to do at this stage aren’t achieved when they’re reaching for power so much as playing an intellectual & status game amongst themselves, and fulfilling their science fiction protagonist syndromes.
I don’t doubt that it would be better for us to have AI alignment solved than to rely on these speculations about how AI will be engineered, but I do not see any good argument as to why it’s a bad strategy.
If we were “doomed” in this way, would you agree that the thing to do—for those who could do it—is to keep trying to solve the problem of alignment? i.e. trying to identify an AI design that could be autonomous, and smarter than human, and yet still safe?
Let me articulate my intuitions in a little bit more of a refined way: “If we ever get to a point where there are few secrets left, or that it’s common knowledge one can solve AGI with ~1000-10,000 million dollars, then delaying tactics probably wouldn’t work, because there’s nothing left for DeepMind to publish that speeds up the timeline.”
Inside those bounds, yes. I still think that people should keep working on alignment today, I just think other dumber people like me should try the delaying tactics I articulated in addition to funding alignment research.