I wonder if you could flesh out your intuitions for avoiding political solutions to problems point a bit more. The first x-risk was nuclear war, which has a technical dimension but is fundamentally a problem of international relations. It also appears to be the most successfully managed x-risk so far, in the sense that it has been an extant threat for 50 years without going off.
I certainly agree we should not content ourselves with an AI ban in lieu of technical progress, but if we use politics in the broad sense of including institutional behavior and conflict I feel this dimension of the problem is currently neglected.
The mere fact that an x-risk hasn’t occured is not evidence that it has been well managed, because that’s the only possible state you could observe (if it wasn’t true you wouldn’t be around). Then again nuclear war is a GCR, so the anthropics might not be that bad.
On another note, if the nuclear situation is what it looks like when humanity “manages” an x-risk, I think we’re in a pretty dire state...
Is there consensus on this? My opinion is also that anthropic effects imply that nuclear war hasn’t necessarily been well-managed (reading stories like Petrov’s it seems like dumb luck has been more important than good institutional management) but my impression is that people are far from universally accepting enough of anthropic reasoning to buy this in general.
Well-managed and best-managed aren’t necessarily the same thing. The fact remains that the nuclear problem absorbed huge amounts of intellectual effort, spurred the development of whole fields expressly for its control, and has been a consistent, global insitutional priority.
The trouble is that no other x-risk has really been managed at all, although we are clearly moving in that direction for the climate. Any management vs. zero management → best management.
The fact that—unlike the case of the nuclear war where the quality of the threat was visible to politicians and the public alike—alignment seems to be a problem which not even all AI researchers understand is worth mentioning. That in itself probably excludes the possibility of a direct political solution. But even politics in the narrow sense can be utilized with a bit of creativity (e.g. by providing politicians a motivation more direct than saving the world, grounded on things they can understand without believing weird-sounding claims of cultish-looking folks).
unlike the case of the nuclear war where the quality of the threat was visible to politicians and the public alike—alignment seems to be a problem which not even all AI researchers understand is worth mentioning. That in itself probably excludes the possibility of a direct political solution.
The failure to recognise/understand/appreciate the problem does seem an important factor. And if it were utterly unchangeable, maybe that would mean all efforts need to just go towards technical solutions. But it’s not utterly unchangeable; in fact, it’s a key variable which “political” (or just “not purely technical”) efforts could intervene on to reduce AI x-risk.
E.g., a lot of EA movement building, outreach by AI safety researchers, Stuart Russell’s book Human-Compatible, etc., is partly targeting at getting more AI researchers (and/or the broader public or certain elites like politicians) to recognise/understand/appreciate the problem. And by doing so, it could have other benefits like increasing the amount of technical work on AI safety, influencing policies that reduce risks of different AI groups rushing to the finish line and compromising on safety, etc.
I think this was actually somewhat similar in the case of nuclear war. Of course, the basic fact that nuclear weapons could be very harmful is a lot more obvious than the fact AGI/superintelligence could be very harmful. But the main x-risk from nuclear war is nuclear winter, and that’s not immediately obvious—it requires some quite modelling, and is something unlike things people have seen in their lifetimes, really. And according to Toby Ord in The Precipice (page 65):
The discovery that atomic weapons may trigger a nuclear winter influenced both Ronald Reagan and Mikhail Gorbachev to reduce their country’s arms to avoid war.
So in that case, technical work on understanding the problem was communicated to politicians (this communication being a non-technical intervention), and helped make the potential harms clearer to politicians, which helped lead to a political (partial) solution.
Basically, I think that technical and non-technical interventions are often intertwined or support each other, and that we should see current levels of recognition that AI risk is a big deal as something we can and should intervene to change, not as something fixed.
I certainly agree we should not content ourselves with an AI ban in lieu of technical progress
Why not? An AI ban isn’t politically possible, but if it was enacted and enforced, I’d expect it to be effective at preventing risks from unaligned AI.
I’ve heard before that an argument against banning AI research (even if you could do such a thing) is that hardware will continue to improve. This is bad because it enables less technically abled parties to weild supercomputer-level AI developments. It’s better that a single company stays ahead in the race than the remote possibility that someone can create a seed-AI in their basement.
I argue this is not enforceable in any meaningful sense. Returning to the nuclear weapons example, there are large industrial facilities and logistical footprints which are required. These can be tracked and targeted for enforcement. By contrast, computers and mathematics are cheap, ubiquitous, and you cannot have a modern civilization without them. As secret projects go, AI would be trivial to conceal. The best we could do is enforce a publishing ban—but stopping the flow of any kind of information is a very expensive task, and we could not confidently say the risk is mitigated, only delayed. Further, voluntary compliance would only mean ceding the first-mover advantage to institutions which are already less concerned with issues like ethics.
I would expect attempts to ban AI research to make it marginally less likely to appear, and much less likely to be aligned if it does. Not a net gain.
I wonder if you could flesh out your intuitions for avoiding political solutions to problems point a bit more. The first x-risk was nuclear war, which has a technical dimension but is fundamentally a problem of international relations. It also appears to be the most successfully managed x-risk so far, in the sense that it has been an extant threat for 50 years without going off.
I certainly agree we should not content ourselves with an AI ban in lieu of technical progress, but if we use politics in the broad sense of including institutional behavior and conflict I feel this dimension of the problem is currently neglected.
The mere fact that an x-risk hasn’t occured is not evidence that it has been well managed, because that’s the only possible state you could observe (if it wasn’t true you wouldn’t be around). Then again nuclear war is a GCR, so the anthropics might not be that bad.
On another note, if the nuclear situation is what it looks like when humanity “manages” an x-risk, I think we’re in a pretty dire state...
Is there consensus on this? My opinion is also that anthropic effects imply that nuclear war hasn’t necessarily been well-managed (reading stories like Petrov’s it seems like dumb luck has been more important than good institutional management) but my impression is that people are far from universally accepting enough of anthropic reasoning to buy this in general.
Well-managed and best-managed aren’t necessarily the same thing. The fact remains that the nuclear problem absorbed huge amounts of intellectual effort, spurred the development of whole fields expressly for its control, and has been a consistent, global insitutional priority.
The trouble is that no other x-risk has really been managed at all, although we are clearly moving in that direction for the climate. Any management vs. zero management → best management.
The fact that—unlike the case of the nuclear war where the quality of the threat was visible to politicians and the public alike—alignment seems to be a problem which not even all AI researchers understand is worth mentioning. That in itself probably excludes the possibility of a direct political solution. But even politics in the narrow sense can be utilized with a bit of creativity (e.g. by providing politicians a motivation more direct than saving the world, grounded on things they can understand without believing weird-sounding claims of cultish-looking folks).
(Very late to this thread)
The failure to recognise/understand/appreciate the problem does seem an important factor. And if it were utterly unchangeable, maybe that would mean all efforts need to just go towards technical solutions. But it’s not utterly unchangeable; in fact, it’s a key variable which “political” (or just “not purely technical”) efforts could intervene on to reduce AI x-risk.
E.g., a lot of EA movement building, outreach by AI safety researchers, Stuart Russell’s book Human-Compatible, etc., is partly targeting at getting more AI researchers (and/or the broader public or certain elites like politicians) to recognise/understand/appreciate the problem. And by doing so, it could have other benefits like increasing the amount of technical work on AI safety, influencing policies that reduce risks of different AI groups rushing to the finish line and compromising on safety, etc.
I think this was actually somewhat similar in the case of nuclear war. Of course, the basic fact that nuclear weapons could be very harmful is a lot more obvious than the fact AGI/superintelligence could be very harmful. But the main x-risk from nuclear war is nuclear winter, and that’s not immediately obvious—it requires some quite modelling, and is something unlike things people have seen in their lifetimes, really. And according to Toby Ord in The Precipice (page 65):
So in that case, technical work on understanding the problem was communicated to politicians (this communication being a non-technical intervention), and helped make the potential harms clearer to politicians, which helped lead to a political (partial) solution.
Basically, I think that technical and non-technical interventions are often intertwined or support each other, and that we should see current levels of recognition that AI risk is a big deal as something we can and should intervene to change, not as something fixed.
Why not? An AI ban isn’t politically possible, but if it was enacted and enforced, I’d expect it to be effective at preventing risks from unaligned AI.
I’ve heard before that an argument against banning AI research (even if you could do such a thing) is that hardware will continue to improve. This is bad because it enables less technically abled parties to weild supercomputer-level AI developments. It’s better that a single company stays ahead in the race than the remote possibility that someone can create a seed-AI in their basement.
I argue this is not enforceable in any meaningful sense. Returning to the nuclear weapons example, there are large industrial facilities and logistical footprints which are required. These can be tracked and targeted for enforcement. By contrast, computers and mathematics are cheap, ubiquitous, and you cannot have a modern civilization without them. As secret projects go, AI would be trivial to conceal. The best we could do is enforce a publishing ban—but stopping the flow of any kind of information is a very expensive task, and we could not confidently say the risk is mitigated, only delayed. Further, voluntary compliance would only mean ceding the first-mover advantage to institutions which are already less concerned with issues like ethics.
I would expect attempts to ban AI research to make it marginally less likely to appear, and much less likely to be aligned if it does. Not a net gain.