My main point of disagreement is that you point to successful coordination in things like not eating sand, or not wearing weird clothing. The upside of these things is limited, but you say the upside of superintelligence is also limited because it could kill us.
But rephrase the question to “Should we create an AI that’s 1% better than the current best AI?” Most of the time this goes well—you get prettier artwork or better protein folding prediction, and it doesn’t kill you. So there’s strong upside to building slightly better AIs, as long as you don’t cross the “kills everyone” level. Which nobody knows the location of. And which (LW conventional wisdom says) most people will be wrong about.
We successfully coordinate a halt to AI advancement at the first point where more than half of the relevant coordination power agrees that the next 1% step forward is in expectation bad rather than good. But “relevant” is a tough qualifier, because if 99 labs think it’s bad, and one lab thinks it’s good, then unless there’s some centralizing force, the one lab can go ahead and take the step. So “half the relevant coordination power” has to include either every lab agreeing on which 1% step is bad, or the agreement of lots of governments, professional organizations, or other groups that have the power to stop the single most reckless lab.
I think it’s possible that we make this work, and worth trying, but that the most likely scenario is that most people underestimate the risk from AI, and so we don’t get half the relevant coordination power united around stopping the 1% step that actually creates dangerous superintelligence—which at the time will look to most people like just building a mildly better chatbot with many great social returns.
Agreed. My main objection to the post is that it considers the involved agents to be optimizing for far future world-states. But I’d say that most people (including academics and AI lab researchers) mostly only think of the next 1% step in front of their nose. The entire game theoretic framing in the arms race etc section seems wrong to me.
This seems to suggest “should we relax nuclear power regulation 1% less expensive to comply?” as a promising way to fix economics of nuclear power, and I don’t buy that at all. Maybe it’s different because Chernobyl happened, and the movie like The China Syndrome was made about nuclear accident?
That sounds very hopeful to me but doesn’t seem true to me. It implies slowing down AI will be easy, it just needs Chernobyl-sized disaster and a good movie about it. Chernobyl disaster was nearly harmless compared to COVID-19, and even COVID-19 was hardly an existential threat. If slowing down AI is this easy we probably shouldn’t waste time worrying about it before Chernobyl.
The difference between regulation and research is that the former has a large amount of friction, making it about as hard to push a 1% regulation through as a 10% one.
In contrast, the incremental 1% improvements in the development of capabilities is just what happens by default, as research organizations follow their charter.
Thank you, this is a good post.
My main point of disagreement is that you point to successful coordination in things like not eating sand, or not wearing weird clothing. The upside of these things is limited, but you say the upside of superintelligence is also limited because it could kill us.
But rephrase the question to “Should we create an AI that’s 1% better than the current best AI?” Most of the time this goes well—you get prettier artwork or better protein folding prediction, and it doesn’t kill you. So there’s strong upside to building slightly better AIs, as long as you don’t cross the “kills everyone” level. Which nobody knows the location of. And which (LW conventional wisdom says) most people will be wrong about.
We successfully coordinate a halt to AI advancement at the first point where more than half of the relevant coordination power agrees that the next 1% step forward is in expectation bad rather than good. But “relevant” is a tough qualifier, because if 99 labs think it’s bad, and one lab thinks it’s good, then unless there’s some centralizing force, the one lab can go ahead and take the step. So “half the relevant coordination power” has to include either every lab agreeing on which 1% step is bad, or the agreement of lots of governments, professional organizations, or other groups that have the power to stop the single most reckless lab.
I think it’s possible that we make this work, and worth trying, but that the most likely scenario is that most people underestimate the risk from AI, and so we don’t get half the relevant coordination power united around stopping the 1% step that actually creates dangerous superintelligence—which at the time will look to most people like just building a mildly better chatbot with many great social returns.
I loved the link to the “Resisted Technological Temptations Project”, for a bunch of examples of resisted/slowed technologies that are not “eating sand”, and have an enormous upside: https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:start
GMOs, in some countries
Nuclear power, in some countries
Genetic engineering of humans
Geoengineering, many actors
Chlorofluorocarbons, many actors, 1985-present
Human challenge trials
Dietary restrictions, in most (all?) human cultures [restrict much more than sand, often quite good stuff!]
I would tentatively add:
organ donor markets (at least for kidneys)
drug development in general (see all of Scott’s posts on the FDA slowing things down, I would love to see an AIA slowing things down)
Agreed. My main objection to the post is that it considers the involved agents to be optimizing for far future world-states. But I’d say that most people (including academics and AI lab researchers) mostly only think of the next 1% step in front of their nose. The entire game theoretic framing in the arms race etc section seems wrong to me.
This seems to suggest “should we relax nuclear power regulation 1% less expensive to comply?” as a promising way to fix economics of nuclear power, and I don’t buy that at all. Maybe it’s different because Chernobyl happened, and the movie like The China Syndrome was made about nuclear accident?
That sounds very hopeful to me but doesn’t seem true to me. It implies slowing down AI will be easy, it just needs Chernobyl-sized disaster and a good movie about it. Chernobyl disaster was nearly harmless compared to COVID-19, and even COVID-19 was hardly an existential threat. If slowing down AI is this easy we probably shouldn’t waste time worrying about it before Chernobyl.
The difference between regulation and research is that the former has a large amount of friction, making it about as hard to push a 1% regulation through as a 10% one.
In contrast, the incremental 1% improvements in the development of capabilities is just what happens by default, as research organizations follow their charter.