From a Facebook discussion with Scott Aaronson yesterday:
Yann: I think neither Yoshua nor Geoff believe that AI is going kill us all with any significant probability.
Scott: Well, Yoshua signed the pause letter, and wrote an accompanying statement about what he sees as the risk to civilization (I agree that there are many civilizational risks short of extinction). In his words: “No one, not even the leading AI experts, including those who developed these giant AI models, can be absolutely certain that such powerful tools now or in the future cannot be used in ways that would be catastrophic to society.”
Geoff said in a widely-shared recent video that it’s “not inconceivable” that AI will wipe out humanity, and didn’t offer any reassurances about it being vanishingly unlikely.
Yann: Scott Aaronson he is worried about catastrophic disruptions of the political, economic, and environmental systems.
I don’t want to speak for him, but I doubt he worries about a Yuddite-style uncontrollable “hard takeoff”
On the one hand, “Yuddite” is (kinda rude but) really rather clever.
On the other hand, the actual Luddites were concerned about technological unemployment which makes “Yuddite” a potentially misleading term, given that there’s something of a rift between the “concerned about ways in which AI might lead to people’s lives being worse within a world that’s basically like the one we have now” and “concerned about the possibility that AI will turn the world so completely upside down that there’s no room for us in it any more” camps and Yudkowsky is very firmly in the latter camp.
On the third hand, the Luddites made a prediction about a bad (for them) outcome, and were absolutely correct. They were against automatic looms because they thought the autolooms would replace their artisan product with lower quality goods and also worsen their wages and working conditions. They were right: https://www.smithsonianmag.com/history/what-the-luddites-really-fought-against-264412/
he is worried about catastrophic disruptions of the political, economic, and environmental systems
Ok, but how is this any different in practice? Or preventable via “corporate law”? It feels to me like people make too much of a distinction between slow and fast take offs scenarios, as if somewhat, if humans appear to be in the loop a bit more, this makes the problem less bad or less AI-related.
Essentially, if your mode of failure follows almost naturally from introducing AI system in current society and basic economic incentives, to the point that you can’t really look at any part of the process and identify anyone maliciously and intentionally setting it up to end the world, yet it does end the world, then it’s an AI problem. It may be a weird, slow, cyborg-like amalgamation of AI and human society that caused the catastrophe instead of a singular agentic AI taking everything over quickly, but the AI is still the main driver, and the only way to avoid the problem is to make AI extremely robust not just to intentional bad use but also to unintentional bad incentives feedback loops, essentially smart and moral enough to stop its own users and creators when they don’t know any better. Or alternatively, to just not make the AI at all.
Honestly, given what Facebook’s recommender systems have already caused, it’s disheartening that the leader of AI research at Meta doesn’t get something like this.
From a Facebook discussion with Scott Aaronson yesterday:
Yann: I think neither Yoshua nor Geoff believe that AI is going kill us all with any significant probability.
Scott: Well, Yoshua signed the pause letter, and wrote an accompanying statement about what he sees as the risk to civilization (I agree that there are many civilizational risks short of extinction). In his words: “No one, not even the leading AI experts, including those who developed these giant AI models, can be absolutely certain that such powerful tools now or in the future cannot be used in ways that would be catastrophic to society.”
Geoff said in a widely-shared recent video that it’s “not inconceivable” that AI will wipe out humanity, and didn’t offer any reassurances about it being vanishingly unlikely.
https://yoshuabengio.org/2023/04/05/slowing-down-development-of-ai-systems-passing-the-turing-test/
https://twitter.com/JMannhart/status/1641764742137016320
Yann: Scott Aaronson he is worried about catastrophic disruptions of the political, economic, and environmental systems. I don’t want to speak for him, but I doubt he worries about a Yuddite-style uncontrollable “hard takeoff”
On the one hand, “Yuddite” is (kinda rude but) really rather clever.
On the other hand, the actual Luddites were concerned about technological unemployment which makes “Yuddite” a potentially misleading term, given that there’s something of a rift between the “concerned about ways in which AI might lead to people’s lives being worse within a world that’s basically like the one we have now” and “concerned about the possibility that AI will turn the world so completely upside down that there’s no room for us in it any more” camps and Yudkowsky is very firmly in the latter camp.
On the third hand, the Luddites made a prediction about a bad (for them) outcome, and were absolutely correct. They were against automatic looms because they thought the autolooms would replace their artisan product with lower quality goods and also worsen their wages and working conditions. They were right: https://www.smithsonianmag.com/history/what-the-luddites-really-fought-against-264412/
Ok, but how is this any different in practice? Or preventable via “corporate law”? It feels to me like people make too much of a distinction between slow and fast take offs scenarios, as if somewhat, if humans appear to be in the loop a bit more, this makes the problem less bad or less AI-related.
Essentially, if your mode of failure follows almost naturally from introducing AI system in current society and basic economic incentives, to the point that you can’t really look at any part of the process and identify anyone maliciously and intentionally setting it up to end the world, yet it does end the world, then it’s an AI problem. It may be a weird, slow, cyborg-like amalgamation of AI and human society that caused the catastrophe instead of a singular agentic AI taking everything over quickly, but the AI is still the main driver, and the only way to avoid the problem is to make AI extremely robust not just to intentional bad use but also to unintentional bad incentives feedback loops, essentially smart and moral enough to stop its own users and creators when they don’t know any better. Or alternatively, to just not make the AI at all.
Honestly, given what Facebook’s recommender systems have already caused, it’s disheartening that the leader of AI research at Meta doesn’t get something like this.