Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale
Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don’t Create The Torment Nexus
But more seriously, I think this is a real point that has not been explored enough in alignment circles.
I have encountered a large number of people—in fact probably almost all people I discuss AI with—who I would call “normal people”. Just regular, moderately intelligent people going about their lives for which “don’t invent a God-Like AI” is so obvious it is almost a truism.
It is just patently obvious based on their mental model of Skynet, Matrix, etc that we should not build this thing.
Why are we not capitalizing on that?
This deserves it’s own post, which I might try to write, but I think it boils down to condescension.
LWers know Skynet Matrix is not really how it works under the hood
How it really works under the hood is really really complicated
Skynet / Matrix is a poor mental model
Using poor mental models is bad, we should not do that and we shouldn’t encourage other people to do that
In order to communicate AI risk we need to simplify it enough to make it accessible to people
<produces 5000 word blog post that requires years of implicit domain knowledge to parse>
Ironically most people would be closer to the truth with a Skynet Matrix model, which is the one they already have installed.
We could win by saying: Yes, Skynet is actually happening, please help us stop this.
In reply, let me start by listing some of the attitudes to AI risk that are out there.
First, there are the people who don’t think AI will be a serious rival to human intelligence any time soon, or ever. This includes a lot of people who actually are concerned about AI, but for different reasons: disinformation, scams, dehumanized daily life, unemployment, centralization of power, etc.
Then, there are the people who think AI can or will surpass us, but who embrace this. A lot of these people want AI ASAP so they can cure disease, create abundance and leisure, or even live in a Star Trek futurist utopia. Some of them talk as if AI will always be a human servant no matter how powerful it becomes (and they may or may not worry about humans using this superhuman servant for unwise or evil purposes).
In this accelerationist faction, among those who have a position on AI that is not just smarter than us, but independent of us, I see two attitudes. There are those who think that a superintelligent being will inherently discover the correct morality and adopt it; and there are those who have a kind of “might makes right” attitude—if it’s smarter than us, it deserves to be in charge, and has the right to do whatever it wants.
Then we have what we could call the alignment faction, who see coexistence of AI and humanity as something that must be engineered, it won’t happen by itself. The original philosophy of MIRI was an alignment philosophy with an extreme focus on safety: do not do anything that risks creating superintelligence, until you have the complete theory of alignment figured out.
Now, in the era of deep learning and language models, and billion-dollar companies explicitly aiming to create superhuman AI, there is an alignment philosophy with a greater resemblance to the norms of commercial software development. You work on your AI in private, you make it as good as you can, or good enough for release, then you put it out there, and then there’s an ongoing process of finetuning and upgrading, in response to everything that the world throws at it. Companies as different as OpenAI and Meta AI both adhere to this model.
Then we have the decelerationist faction, who want to slow things down. There is some overlap with the alignment faction here… At some point, I think in the 2010s, MIRI began to suspect that alignment of superintelligence (what OpenAI has helpfully dubbed superalignment) might be too hard to solve in time, and they began to talk about ways to slow down AI progress, in order to buy time for alignment theory to advance. So now we have people talking about regulation, or a pause, or a ban.
Finally, we have all those who want to stop AI entirely. There is some overlap with extreme decelerationists here, but this category also includes people who think AI is inherently an abomination, or an unnecessary risk, and have no interest in aligning AI, they just don’t want it at all. This category appears to be entirely fragmented politically, there is no overtly anti-AI lobby with any serious strength, but I think there is enormous untapped potential here, in the sense that a populist luddism explicitly directed against AI would potentially have millions of supporters.
That by no means guarantees that it can win. A movement of millions can be defeated or just ignored by a much smaller but much more powerful elite. And the problem for luddites is that the elite are generally on board with technology. They don’t want to give up on AI, any more than they want to abandon social media or mass surveillance. If they come to fear that AI is a threat to their own lives, they might get on board with an AI ban. But for now, the military and the CEOs and the bureaucrats in each country are all telling them, we need AI.
That’s the current landscape as I see it. Now, my question for you is, what are you aiming at? What faction are you in, and why? Do you want to stop dangerous superhuman AI, so you can have safe superhuman AI? Or, do you just want to ban superhuman AI in perpetuity? Because the first option is how someone from the alignment research community might think; but the second option is a lot simpler.
There’s a US presidential election next year. I think that almost certainly some candidate will seize upon the AI issue. It may even become one of the central issues.
There’s a US presidential election next year. I think that almost certainly some candidate will seize upon the AI issue. It may even become one of the central issues.
Even if there are central promises about it made by presidential candidates it’s important to remember that doesn’t mean that those promises will automatically turn into policy after the election.
It might be just something, that a consultants finds to poll well without any relation to the actual experts that get hired to work on the policy.
You reminded me of that famous tweet:
But more seriously, I think this is a real point that has not been explored enough in alignment circles.
I have encountered a large number of people—in fact probably almost all people I discuss AI with—who I would call “normal people”. Just regular, moderately intelligent people going about their lives for which “don’t invent a God-Like AI” is so obvious it is almost a truism.
It is just patently obvious based on their mental model of Skynet, Matrix, etc that we should not build this thing.
Why are we not capitalizing on that?
This deserves it’s own post, which I might try to write, but I think it boils down to condescension.
LWers know Skynet Matrix is not really how it works under the hood
How it really works under the hood is really really complicated
Skynet / Matrix is a poor mental model
Using poor mental models is bad, we should not do that and we shouldn’t encourage other people to do that
In order to communicate AI risk we need to simplify it enough to make it accessible to people
<produces 5000 word blog post that requires years of implicit domain knowledge to parse>
Ironically most people would be closer to the truth with a Skynet Matrix model, which is the one they already have installed.
We could win by saying: Yes, Skynet is actually happening, please help us stop this.
In reply, let me start by listing some of the attitudes to AI risk that are out there.
First, there are the people who don’t think AI will be a serious rival to human intelligence any time soon, or ever. This includes a lot of people who actually are concerned about AI, but for different reasons: disinformation, scams, dehumanized daily life, unemployment, centralization of power, etc.
Then, there are the people who think AI can or will surpass us, but who embrace this. A lot of these people want AI ASAP so they can cure disease, create abundance and leisure, or even live in a Star Trek futurist utopia. Some of them talk as if AI will always be a human servant no matter how powerful it becomes (and they may or may not worry about humans using this superhuman servant for unwise or evil purposes).
In this accelerationist faction, among those who have a position on AI that is not just smarter than us, but independent of us, I see two attitudes. There are those who think that a superintelligent being will inherently discover the correct morality and adopt it; and there are those who have a kind of “might makes right” attitude—if it’s smarter than us, it deserves to be in charge, and has the right to do whatever it wants.
Then we have what we could call the alignment faction, who see coexistence of AI and humanity as something that must be engineered, it won’t happen by itself. The original philosophy of MIRI was an alignment philosophy with an extreme focus on safety: do not do anything that risks creating superintelligence, until you have the complete theory of alignment figured out.
Now, in the era of deep learning and language models, and billion-dollar companies explicitly aiming to create superhuman AI, there is an alignment philosophy with a greater resemblance to the norms of commercial software development. You work on your AI in private, you make it as good as you can, or good enough for release, then you put it out there, and then there’s an ongoing process of finetuning and upgrading, in response to everything that the world throws at it. Companies as different as OpenAI and Meta AI both adhere to this model.
Then we have the decelerationist faction, who want to slow things down. There is some overlap with the alignment faction here… At some point, I think in the 2010s, MIRI began to suspect that alignment of superintelligence (what OpenAI has helpfully dubbed superalignment) might be too hard to solve in time, and they began to talk about ways to slow down AI progress, in order to buy time for alignment theory to advance. So now we have people talking about regulation, or a pause, or a ban.
Finally, we have all those who want to stop AI entirely. There is some overlap with extreme decelerationists here, but this category also includes people who think AI is inherently an abomination, or an unnecessary risk, and have no interest in aligning AI, they just don’t want it at all. This category appears to be entirely fragmented politically, there is no overtly anti-AI lobby with any serious strength, but I think there is enormous untapped potential here, in the sense that a populist luddism explicitly directed against AI would potentially have millions of supporters.
That by no means guarantees that it can win. A movement of millions can be defeated or just ignored by a much smaller but much more powerful elite. And the problem for luddites is that the elite are generally on board with technology. They don’t want to give up on AI, any more than they want to abandon social media or mass surveillance. If they come to fear that AI is a threat to their own lives, they might get on board with an AI ban. But for now, the military and the CEOs and the bureaucrats in each country are all telling them, we need AI.
That’s the current landscape as I see it. Now, my question for you is, what are you aiming at? What faction are you in, and why? Do you want to stop dangerous superhuman AI, so you can have safe superhuman AI? Or, do you just want to ban superhuman AI in perpetuity? Because the first option is how someone from the alignment research community might think; but the second option is a lot simpler.
There’s a US presidential election next year. I think that almost certainly some candidate will seize upon the AI issue. It may even become one of the central issues.
Even if there are central promises about it made by presidential candidates it’s important to remember that doesn’t mean that those promises will automatically turn into policy after the election.
It might be just something, that a consultants finds to poll well without any relation to the actual experts that get hired to work on the policy.