A profit optimizing AI is not even a little bit aligned. It’s very important to make sure that everyone knows that if they create a being that’s smarter than them and tell it to optimize a number in a company bank account, the most likely outcome is that the agent hijacks democracy and does whatever it had to do to make sure the company (and the financial system) outlasts humanity.
Succeeding at the technical problem never looks like the scenarios you’re describing. If we had even a barely superhuman intelligence that was truly honest, and helpful (no intuitive sense of “harmless” is ultimately compatible with these things), it would be obvious to it that none of us want war, and it would tell us that.
The conversation about whether superintelligent agents would fall into molochean dynamics has been going on for a while. Personally, I think it’s pretty clear that coordination is just not that hard, humans are uniquely bad at it, but still impressively good at it. Even slightly more advanced agents (for instance, humans plus cheaper cameras and a better internet) could just make binding commitments to avoid wasteful conflict, especially conflict so destructive that it would force them all to carcinise.
That’s a very fair response. My claim here is really about the outer alignment problem, and that if lots of people have access to the ability to create / fine tune AI agents, many agents that have goals misaligned with humanity as a whole will be created, and we will lose control of the future.
A profit optimizing AI is not even a little bit aligned. It’s very important to make sure that everyone knows that if they create a being that’s smarter than them and tell it to optimize a number in a company bank account, the most likely outcome is that the agent hijacks democracy and does whatever it had to do to make sure the company (and the financial system) outlasts humanity.
Succeeding at the technical problem never looks like the scenarios you’re describing. If we had even a barely superhuman intelligence that was truly honest, and helpful (no intuitive sense of “harmless” is ultimately compatible with these things), it would be obvious to it that none of us want war, and it would tell us that.
The conversation about whether superintelligent agents would fall into molochean dynamics has been going on for a while.
Personally, I think it’s pretty clear that coordination is just not that hard, humans are uniquely bad at it, but still impressively good at it. Even slightly more advanced agents (for instance, humans plus cheaper cameras and a better internet) could just make binding commitments to avoid wasteful conflict, especially conflict so destructive that it would force them all to carcinise.
That’s a very fair response. My claim here is really about the outer alignment problem, and that if lots of people have access to the ability to create / fine tune AI agents, many agents that have goals misaligned with humanity as a whole will be created, and we will lose control of the future.