I’ve actually just posted something on ideas I’m mulling on about this (post is called “AGI deployment as an act of aggression”). The general idea being that honestly, if we get out of our bubble, I think most people won’t just see the threat of misaligned AGI as dangerous, they’ll also hate the idea of aligned AGI that was deployed by a big state or a corporation (namely, most of the likely AGIs) and thus contributes to concentrating political and/or economic power in their hands. Combined, these two outcomes pretty much guarantee that large groups will be against AGI development altogether.
I think it’s important to consider all these perspective and lay down a “unified theory of AI danger”, so to speak. In many ways there are two axes that really define the danger: how powerful the AI is, and what is it aligned (or not aligned) to. Worrying about X-risk means worrying it will be very powerful (at which point, even a small misalignment is easily amplified to the point of lethality). Many people still seem to see X-risk as too unrealistic, or aren’t willing to concede to it because they feel like it’s been hyped up as a way to indirectly hype up the AI’s power itself. They expect AI to be less inherently powerful. But it should be made clear that power is a function of both capability and control handed off: if I was so stupid as to hand off a nuclear reactor control system to a GPT-4 instance, it could cause a meltdown and kill people tomorrow. Very powerful ASI might seize that control for itself (namely, even very little control is enough for it to leverage it), but even if you don’t believe that to be possible it’s almost irrelevant, because we’re obviously en route to give way too much control to AIs way too unreliable to deserve it. At the extreme end of that is X-risk, but it’s a whole spectrum of “we built an AI that doesn’t really do what we want, put it in charge of something and we done fucked up” from the recent news of a man who committed suicide after his chats with a likely GPT-4 powered bot to paperclipping the world. Solutions that fix paperclipping fix all the lesser problems in a very definite way too, so they should be shared ground.
I’ve actually just posted something on ideas I’m mulling on about this (post is called “AGI deployment as an act of aggression”). The general idea being that honestly, if we get out of our bubble, I think most people won’t just see the threat of misaligned AGI as dangerous, they’ll also hate the idea of aligned AGI that was deployed by a big state or a corporation (namely, most of the likely AGIs) and thus contributes to concentrating political and/or economic power in their hands. Combined, these two outcomes pretty much guarantee that large groups will be against AGI development altogether.
I think it’s important to consider all these perspective and lay down a “unified theory of AI danger”, so to speak. In many ways there are two axes that really define the danger: how powerful the AI is, and what is it aligned (or not aligned) to. Worrying about X-risk means worrying it will be very powerful (at which point, even a small misalignment is easily amplified to the point of lethality). Many people still seem to see X-risk as too unrealistic, or aren’t willing to concede to it because they feel like it’s been hyped up as a way to indirectly hype up the AI’s power itself. They expect AI to be less inherently powerful. But it should be made clear that power is a function of both capability and control handed off: if I was so stupid as to hand off a nuclear reactor control system to a GPT-4 instance, it could cause a meltdown and kill people tomorrow. Very powerful ASI might seize that control for itself (namely, even very little control is enough for it to leverage it), but even if you don’t believe that to be possible it’s almost irrelevant, because we’re obviously en route to give way too much control to AIs way too unreliable to deserve it. At the extreme end of that is X-risk, but it’s a whole spectrum of “we built an AI that doesn’t really do what we want, put it in charge of something and we done fucked up” from the recent news of a man who committed suicide after his chats with a likely GPT-4 powered bot to paperclipping the world. Solutions that fix paperclipping fix all the lesser problems in a very definite way too, so they should be shared ground.