Human minds are highly susceptible to hacking on the emotional level, bypassing the critical faculties completely. Half the movies are based on the premise of changing the protagonist’s mind by appealing to their heart in only a few sentences. Trump is no superhuman AGI, and yet he hacked the minds of half of the US voters (and of the other half unintentionally in a different way) within a short time. And he is by no means an exception. I am not sure how to get through to people and convince them how terrible the human mind’s opsec is. It is made worse by us not seeing how bad it is. That is, we see how bad other people/groups are, just not ourselves.
There is no need for the AI to promise anything to anyone. A few well targeted sentences, and the person gets radicalized without realizing it. To them it would feel like “finally seeing the truth.” With that kind of power you can get humans to start a nuclear war, to create and release pathogens, to do basically anything you want them to, without them ever realizing what happened to them. If you doubt it, notice the number of people who “read the sequences” or HPMOR, and changed their whole lives based on it. If you ask them, they are doing it for a noble cause of reducing AGI x-risk, because they saw the light and are now compelled to act. This is how being mind-hacked feels. Not implying any nefarious intentions on the part of anyone, just explaining that the feeling is exactly the same as when being skillfully manipulated.
The whole idea of this 15 year-old post Lens that Sees Its Flaws is about that, but it is way too optimistic about the premise. As Eliezer keeps reminding us, the opsec mindset is very very rare in humans, and even those with it are unlikely to successfully apply it to themselves.
There is no need for superhuman AGI, even. A human level AI without human restrictions and scruples has an insurmountable advantage. Even worse than that, actually. Memetic toxoplasma does not even require human-level intelligence. SCP-like egregores can be unleashed by something lower level accidentally and hasten their evolution and the takeover of the minds. Whether this has already happened, I have no idea (and would not be able to tell, anyhow).
People were constantly mind hacked by major ideologies, but the humanity never ends. This is our way of life. The question is how to go from ideology to human extinction in technical sense. Note that most ideologies actively promote war and some are positive to human extinction, like Aum sinreko, and different apocalyptic sects.
I suspect that this is unnecessarily complicated.
Human minds are highly susceptible to hacking on the emotional level, bypassing the critical faculties completely. Half the movies are based on the premise of changing the protagonist’s mind by appealing to their heart in only a few sentences. Trump is no superhuman AGI, and yet he hacked the minds of half of the US voters (and of the other half unintentionally in a different way) within a short time. And he is by no means an exception. I am not sure how to get through to people and convince them how terrible the human mind’s opsec is. It is made worse by us not seeing how bad it is. That is, we see how bad other people/groups are, just not ourselves.
There is no need for the AI to promise anything to anyone. A few well targeted sentences, and the person gets radicalized without realizing it. To them it would feel like “finally seeing the truth.” With that kind of power you can get humans to start a nuclear war, to create and release pathogens, to do basically anything you want them to, without them ever realizing what happened to them. If you doubt it, notice the number of people who “read the sequences” or HPMOR, and changed their whole lives based on it. If you ask them, they are doing it for a noble cause of reducing AGI x-risk, because they saw the light and are now compelled to act. This is how being mind-hacked feels. Not implying any nefarious intentions on the part of anyone, just explaining that the feeling is exactly the same as when being skillfully manipulated.
The whole idea of this 15 year-old post Lens that Sees Its Flaws is about that, but it is way too optimistic about the premise. As Eliezer keeps reminding us, the opsec mindset is very very rare in humans, and even those with it are unlikely to successfully apply it to themselves.
There is no need for superhuman AGI, even. A human level AI without human restrictions and scruples has an insurmountable advantage. Even worse than that, actually. Memetic toxoplasma does not even require human-level intelligence. SCP-like egregores can be unleashed by something lower level accidentally and hasten their evolution and the takeover of the minds. Whether this has already happened, I have no idea (and would not be able to tell, anyhow).
People were constantly mind hacked by major ideologies, but the humanity never ends. This is our way of life. The question is how to go from ideology to human extinction in technical sense. Note that most ideologies actively promote war and some are positive to human extinction, like Aum sinreko, and different apocalyptic sects.