Not sure I have anything to add to the question but do find myself having to ask why the general presumption so often seems to be that of AI gets annoyed at stupid people and kills humanity.
It’s true that we can think of situation where that might be possible, and maybe even a predictable AI response, but I just wonder if such settings are all that probable.
Has anyone ever sat down and tried to list out the situations where an AI would have some incentive to kill off humanity and then assess how reasonable thinking such a situation might be?
Not sure I have anything to add to the question but do find myself having to ask why the general presumption so often seems to be that of AI gets annoyed at stupid people and kills humanity.
It’s true that we can think of situation where that might be possible, and maybe even a predictable AI response, but I just wonder if such settings are all that probable.
Has anyone ever sat down and tried to list out the situations where an AI would have some incentive to kill off humanity and then assess how reasonable thinking such a situation might be?
It will kill humanity not because it will be annoyed, but for two main goals: its own safety, or to use human atoms. Other variants also possible, I explored them here: http://lesswrong.com/lw/mgf/a_map_agi_failures_modes_and_levels/