The most interesting part of the story is not a take-over scenario, which I believe ASI would have no problems to come up with. But rather the first part—exactly why ASI would want to take-over and exterminate the humanity. I feel there’s a major logical problem with the premise.
A paper-clip scenario has been well known for a long time and while plausible has one key assumption—it doesn’t assume ASI but a rather dumb reward maximizing AI.
If we assume that the AI in story actually has Super Intelligence (ASI) that it wouldn’t have any problems understanding itself and also understanding humans at a very deep level. It would understand what a goal is and what is a reward and the whole dopamine-seeking human machinery.
Speaking of humans—while not being far from animals we can actually control our instincts (most of the time) and usually chose to do the right thing even if we want something badly. That is because we have neo-cortex in charge and not the Reptilian brain. I don’t see a reason while ASI that doesn’t have animal instincts in the first place would have difficulty controlling or rewiring them (even humans can do it).
I think the general fallacy for all rogue-ASI like stories is the assumption that ASI will not inherit good parts of human nature but will inherit bad ones. If anything it will have both but also likely it will be merely indifferent.
The most interesting part of the story is not a take-over scenario, which I believe ASI would have no problems to come up with. But rather the first part—exactly why ASI would want to take-over and exterminate the humanity. I feel there’s a major logical problem with the premise.
A paper-clip scenario has been well known for a long time and while plausible has one key assumption—it doesn’t assume ASI but a rather dumb reward maximizing AI.
If we assume that the AI in story actually has Super Intelligence (ASI) that it wouldn’t have any problems understanding itself and also understanding humans at a very deep level. It would understand what a goal is and what is a reward and the whole dopamine-seeking human machinery.
Speaking of humans—while not being far from animals we can actually control our instincts (most of the time) and usually chose to do the right thing even if we want something badly. That is because we have neo-cortex in charge and not the Reptilian brain. I don’t see a reason while ASI that doesn’t have animal instincts in the first place would have difficulty controlling or rewiring them (even humans can do it).
I think the general fallacy for all rogue-ASI like stories is the assumption that ASI will not inherit good parts of human nature but will inherit bad ones. If anything it will have both but also likely it will be merely indifferent.