I consider “6. Aims to kill us all” to be an unnecessary assumption, one that overlooks many existential risks.
People were only somewhat smarter than North America’s indigenous horses and other large animals, most of which were wiped out (with our help) long ago. However, eliminating horses and megafauna probably wasn’t a conscious aim. Those were most likely inadvertent oopsies, similar to wiping out the passenger pigeon (complete with shooting and eating the last survivor found in the wild) and our other missteps. I can only barely imagine ASI objectives where all the atoms in the universe are required, where human extinction is thus central to the goal. The more plausible worry, to me, is ASI’s indifference, where eventually ours would be the anthill that gets stepped on simply because the shortest path includes that step. Same outcome, but a different mechanism.
It’s probably important to consider all ASI objectives that may lead to our obliteration, not just malice. Limiting ASI designs to those that are not malignant is insufficient for human survival, and only focusing on malice may lead people to underemphasize the magnitude of the overall ASI risk. In addition to the risk you are talking about more generally, that our future with AI will eventually be outside our control, a second factor is that Ernst Stavro Blofeld exists and would certainly use any earlier AGI to, as he would put it to ChatGPT-11.2, help him write the villain’s plan for his next 007 novel/movie: “I’m Ian Fleming’s literary heir, his great-grandniece—and I’m writing my next …”
On the positive side, kids have been known to take care of an ant farm without applying a magnifying glass to heat it in the sun. Perhaps our trivial but to us purposeful motions will be fun for ASI to watch and will continue to entertain.
The reason I included that was so I didn’t have to get into arguments about it or have people harp on it, not because I thought you actually needed it. The whole idea is to isolate different objections.
Perhaps our trivial but to us purposeful motions will be fun for ASI to watch and will continue to entertain.
This is something I see bandied about at different levels of seriousness, sometimes even as full defense of AI x-risk. But why would an AI experience entertainment? That type of experience in humans is caused by a feedback loop between the brain and the body. Without physical biological bodies to interrupt or interfere with a programmatically defined reward function in that way, the reward function maintains at state indefinitely.
I think it’s reasonable to assume that AI would build one logical conclusion on another with exceptional rapidity, relative to slower thinkers. Eventually, and probably soon because of its speed, I expect that AI would hit a dead end where it simply doesn’t have the facts to add to its already complete analysis of the information it started with plus the information it has collected. In that situation, many people would want entertainment, so we can speculate that maybe AI would want entertainment too. Generalizing from one example is not anywhere near conclusive, but it provides a plausible scenario.
I consider “6. Aims to kill us all” to be an unnecessary assumption, one that overlooks many existential risks.
People were only somewhat smarter than North America’s indigenous horses and other large animals, most of which were wiped out (with our help) long ago. However, eliminating horses and megafauna probably wasn’t a conscious aim. Those were most likely inadvertent oopsies, similar to wiping out the passenger pigeon (complete with shooting and eating the last survivor found in the wild) and our other missteps. I can only barely imagine ASI objectives where all the atoms in the universe are required, where human extinction is thus central to the goal. The more plausible worry, to me, is ASI’s indifference, where eventually ours would be the anthill that gets stepped on simply because the shortest path includes that step. Same outcome, but a different mechanism.
It’s probably important to consider all ASI objectives that may lead to our obliteration, not just malice. Limiting ASI designs to those that are not malignant is insufficient for human survival, and only focusing on malice may lead people to underemphasize the magnitude of the overall ASI risk. In addition to the risk you are talking about more generally, that our future with AI will eventually be outside our control, a second factor is that Ernst Stavro Blofeld exists and would certainly use any earlier AGI to, as he would put it to ChatGPT-11.2, help him write the villain’s plan for his next 007 novel/movie: “I’m Ian Fleming’s literary heir, his great-grandniece—and I’m writing my next …”
On the positive side, kids have been known to take care of an ant farm without applying a magnifying glass to heat it in the sun. Perhaps our trivial but to us purposeful motions will be fun for ASI to watch and will continue to entertain.
The reason I included that was so I didn’t have to get into arguments about it or have people harp on it, not because I thought you actually needed it. The whole idea is to isolate different objections.
This is something I see bandied about at different levels of seriousness, sometimes even as full defense of AI x-risk. But why would an AI experience entertainment? That type of experience in humans is caused by a feedback loop between the brain and the body. Without physical biological bodies to interrupt or interfere with a programmatically defined reward function in that way, the reward function maintains at state indefinitely.
“But why would an AI experience entertainment?”
I think it’s reasonable to assume that AI would build one logical conclusion on another with exceptional rapidity, relative to slower thinkers. Eventually, and probably soon because of its speed, I expect that AI would hit a dead end where it simply doesn’t have the facts to add to its already complete analysis of the information it started with plus the information it has collected. In that situation, many people would want entertainment, so we can speculate that maybe AI would want entertainment too. Generalizing from one example is not anywhere near conclusive, but it provides a plausible scenario.