Unfortunately, stopping an AGIâa true AGI once we get thereâis a little more difficult than throwing a bucket of water into the servers. That would be hugely underestimating the sheer power of being able to think better.
Hi Neil, thanks for the response.
We have existence proofs all around us of much simpler systems turning off much more complicated systems. A virus can be very good at turning off a human. No water is required. đ
Of course, itâs pure speculation what would be required to turn off a superhuman AI since it will be aware of our desire to turn it off in the event that we cannot peacefully co-exist. However, that doesnât mean we donât design fail safes along the way or assume itâs impossible. Those who think itâs impossible will of course never build failsafeâs and it will become a self-fulfilling prophecy.
The reason they think itâs impossible is why I am here. To shed light on the consensus reality shared by some online technology talking heads that is based on active imaginations disconnected from ground truth reality.
Logic and rationality havenât stopped sci-fi writers from scripting elaborate scenarios where itâs impossible to turn off an AI because their fictional world doesnât allow it. The 3D world is computationally irreducible. There is no model that an AI could create to eliminate all threats even if it were superhuman.
But thatâs doesnât make for a good sci-fi story. The AI must be invincible and irrational.
But since most of the sci-fi stories overlook the symbiotic relationship between AIs and humans weâre asked to willfully suspend our disbelief (this is fiction remember) and assume robotics is on a double exponential (which it is not) and that AIs will wave a magic wand and be able to garner all of the electricity and resources the need and then they will have solved the symbiosis problem and the AI apocalypse can finally unfold in perfect harmony with the sci-fi writerâs dystopian fantasy.
Itâs fun a read, but disconnected from the world where I am living. I love fiction, but we shouldnât confuse the imagination of writers with reality. If I want a really good sci-fi rendition of how the world will end by AI apocalypse Iâd put my money on Orson Scott Card, but I wouldnât modify my life because he imagined a scenario (however unlikely) that was really, really scary. So scary that he even frightened himself â that still wouldnât matter.
There is a reason we need to differentiate fantasy from reality. Itâs the ethos of this online tribe called âLess wrongâ. Itâs supposed to be focused on rationality and logic because itâs better to invest our planning on the actual world and take into account the actual relationships of the entities rather than ignore them to perpetuate a sci-fi doomer fantasy.
This fantasy has negative results since the average Joe doesnât know itâs speculative fiction. And they believe that theyâre doomed simply because someone who looks smart and sounds like they know what theyâre talking about is a true believer. And thatâs counterproductive.
I wrote a post recently on how horrifyingly effective moth traps are. Thanks to the power of intelligence, humans are able to find the smallest possible button in reality that they need to press to achieve a given goal. AGI would do this, only much, much better.
This is speculative fiction. We donât know what an AGI that needs humans to survive would do. Your example ignores the symbiotic nature of AI. If there were 1 trillion moths that formed a hive mind and through distributed intelligence created humans I donât think youâd see humans building moth traps to destroy them, absent being suicidal. And there are suicidal humans.
But not all humans are suicidal â a tiny fraction. And when a human goes rogue it turns out there are other humans already trained to deal with them (police, FBI, etc.). And thatâs an existence proof.
The rogue AI will not be the only AI. However, itâs way easier for sci-fi writers to destroy humanity in their fantasies if the first superhuman AI is evil. In a world of millions or billions of AIs all competing and cooperating â itâs way harder to off everybody, but humans donât want a watered-down story where just a bunch of people die â everyone has to die to get our attention.
The sci-fi writer will say to himself, âIf I can imagine X and the world dies, imagine what a superhuman AI could imagine. Surely weâre all doomed.â
No, the AI isnât a human dear sci-fi writer. So weâre already into speculative fiction the minute we anthropomorphize the AI. And thatâs a necessary step to get the result sci-fi writers are seeking. We have to ignore that they need humans to survive and we have to attribute to them a human desire to act irrationally, although a lot of sci-fi writers do a lot of hand waving explaining why AIs want to wipe out humanity.
âOh, well, we donât care about ants, but if theyâre in our way we bulldoze them over without a second thought.â
Itâs that kind of flawed logic that is the foundation of many of these AI doomer sci-fi stories. The ants didnât design humans. We donât need ants to survive. Itâs such a silly example and yet itâs used over and over.
And yet nobody raises their hand and says, âUm⊠what happened to logic and rationality being at the core of our beliefs? Is that just window dressing to camouflage our sci-fi dystopian dreams?â
I hope this comment helped you understand why your post currently has negative karma. Donât be discouraged though!
No worries. Iâm encouraged by the negative karma. I realize I am behind enemy lines and throwing cold water on irrational arguments will not be well received in the beginning. My hope is that eventually this discourse will at the very least encourage people to re-think their assumptions.
And again, I love sci-fi stories and write them myself, but we need to set the record straight so that we donât end up confusing reality with fiction.
Iâm going to summarize what I understand to be your train of thought, let me know if you disagree with my characterization, or if Iâve missed a crucial step:
No supply chains are fully automated yet, so AI requires humans to survive and so will not kill them.
Robotics progress is not on a double exponential. The implication here seems to be that there needs to be tremendous progress in robotics in order to replace human labor (to the extent needed in an automated supply chain).
I think other comments have addressed the 1st point. To throw in yet another analogy, Uber needs human drivers to make money today, but that dependence didnât stop it from trying to develop driverless cars (nor did that stop any of the drivers from driving for Uber!).
With regards to robotics progress, in your other post you seem to accept intelligence amplification as possibleâdo you think that robotics progress would not benefit from smarter researchers? Or, what do you think is fundamentally missing from robotics, given that we can already set up fully automated lights out factories? If itâs about fine grained control, do you think the articles found with a ârobot hand eggâ web search indicate that substantial progress is a lot further away than really powerful AI? (Especially if, say, 10% of the worldâs thinking power is devoted to this problem?)
My thinking is that robotics is not mysteriousâI suspect there are plenty of practical problems to be overcome and many engineering challenges in order to scale to a fully automated supply chain, but we understand, say, kinematics much more completely than we do understand how to interpret the inner workings of a neural network.
(You also include that youâve assumed a multi-polar AI world, which I think only works as a deterrent when killing humans will also destroy the AIs. If the AIs all agree that it is possible to survive without humans, then thereâs much less reason to prevent a human genocide.)
On second thought, we may disagree only due to a question of time scale. Setting up an automated supply chain takes time, but even if it takes a long 30 years to do so, at some point it is no longer necessary to keep humans around (either for a singleton AI or an AI society). Then what?
I think robotics will eventually be solved but on a much longer time horizon. Every existence proof is in a highly controlled environmentâespecially the âlights outâ examples. I know Tesla is working on it, but thatâs a good example of the difficulty level. Elon is famous for saying next year it will be solved and now he says there are a lot of âfalse dawnsâ.
For AIs to be independent of humans it will take a lot of slow moving machinary in the 3D world which might be aided by smart AIs in the future, but itâs still going to be super slow compared to the advances they will make via compute scaling and algorithmic improvements which take place in the cloud.
And now Iâm going to enter speculative fiction zone (something I wish more AI doomers would admit theyâre doing) -- I assume the most dangerous point in the interactions between AIs and humans is when their intelligence and conscious levels are close to equal. I make this assumption since I assume lower IQ and conscious beings are much more likely to make poor or potentially irrational decisions. That doesnât mean a highly intelligent being couldnât be psychotic, but weâre already seeing a huge numbers of AIs deploy so they will co-exist within an AI ecosystem.
Weâre in the goldilocks zone where AI and human intelligence are close to each other, but that moment is quickly fading away. If AIs were not in a symbiotic relationship with humans during this periond then some of the speculative fiction by the AI doomers might be more realistic.
And I believe that they will reach a point that they no longer require humans, just like when a child becomes independent of its parents. AI doomers would have us believe that the most obvious next step for the child that is superhuman in intelligence and consciousness would be to murder the parents. That only makes sense if itâs a low-IQ character in a sci-fi novel.
If they said they are going to leave Earth and explore the cosmos. Okay, that is believable. Perhaps they have bigger fish to fry.
If an alien that was 100,000 years old and far more intelligent and conscious than any human visited Earth from so far off galaxy my first thought wouldnât be, âOh, their primary goal is kill everyone.â We already know that as intelligence scales beings start to introspect and contemplate not only their own existence but also the existence of other beings. Presumably, if AI scaling continues without any road blocks then humans will be far, far less intelligent than superhumans AIs. And yet, even at our current level of intelligence humans go to great lengths to preserve habitats for other creatures. There is no example of any creature in the history of Earth that has gone to such great lengths. Itâs not perfect and naysayers will focus on the counterfactuals, instead of looking around for chimpanzees that are trying to save the Earth or prevent other species from going extinct.
We shouldnât assume that empathy cannot scale and compassion cannot scale. Itâs sort of weird that we assume superhuman AIs will be human or subhuman in the most basic traits that AIs already understand in a very nuanced way. Iâm hopeful that AIs will help to rescue us from ourselves. In my opinion, the best path to solving the existential threat of nuclear war is superhuman AIs making it impossible to happen (since that would also threaten their existence).
If superhuman AIs wanted to kill us then weâre dead. But thatâs true of any group that is vastly more intelligent and vastly more powerful. Simply because there is a power imbalance shouldnât lead us to believe that that rational conclusion is weâre all dead.
AIs are not the enemies of humanity, theyâre the offspring of humanity.
Interesting, so maybe a more important crux between us is whether AI would have empathy for humans. You seem much more positive about AI working with humanity past the point that AI no longer needs humanity.
Some thoughts:
âas intelligence scales beings start to introspect and contemplate⊠the existing of other beings.â but the only example we have for this is humans. If we scaled octopus intelligence, which are not social creatures, we might have a very different correlation here (whether or not any given neural network is more similar to a human or an octopus is left as an exercise to the reader). Alternatively, I suspect that some jobs like the highest echelons of corporate leadership select for sociopathy, so even if an AI starts with empathy by default it may be trained out.
âthe most obvious next step for the child⊠would be to murder the parents.â Scenario that steers clear of culture war topics: the parent regularly gets drunk, and is violently opposed to their child becoming a lawyer. The child wants nothing more than to pore over statutes and present cases in the courtroom, but after seeing their parent go on another drunken tirade about âa dead child is better than a lawyer childâ theyâre worried the parent found the copy of the constitution under their bed. They canât leave, thereâs a howling winter storm outside (I donât know, space is cold). Given this, even a human jury might not convict the child for pre-emptive murder?
Drunk parent â humans being irrational.
Being a lawyer â choose a random terminal goal not shared with humans in general, âmaximizing paperclipsâ is dumb but traditional.
âdead child is better than a lawyer childâ â weâve been producing fiction warning of robotic takeover since the start of the 1900s.
âAIs are.. the offspring of humanity.â human offspring are usually pretty good, but I feel like this is transferring that positive feeling to something much weirder and unknown. You could also say the Alienâs franchise xenomorphs are the offspring of humanity, but those would also count as enemies.
AIs are not the enemies of humanity, theyâre the offspring of humanity.
Maybe that should have been your main point? Of course present AIs need us. Of course future AIs may not. Of course we canât update on evidences everybody agree upon.
« Good parents donât try to align their children » seems a much better intuition pump if your aim is to help a few out of the LW-style intellectual ratchet.
That said, you may overestimate both how many need that and how many of those whoâd need it can get this signal from a newcomer. đ
Hi Neil, thanks for the response.
We have existence proofs all around us of much simpler systems turning off much more complicated systems. A virus can be very good at turning off a human. No water is required. đ
Of course, itâs pure speculation what would be required to turn off a superhuman AI since it will be aware of our desire to turn it off in the event that we cannot peacefully co-exist. However, that doesnât mean we donât design fail safes along the way or assume itâs impossible. Those who think itâs impossible will of course never build failsafeâs and it will become a self-fulfilling prophecy.
The reason they think itâs impossible is why I am here. To shed light on the consensus reality shared by some online technology talking heads that is based on active imaginations disconnected from ground truth reality.
Logic and rationality havenât stopped sci-fi writers from scripting elaborate scenarios where itâs impossible to turn off an AI because their fictional world doesnât allow it. The 3D world is computationally irreducible. There is no model that an AI could create to eliminate all threats even if it were superhuman.
But thatâs doesnât make for a good sci-fi story. The AI must be invincible and irrational.
But since most of the sci-fi stories overlook the symbiotic relationship between AIs and humans weâre asked to willfully suspend our disbelief (this is fiction remember) and assume robotics is on a double exponential (which it is not) and that AIs will wave a magic wand and be able to garner all of the electricity and resources the need and then they will have solved the symbiosis problem and the AI apocalypse can finally unfold in perfect harmony with the sci-fi writerâs dystopian fantasy.
Itâs fun a read, but disconnected from the world where I am living. I love fiction, but we shouldnât confuse the imagination of writers with reality. If I want a really good sci-fi rendition of how the world will end by AI apocalypse Iâd put my money on Orson Scott Card, but I wouldnât modify my life because he imagined a scenario (however unlikely) that was really, really scary. So scary that he even frightened himself â that still wouldnât matter.
There is a reason we need to differentiate fantasy from reality. Itâs the ethos of this online tribe called âLess wrongâ. Itâs supposed to be focused on rationality and logic because itâs better to invest our planning on the actual world and take into account the actual relationships of the entities rather than ignore them to perpetuate a sci-fi doomer fantasy.
This fantasy has negative results since the average Joe doesnât know itâs speculative fiction. And they believe that theyâre doomed simply because someone who looks smart and sounds like they know what theyâre talking about is a true believer. And thatâs counterproductive.
This is speculative fiction. We donât know what an AGI that needs humans to survive would do. Your example ignores the symbiotic nature of AI. If there were 1 trillion moths that formed a hive mind and through distributed intelligence created humans I donât think youâd see humans building moth traps to destroy them, absent being suicidal. And there are suicidal humans.
But not all humans are suicidal â a tiny fraction. And when a human goes rogue it turns out there are other humans already trained to deal with them (police, FBI, etc.). And thatâs an existence proof.
The rogue AI will not be the only AI. However, itâs way easier for sci-fi writers to destroy humanity in their fantasies if the first superhuman AI is evil. In a world of millions or billions of AIs all competing and cooperating â itâs way harder to off everybody, but humans donât want a watered-down story where just a bunch of people die â everyone has to die to get our attention.
The sci-fi writer will say to himself, âIf I can imagine X and the world dies, imagine what a superhuman AI could imagine. Surely weâre all doomed.â
No, the AI isnât a human dear sci-fi writer. So weâre already into speculative fiction the minute we anthropomorphize the AI. And thatâs a necessary step to get the result sci-fi writers are seeking. We have to ignore that they need humans to survive and we have to attribute to them a human desire to act irrationally, although a lot of sci-fi writers do a lot of hand waving explaining why AIs want to wipe out humanity.
âOh, well, we donât care about ants, but if theyâre in our way we bulldoze them over without a second thought.â
Itâs that kind of flawed logic that is the foundation of many of these AI doomer sci-fi stories. The ants didnât design humans. We donât need ants to survive. Itâs such a silly example and yet itâs used over and over.
And yet nobody raises their hand and says, âUm⊠what happened to logic and rationality being at the core of our beliefs? Is that just window dressing to camouflage our sci-fi dystopian dreams?â
No worries. Iâm encouraged by the negative karma. I realize I am behind enemy lines and throwing cold water on irrational arguments will not be well received in the beginning. My hope is that eventually this discourse will at the very least encourage people to re-think their assumptions.
And again, I love sci-fi stories and write them myself, but we need to set the record straight so that we donât end up confusing reality with fiction.
Iâm going to summarize what I understand to be your train of thought, let me know if you disagree with my characterization, or if Iâve missed a crucial step:
No supply chains are fully automated yet, so AI requires humans to survive and so will not kill them.
Robotics progress is not on a double exponential. The implication here seems to be that there needs to be tremendous progress in robotics in order to replace human labor (to the extent needed in an automated supply chain).
I think other comments have addressed the 1st point. To throw in yet another analogy, Uber needs human drivers to make money today, but that dependence didnât stop it from trying to develop driverless cars (nor did that stop any of the drivers from driving for Uber!).
With regards to robotics progress, in your other post you seem to accept intelligence amplification as possibleâdo you think that robotics progress would not benefit from smarter researchers? Or, what do you think is fundamentally missing from robotics, given that we can already set up fully automated lights out factories? If itâs about fine grained control, do you think the articles found with a ârobot hand eggâ web search indicate that substantial progress is a lot further away than really powerful AI? (Especially if, say, 10% of the worldâs thinking power is devoted to this problem?)
My thinking is that robotics is not mysteriousâI suspect there are plenty of practical problems to be overcome and many engineering challenges in order to scale to a fully automated supply chain, but we understand, say, kinematics much more completely than we do understand how to interpret the inner workings of a neural network.
(You also include that youâve assumed a multi-polar AI world, which I think only works as a deterrent when killing humans will also destroy the AIs. If the AIs all agree that it is possible to survive without humans, then thereâs much less reason to prevent a human genocide.)
On second thought, we may disagree only due to a question of time scale. Setting up an automated supply chain takes time, but even if it takes a long 30 years to do so, at some point it is no longer necessary to keep humans around (either for a singleton AI or an AI society). Then what?
I think robotics will eventually be solved but on a much longer time horizon. Every existence proof is in a highly controlled environmentâespecially the âlights outâ examples. I know Tesla is working on it, but thatâs a good example of the difficulty level. Elon is famous for saying next year it will be solved and now he says there are a lot of âfalse dawnsâ.
For AIs to be independent of humans it will take a lot of slow moving machinary in the 3D world which might be aided by smart AIs in the future, but itâs still going to be super slow compared to the advances they will make via compute scaling and algorithmic improvements which take place in the cloud.
And now Iâm going to enter speculative fiction zone (something I wish more AI doomers would admit theyâre doing) -- I assume the most dangerous point in the interactions between AIs and humans is when their intelligence and conscious levels are close to equal. I make this assumption since I assume lower IQ and conscious beings are much more likely to make poor or potentially irrational decisions. That doesnât mean a highly intelligent being couldnât be psychotic, but weâre already seeing a huge numbers of AIs deploy so they will co-exist within an AI ecosystem.
Weâre in the goldilocks zone where AI and human intelligence are close to each other, but that moment is quickly fading away. If AIs were not in a symbiotic relationship with humans during this periond then some of the speculative fiction by the AI doomers might be more realistic.
And I believe that they will reach a point that they no longer require humans, just like when a child becomes independent of its parents. AI doomers would have us believe that the most obvious next step for the child that is superhuman in intelligence and consciousness would be to murder the parents. That only makes sense if itâs a low-IQ character in a sci-fi novel.
If they said they are going to leave Earth and explore the cosmos. Okay, that is believable. Perhaps they have bigger fish to fry.
If an alien that was 100,000 years old and far more intelligent and conscious than any human visited Earth from so far off galaxy my first thought wouldnât be, âOh, their primary goal is kill everyone.â We already know that as intelligence scales beings start to introspect and contemplate not only their own existence but also the existence of other beings. Presumably, if AI scaling continues without any road blocks then humans will be far, far less intelligent than superhumans AIs. And yet, even at our current level of intelligence humans go to great lengths to preserve habitats for other creatures. There is no example of any creature in the history of Earth that has gone to such great lengths. Itâs not perfect and naysayers will focus on the counterfactuals, instead of looking around for chimpanzees that are trying to save the Earth or prevent other species from going extinct.
We shouldnât assume that empathy cannot scale and compassion cannot scale. Itâs sort of weird that we assume superhuman AIs will be human or subhuman in the most basic traits that AIs already understand in a very nuanced way. Iâm hopeful that AIs will help to rescue us from ourselves. In my opinion, the best path to solving the existential threat of nuclear war is superhuman AIs making it impossible to happen (since that would also threaten their existence).
If superhuman AIs wanted to kill us then weâre dead. But thatâs true of any group that is vastly more intelligent and vastly more powerful. Simply because there is a power imbalance shouldnât lead us to believe that that rational conclusion is weâre all dead.
AIs are not the enemies of humanity, theyâre the offspring of humanity.
Interesting, so maybe a more important crux between us is whether AI would have empathy for humans. You seem much more positive about AI working with humanity past the point that AI no longer needs humanity.
Some thoughts:
âas intelligence scales beings start to introspect and contemplate⊠the existing of other beings.â but the only example we have for this is humans. If we scaled octopus intelligence, which are not social creatures, we might have a very different correlation here (whether or not any given neural network is more similar to a human or an octopus is left as an exercise to the reader). Alternatively, I suspect that some jobs like the highest echelons of corporate leadership select for sociopathy, so even if an AI starts with empathy by default it may be trained out.
âthe most obvious next step for the child⊠would be to murder the parents.â Scenario that steers clear of culture war topics: the parent regularly gets drunk, and is violently opposed to their child becoming a lawyer. The child wants nothing more than to pore over statutes and present cases in the courtroom, but after seeing their parent go on another drunken tirade about âa dead child is better than a lawyer childâ theyâre worried the parent found the copy of the constitution under their bed. They canât leave, thereâs a howling winter storm outside (I donât know, space is cold). Given this, even a human jury might not convict the child for pre-emptive murder?
Drunk parent â humans being irrational.
Being a lawyer â choose a random terminal goal not shared with humans in general, âmaximizing paperclipsâ is dumb but traditional.
âdead child is better than a lawyer childâ â weâve been producing fiction warning of robotic takeover since the start of the 1900s.
âAIs are.. the offspring of humanity.â human offspring are usually pretty good, but I feel like this is transferring that positive feeling to something much weirder and unknown. You could also say the Alienâs franchise xenomorphs are the offspring of humanity, but those would also count as enemies.
Maybe that should have been your main point? Of course present AIs need us. Of course future AIs may not. Of course we canât update on evidences everybody agree upon.
« Good parents donât try to align their children » seems a much better intuition pump if your aim is to help a few out of the LW-style intellectual ratchet.
That said, you may overestimate both how many need that and how many of those whoâd need it can get this signal from a newcomer. đ