In the Death Note anime, the Death Note gives the protagonist the ability to create a totalitarian justice system. That is, if there isn’t a force to stop him. But a character named L quickly catches onto him, and makes his ability to “serve justice” harder. In other words, the Death Note is a powerful tool (like AGI), but opposing forces hamper its ability to affect the world.
In the Death Note series, L only responded when he noticed the anomalous effects of the Death Note. He took a reactive approach.
When talking about AGI, usually the discussion gravitates towards “and then the uptick in intelligence goes out of control, leading to completely unpredictable behavior”. Which then leads to the conclusion that “in such chaotic situations, the chance that things go wrong is significantly higher than that all stars align”.
Although I agree with the sentiment, and see truth in all the ways AGI could be dangerous, sometimes we disregard the “L” in reasoning about AGI.
In nature, many processes evolve to completely chaotic situations unless constrained by opposing forces (Entropy Theory). Any species seeks dominance, until another species comes along to keep populations under control. Physical processes gather entropy till we step in and reduce it (hence why I have to clean my room every now and then). Even psychology shows these patterns, where self-serving thoughts radicalize, giving rise to wide-ranging mental issues, till a force stops them .
I often hear the argument “the required IQ to destroy the world decreases every year”. I don’t fully agree with that sentiment. As new technologies arise to do a lot of harm with little IQ, new constraints arise to reduce harm. Nuclear weapons are incredibly scary and a huge risk, so we have seen global nuclear disarmament, which shows in the sharp reduction of recent nuclear close calls. The IQ required to destroy the world through nuclear weapons has decreased because of the “L” that arose to oppose it.
As computer intelligence evolves, we are ramping up the “nuclear warheads” that machines have access to. We might need an “AGI close calls” Wikipedia page soon to document how this evolution leads to dangerous situations. There might be disastrous accidents involved. But, as nature dictates, an opposing force will come along.
In conclusion: Like Death Note’s L, any force in nature has its opposing forces. As AGI emerges and gains power, forces will arise that weaken its power. This isn’t a reason for inaction. Someone still needs to step up and fight the emerging force. But I would say this is an argument for some degree of optimism.
AGI’s Opposing Force
In the Death Note anime, the Death Note gives the protagonist the ability to create a totalitarian justice system. That is, if there isn’t a force to stop him. But a character named L quickly catches onto him, and makes his ability to “serve justice” harder. In other words, the Death Note is a powerful tool (like AGI), but opposing forces hamper its ability to affect the world.
In the Death Note series, L only responded when he noticed the anomalous effects of the Death Note. He took a reactive approach.
When talking about AGI, usually the discussion gravitates towards “and then the uptick in intelligence goes out of control, leading to completely unpredictable behavior”. Which then leads to the conclusion that “in such chaotic situations, the chance that things go wrong is significantly higher than that all stars align”.
Although I agree with the sentiment, and see truth in all the ways AGI could be dangerous, sometimes we disregard the “L” in reasoning about AGI.
In nature, many processes evolve to completely chaotic situations unless constrained by opposing forces (Entropy Theory). Any species seeks dominance, until another species comes along to keep populations under control. Physical processes gather entropy till we step in and reduce it (hence why I have to clean my room every now and then). Even psychology shows these patterns, where self-serving thoughts radicalize, giving rise to wide-ranging mental issues, till a force stops them .
I often hear the argument “the required IQ to destroy the world decreases every year”. I don’t fully agree with that sentiment. As new technologies arise to do a lot of harm with little IQ, new constraints arise to reduce harm. Nuclear weapons are incredibly scary and a huge risk, so we have seen global nuclear disarmament, which shows in the sharp reduction of recent nuclear close calls. The IQ required to destroy the world through nuclear weapons has decreased because of the “L” that arose to oppose it.
As computer intelligence evolves, we are ramping up the “nuclear warheads” that machines have access to. We might need an “AGI close calls” Wikipedia page soon to document how this evolution leads to dangerous situations. There might be disastrous accidents involved. But, as nature dictates, an opposing force will come along.
In conclusion: Like Death Note’s L, any force in nature has its opposing forces. As AGI emerges and gains power, forces will arise that weaken its power. This isn’t a reason for inaction. Someone still needs to step up and fight the emerging force. But I would say this is an argument for some degree of optimism.