The point is that sanctions should be applied as necessary to discourage AGI, however, approximate grim triggers should apply as needed to prevent dystopia.
As the other commentators have mentioned, my reaction is not unusual and thus this is why the concerns of doom have been widespread.
So the answer is: enough.
I don’t think it is magic but it is still sufficiently disgusting to treat it with equal threat now. Red button now.
Its not a good idea to treat a disease right before it kills you: prevention is the way to go.
So no, I don’t think it is magic. But I do think just as the world agreed against human cloning long before there was a human clone, now is the time to act.
I’ll look for the article later but basically the Air Force has found pilotless aircraft to be useful for around thirty years but organized rejection has led to most such programs meeting an early death.
The rest is a lot of AGI is magic without considering the actual costs of computation or noncomputable situations. Nukes would just scale up: it costs much less to destroy than it is to build and the significance of modern economics is indeed that they require networks which do not take shocks well. Everything else basically is “ASI is magic.”
I would bet on the bomb.
This frames things as an inevitability which is almost certainly wrong, but more specifically opposition to a technology leads to alternatives being developed. E.g. widespread nuclear control led to alternatives being pursued for energy.
Being controllable is unlikely even if it is tractable by human controllers: it still represents power which means it’ll be treated as a threat by established actors and its terroristic implications mean there is moral valence to police it.
In a world with controls, grim triggers or otherwise, AI would have to develop along different lines and likely in ways that are more human compatible. In a world of intense grim triggers, it may be that is too costly to continue to develop beyond a point. “Don’t build ASI or we nuke” is completely reasonable if both “build ASI” and “nuking” is negative, but the former is more negative.
Autonomous weapons actually are an excellent example of delay: despite excellent evidence of the superiority of drones, pilots have continued to mothball it for at least 40 years and so have governments in spite of wartime benefits.
The argument seems to similar to the flaw in the “billion year” argument: we may die eventually, but life only persists by resisting death, long enough for it to replicate.
As far as real world utility, notwithstanding some recent successes, going down without fighting for myself and my children is quite silly.
He discussed it here:
https://youtu.be/Ufm85wHJk5A?list=PLQk-vCAGvjtcMI77ChZ-SPP—cx6BWBWm
No, I wouldn’t want it even if it was possible since by nature it is a replacement of humanity. I’d only accept Elon’s vision of AI bolted onto humans, so it effectively is part of us and thus can be said to be an evolution rather than replacement.
My main crux is that humanity has to be largely biological due to holobiont theory. There’s a lot of flexibility around that but anything that threatens that is a nonstarter.
Lead is irrelevant to human extinction, obviously. The first to die is still dead.
In a democratic world, those affected have a say in how they should be inflicted with AI and how much they want to die or suffer.
The government represents the people.
I think even the wealthy supporters of it are more complex: I was surprised that Palantir’s Peter Thiel came out discussing how AI “must not be allowed to surpass the human spirit” even as he clearly is looking to use AI in military operations. This all suggests significant controls incoming, even from those looking to benefit from it.
The UK has already mentioned that perhaps there should be a ban on models above a certain level. Though it’s not official, I have pretty good record that Chinese party members have already discussed worldwide war as potentially necessary(Eric Hoel also mentioned it, separately). Existential risk has been mentioned and of course, national risk is already a concern, so even for “mundane” reasons, it’s a matter of priority/concern and grim triggers are a natural consequence.
Elon had a personal discussion with China recently as well, and given his well known perspective on the dangers of AI, I expect that this point of view has only been reinforced.
And this is with barely reasoning chatbots!
As for Luddites, I don’t see why inflicting dystopia upon humanity because it fits some sort of cute agenda has any good purpose. But notably the Luddites did not have the support of the government and the government was not threatened by textile mills. Obviously this isn’t the case with nuclear, AI or bio. We’ve seen slowdowns on all of those.
“Worlds change” has no meaning: human culture and involvement influence the change of the world.
I find it happy that we probably don’t have enough compute and it is likely this will be restricted even at this fairly early level, long before more extreme measures are needed.
Additionally, I think one should support the Grim Trigger even if you want ASI, because it forces development along more “safe” lines to prevent being Grimmed. It also encourages non-ASI advancement as alternate routes, effectively being a form of regulation.
ASI is unnecessary when we have other options and grim game dynamics apply to avoid extinction or dystopia. I find even most such descriptions of tool level AI as disgusting(as do many others, I find).
Inevitability only applies if we have perfect information about the future, which we do not.
If it was up to me alone, I think we can give it at least a thousand years. Perhaps we can first raise the IQ of humanity by 1 SD via simple embryo selection before we go about extinctioning ourselves.
I actually do not think that we’re that close to cracking AGI: however, the intensity of the reaction imo is an excellent litmus test of how disgusting it is to most.
I strongly suspect the grim game dynamics have already begun, too, which has been one reason I’ve found comfort in the future.
Humans building AGI ASI likely leads to human extinction.
I disagree: we have many other routes of expansion, including biological improvement, cyborgism, etc. This seems akin to a cultic thinking and akin to Spartan ideas of “only hoplite warfare must be adopted or defeat ensues.”
The “limitations of physics” is quite extensive, and applies even to the pipeline leading up to anything like ASI. I am quite confident that any genuine dedication to the grim game would be more than enough to prevent it, and defiance of it leads to much more likelihood of nuclear winter worlds than ASI dominance.
But I also disagree on your prior of “this world in months”, I suppose we will see in December.
Notably they also have not used nuclear weaponry recently and overall nuclear stockpiles have decreased by 80 percent. Part of playing the grim game is not giving the other player reasons to go grim by defecting. Same goes for ASI: they can suppress each other but if one defects, the consequences is that they can’t benefit.
The mutual result is actually quite stable with only government control as their incentives against self-destruction is high.
Basically only North Korea-esque nations in this scenario have the most incentive to defect, but would be suppressed by all extant powers. Since they would be essentially seen as terrorist speciciders, it’s hard to see why any actions against them wouldn’t be justified.
In the event of such a war, there is no labor and there is no supply chain for microchips. The result has been demonstrated historically: technological reversion.
Technology isn’t magic: it’s the result of capital inputs and trade, and without large scale interconnection, it’ll be hard to make modern aircraft, let alone high quality chips. In fact, we personally experienced this from the very minimal disruption of COVID to supply chains. The killer app in this world would the widespread use of animal power, not robots, due to overall lower energy provisions.
And since the likely result would be what I want, but since I’m dead, I wouldn’t be bothered one way or another and therefore there is even more reason for me to punish the defector. This also sets precedent to others that this form of punishment is acceptable and increases the likelihood of it.
This is pretty simple game theory known as the grim game and is essential to a lot of life as a whole tbh.
Converging timelines is as irrelevant as a billion years. I(or someone like me) will do it as many times as needed, just like animals try to resist extinction via millions of “timelines” or lives.
Disagree: since building ASI results in dystopia even if I win in this scenario, the correct choice is to push the red button and ensure that no one has it. While I might die, this likely ensures humanity to survive.
The payoff in this case is maximal(unpleasant but realistic future for humanity) versus total loss(dystopia/extinction).
Many arguments here it seems feels like come from a near total terror of death while game theory clearly has always demonstrated against that: the reason why deterrence works is the confidence that a “spiteful action” to equally destroy an defecting adversary is expected, even if it results in personal death.
In this case, one nation pursuing the extinction of humanity would necessarily expect to be sent into extinction so that at least it cannot benefit from defection.
As others have mentioned, this entire line of reasoning is grotesque and sometimes I wonder if it is performative. Coordinating against ASI and dying of old age is completely reasonable as it’ll increase the odds of your genetic replacements remaining while technology continues to advance along safer routes
The alternate gamble of killing everyone is so insane that full scale nuclear war which will destroy all supply chains for ASI seems completely justified. While it’ll likely kill 90 percent of humanity, the remaining population will survive and repopulate sufficiently.
One billion years is not a reasonable argument for taking risks to end humanity now: extrapolated sufficiently, it would be the equivalent of killing yourself now because the heat death of the universe is likely.
We will always remain helpless against some aspects of reality, especially what we don’t know about: for all we know, there is damage to spacetime in our local region.
This is not an argument to risk the lives of others who do not want to be part of this. I would violently resist this and push the red button on nukes, for one.
It will still take awhile for AGI to get to that point, and Chinese and American coordination would pretty easily disrupt any rivals who try for that: they would essentially be terrorist actors endangering the world and the appropriate sanctions would be handed out.
I apologize for being a normie but I can’t accept anything that involves non-existence of humanity and would indeed accept an enormous amount of suffering if those were the options.
Depends on the degree of suffering to be totally honest- obviously I’m fine with them suffering to some extent, which is why we drive then to behave, etc so they can have better futures and sometimes conjoin them to have children so that we can continue the family line.
I think my answer actually is yes, if hypothetically their suffering allows the existence of 40th century humans, it’s pretty noble and yes, I’d be fine with it.
I would campaign against lead pipes and support the goths in destroying Rome which likely improved human futures over an alternative of widespread lead piping.