I know this post is a spoof, but the error you are pointing out is the “spherical cow” one. Where an AI system has all these capabilities, exists in a vacuum where there are no slightly weaker AI systems under human control, or earlier betrayal attempts from much weaker AIs that have taught humans how to handle them. Where it is even possible to just develop arbitrarily powerful capabilities without the “scaffolding” of a much broader tech base than humans have today.
And yes, if the AI system is a spherical cow with the density of a black hole it will doom the earth. By simply declaring such a thing exists your argument is tautologically correct.
That is, if I just say “and then a cow sized black hole exists” of course we are doomed. But like black holes are insanely hard to build, there would need to be machinery across the solar system at the point, one way to build one is to slam relativistic mountains together. Such a civilization probably doesn’t need the earth.
Right?
For specifics, if by the time the really strong AI exists humans already have nanotechnology and sniffers to detect pathogens and rogue nanobots in the air and fleets of drones monitoring all of the land and seas, then rebel AIs face a much greater challenge. If humans already have trustworthy weaker AIs they can give tasks to, they probably won’t be scammed by delegating large complex efforts to AI, especially ones that are black boxes and constantly self modifying.
Aka : Rogue ASI : psst I can solve this problem for you.
Humans: nah fam I got this. I already have a fleet of 100 separate stateless AI instances working on the task and making progress. I don't need to trust a shady ASI with the task. Trusting ASI? Not even once.
(Low confidence:) think you hit the nail on the head. It’s all about the path you take and what you have available to you at step n-1. I think this is not much discussed because it’s hard to be objective or rigorous. Although it’s getting light with redwoods latest stuff.
I know this post is a spoof, but the error you are pointing out is the “spherical cow” one. Where an AI system has all these capabilities, exists in a vacuum where there are no slightly weaker AI systems under human control, or earlier betrayal attempts from much weaker AIs that have taught humans how to handle them. Where it is even possible to just develop arbitrarily powerful capabilities without the “scaffolding” of a much broader tech base than humans have today.
And yes, if the AI system is a spherical cow with the density of a black hole it will doom the earth. By simply declaring such a thing exists your argument is tautologically correct.
That is, if I just say “and then a cow sized black hole exists” of course we are doomed. But like black holes are insanely hard to build, there would need to be machinery across the solar system at the point, one way to build one is to slam relativistic mountains together. Such a civilization probably doesn’t need the earth.
Right?
For specifics, if by the time the really strong AI exists humans already have nanotechnology and sniffers to detect pathogens and rogue nanobots in the air and fleets of drones monitoring all of the land and seas, then rebel AIs face a much greater challenge. If humans already have trustworthy weaker AIs they can give tasks to, they probably won’t be scammed by delegating large complex efforts to AI, especially ones that are black boxes and constantly self modifying.
Aka : Rogue ASI : psst I can solve this problem for you.
(Low confidence:) think you hit the nail on the head. It’s all about the path you take and what you have available to you at step n-1. I think this is not much discussed because it’s hard to be objective or rigorous. Although it’s getting light with redwoods latest stuff.