I agree the ant analogy is flawed. But I don’t think it’s as flawed as you do.
In this scenario, the ‘trade’ we would make would plausibly be “do this stuff or we kill you”, which is not amazing for the ants.
I think another disanalogy is that humans can’t re-arrange ants to turn them into better trading partners (or just raw materials), but AI could do that to us. (h/t to Dustin Crummett for reminding me of this). And the fact that we might not be able to understand fancy AI concepts could make this option more appealing.
In this scenario, the ‘trade’ we would make would plausibly be “do this stuff or we kill you”, which is not amazing for the ants.
It costs money to kill ants with ant poison. If the ants would accept a cheaper amount of food to evacuate my house forever, I would take that trade.
Similarly, it requires resources (compute, money, energy, etc) for an AGI to kill all humans or recursively improve. If the humans would accept a cheaper quantity of resources to help an AGI with its goals, the AGI might accept that trade?
If the ants accept the trade “leave and I’ll spare you,” I don’t have to spend any money on actually poisoning the ants. But I would consider the counteroffer “if you kill us, it will cost $20, and we’re willing to leave for $1.”
I think that if ants were smart enough to make that counter-offer, humans would probably regard them as smart enough to be blameworthy for invading the house in the first place, and the counter-offer would be rejected as extortion.
Analogy: Imagine some humans from country A move into country B and start living there. Country B says “we didn’t give you permission to live in our country; leave or we’ll kill you”. The humans say “killing us would cost you $20k; we’ll leave if you pay us $1k.” How do you predict this negotiation ends?
Now, if we’re talking about asking the ants to vacate an empty lot where they’ve lived for many years so that you can start building a new house there, then I could see humans paying the ants to leave. (Though note that the ants may still lose more value by giving up their hive than the humans are willing to pay them to avoid the cost of exterminating them.)
There are lots and lots of good reasons to recursively self improve. The point where you stop because of resources is a dyson sphere of quantum computronium.
I am not convinced that the resource cost of killing all humans is > the resource cost of 1 day’s food.
“If the humans would accept a cheaper quantity of resources to help the AI with it’s goals” The AI has goals that clearly oppose human wellbeing, and is offering us peanuts.
“It takes some resources.” is I think not a great model at all. I think you are modeling the system as having resources that are in the AI’s control or humans control. But the AI taking over may well have the structure of a computer exploit. A bunch of seeming coincidences that push the world into an increasingly strange state.
There is no sense of “this money/energy is controlled by humans, that is controlled by AI”. The powerplant was built by humans. The LHC was built by humans. But the magnet control system was hacked, and a few people have been given subtle psycological nudges. In this model, how much resources does it cost to spoof a nuclear attack and trick the humans into a nuclear war? The large amount of damage done, the amount of uranium used or the tiny amount of compute used to form the plan? There is no “cost of resources” structure to this interaction.
Ants are tiny and hard to find; they could plausibly take your money, defect, and keep eating for a long time before you found them again. Then you need to buy ant poison, anyway.
I agree the ant analogy is flawed. But I don’t think it’s as flawed as you do.
In this scenario, the ‘trade’ we would make would plausibly be “do this stuff or we kill you”, which is not amazing for the ants.
I think another disanalogy is that humans can’t re-arrange ants to turn them into better trading partners (or just raw materials), but AI could do that to us. (h/t to Dustin Crummett for reminding me of this). And the fact that we might not be able to understand fancy AI concepts could make this option more appealing.
It costs money to kill ants with ant poison. If the ants would accept a cheaper amount of food to evacuate my house forever, I would take that trade.
Similarly, it requires resources (compute, money, energy, etc) for an AGI to kill all humans or recursively improve. If the humans would accept a cheaper quantity of resources to help an AGI with its goals, the AGI might accept that trade?
If the ants believe the threat, you don’t have to spend any money on actually poisoning the ants.
If the ants accept the trade “leave and I’ll spare you,” I don’t have to spend any money on actually poisoning the ants. But I would consider the counteroffer “if you kill us, it will cost $20, and we’re willing to leave for $1.”
I think that if ants were smart enough to make that counter-offer, humans would probably regard them as smart enough to be blameworthy for invading the house in the first place, and the counter-offer would be rejected as extortion.
Analogy: Imagine some humans from country A move into country B and start living there. Country B says “we didn’t give you permission to live in our country; leave or we’ll kill you”. The humans say “killing us would cost you $20k; we’ll leave if you pay us $1k.” How do you predict this negotiation ends?
Now, if we’re talking about asking the ants to vacate an empty lot where they’ve lived for many years so that you can start building a new house there, then I could see humans paying the ants to leave. (Though note that the ants may still lose more value by giving up their hive than the humans are willing to pay them to avoid the cost of exterminating them.)
There are lots and lots of good reasons to recursively self improve. The point where you stop because of resources is a dyson sphere of quantum computronium.
I am not convinced that the resource cost of killing all humans is > the resource cost of 1 day’s food.
“If the humans would accept a cheaper quantity of resources to help the AI with it’s goals” The AI has goals that clearly oppose human wellbeing, and is offering us peanuts.
“It takes some resources.” is I think not a great model at all. I think you are modeling the system as having resources that are in the AI’s control or humans control. But the AI taking over may well have the structure of a computer exploit. A bunch of seeming coincidences that push the world into an increasingly strange state.
There is no sense of “this money/energy is controlled by humans, that is controlled by AI”. The powerplant was built by humans. The LHC was built by humans. But the magnet control system was hacked, and a few people have been given subtle psycological nudges. In this model, how much resources does it cost to spoof a nuclear attack and trick the humans into a nuclear war? The large amount of damage done, the amount of uranium used or the tiny amount of compute used to form the plan? There is no “cost of resources” structure to this interaction.
Ants are tiny and hard to find; they could plausibly take your money, defect, and keep eating for a long time before you found them again. Then you need to buy ant poison, anyway.