There are lots and lots of good reasons to recursively self improve. The point where you stop because of resources is a dyson sphere of quantum computronium.
I am not convinced that the resource cost of killing all humans is > the resource cost of 1 day’s food.
“If the humans would accept a cheaper quantity of resources to help the AI with it’s goals” The AI has goals that clearly oppose human wellbeing, and is offering us peanuts.
“It takes some resources.” is I think not a great model at all. I think you are modeling the system as having resources that are in the AI’s control or humans control. But the AI taking over may well have the structure of a computer exploit. A bunch of seeming coincidences that push the world into an increasingly strange state.
There is no sense of “this money/energy is controlled by humans, that is controlled by AI”. The powerplant was built by humans. The LHC was built by humans. But the magnet control system was hacked, and a few people have been given subtle psycological nudges. In this model, how much resources does it cost to spoof a nuclear attack and trick the humans into a nuclear war? The large amount of damage done, the amount of uranium used or the tiny amount of compute used to form the plan? There is no “cost of resources” structure to this interaction.
There are lots and lots of good reasons to recursively self improve. The point where you stop because of resources is a dyson sphere of quantum computronium.
I am not convinced that the resource cost of killing all humans is > the resource cost of 1 day’s food.
“If the humans would accept a cheaper quantity of resources to help the AI with it’s goals” The AI has goals that clearly oppose human wellbeing, and is offering us peanuts.
“It takes some resources.” is I think not a great model at all. I think you are modeling the system as having resources that are in the AI’s control or humans control. But the AI taking over may well have the structure of a computer exploit. A bunch of seeming coincidences that push the world into an increasingly strange state.
There is no sense of “this money/energy is controlled by humans, that is controlled by AI”. The powerplant was built by humans. The LHC was built by humans. But the magnet control system was hacked, and a few people have been given subtle psycological nudges. In this model, how much resources does it cost to spoof a nuclear attack and trick the humans into a nuclear war? The large amount of damage done, the amount of uranium used or the tiny amount of compute used to form the plan? There is no “cost of resources” structure to this interaction.