The main way humans have the tragedy of the commons dealt with is by forming powerful forces that force us to treat the common resources in a restrained fashion. AGIs may have quite a bit of trouble making such entities, especially since burning the cosmic commons might be enough to allow an AGI to quickly overtake its fellow AGIs if the others are not willing to consume the resources.
OK, so a reason a group of AIs wouldn’t be able to do that is because the advantage of exploiting the commons might be nearly infinite. How likely is this?
If there were millions of AIs, what’s a scenario in which one gets so much more powerful than all the others combined by striking first, despite all of them guarding against it?
As AIs can merge, I would think refusal to do so and combine utility functions might be a first sign of intent to defect, that’s a warning humans never have.
Regardless, a million is a constant factor. Sufficient self-reinforcing development (as is kind of the point of seed AI) can outstrip any such factor. And the more self-reinforced the development of our AI pool becomes, the less relevant are “mere” constant factors.
I’m not saying it won’t work, but I wouldn’t like to bet on it.
Don’t worry, I’m not saying it would work! We might put similar odds on it, or maybe not—less than .5 and more than .001, I’m not sure about what the full range of fleshing out the possible scenarios would look like, but there’s probably a way in which I could fill in variables to end up with a .05 confidence in a specific scenario working out.
The main way humans have the tragedy of the commons dealt with is by forming powerful forces that force us to treat the common resources in a restrained fashion. AGIs may have quite a bit of trouble making such entities, especially since burning the cosmic commons might be enough to allow an AGI to quickly overtake its fellow AGIs if the others are not willing to consume the resources.
Humans have dealt with the tragedy of the commons and we can’t even merge resources and utility functions to become a new, larger entity!
We have dealt with TotC by imposing costs larger than the benefits that could be derived from abusing the commons.
The benefits an AI could derive from abusing the commons are possibly unlimited.
OK, so a reason a group of AIs wouldn’t be able to do that is because the advantage of exploiting the commons might be nearly infinite. How likely is this?
If there were millions of AIs, what’s a scenario in which one gets so much more powerful than all the others combined by striking first, despite all of them guarding against it?
As AIs can merge, I would think refusal to do so and combine utility functions might be a first sign of intent to defect, that’s a warning humans never have.
Regardless, a million is a constant factor. Sufficient self-reinforcing development (as is kind of the point of seed AI) can outstrip any such factor. And the more self-reinforced the development of our AI pool becomes, the less relevant are “mere” constant factors.
I’m not saying it won’t work, but I wouldn’t like to bet on it.
Don’t worry, I’m not saying it would work! We might put similar odds on it, or maybe not—less than .5 and more than .001, I’m not sure about what the full range of fleshing out the possible scenarios would look like, but there’s probably a way in which I could fill in variables to end up with a .05 confidence in a specific scenario working out.