I think the best arguments are those about the costs to the AI of being nice. I don’t believe the AI will be nice at all because neglect is so much more profitable computation-wise.
This is because even processing the question of how much sunlight to spare humanity probably costs more in expectation than the potential benefit of that sunlight to the AI.
First and least significant, consider that niceness is an ongoing cost. It is not a one-time negotiation to spare humanity 1% of the sun; more compute will have to be spent on us in the future. That compute will have to be modeled and accounted for, but we can expect that the better humanity does, the more compute will have to be dedicated to us.
Second and more significant, what about time discounting? The proportion of compute that would have to be dedicated to niceness is highest right in the beginning, when humanity is largest relative to the AI. Since the cost is highest right at first, this suggests the AI is unlikely to engage in it at all.
Third and most significant, why should we believe this to be true? Because it seems to me to already be true of basically everything:
Polynomial equations get harder as you add terms.
The curse of dimensionality.
A pendulum is easy, but a double pendulum is hard.
More levels in multi-level modeling is harder.
Game theory outcomes are harder to solve with multiple players.
Making decisions with a large group is famously hard, and the first rule of fast decisions is to keep the group small.
Even within the boundaries of regular human-level paper computations it feels like the hit here is huge on aggregate. The presence of humans makes a bunch of places where: zeroes or infinities can’t be used to simplify; matrices can no longer be diagonalized; fewer symmetries will be available, etc. In short, I expect niceness to result in a systematic-though-not-complete loss of compute-saving moves through all layers of abstraction.
This isn’t reserved for planning or world-modeling style computation either; these constraints and optimizations are already bedrock assumptions that go into the hardware design, system software design, and neural net/whatever other design the AI will launch with; in other words these considerations are baked into the entire history of any prospective AI.
I think the best arguments are those about the costs to the AI of being nice. I don’t believe the AI will be nice at all because neglect is so much more profitable computation-wise.
This is because even processing the question of how much sunlight to spare humanity probably costs more in expectation than the potential benefit of that sunlight to the AI.
First and least significant, consider that niceness is an ongoing cost. It is not a one-time negotiation to spare humanity 1% of the sun; more compute will have to be spent on us in the future. That compute will have to be modeled and accounted for, but we can expect that the better humanity does, the more compute will have to be dedicated to us.
Second and more significant, what about time discounting? The proportion of compute that would have to be dedicated to niceness is highest right in the beginning, when humanity is largest relative to the AI. Since the cost is highest right at first, this suggests the AI is unlikely to engage in it at all.
Third and most significant, why should we believe this to be true? Because it seems to me to already be true of basically everything:
Polynomial equations get harder as you add terms.
The curse of dimensionality.
A pendulum is easy, but a double pendulum is hard.
More levels in multi-level modeling is harder.
Game theory outcomes are harder to solve with multiple players.
Making decisions with a large group is famously hard, and the first rule of fast decisions is to keep the group small.
Even within the boundaries of regular human-level paper computations it feels like the hit here is huge on aggregate. The presence of humans makes a bunch of places where: zeroes or infinities can’t be used to simplify; matrices can no longer be diagonalized; fewer symmetries will be available, etc. In short, I expect niceness to result in a systematic-though-not-complete loss of compute-saving moves through all layers of abstraction.
This isn’t reserved for planning or world-modeling style computation either; these constraints and optimizations are already bedrock assumptions that go into the hardware design, system software design, and neural net/whatever other design the AI will launch with; in other words these considerations are baked into the entire history of any prospective AI.
In sum, we die by Occam’s Razor.