So, the analogy here is that there’s hundreds (or more) of Godzillas all running around, doing whatever it is Godzillas want to do. Humanity helps out whatever Godzillas humanity likes best, which in turn creates an incentive for the Godzillas to make humanity like them.
THIS DOES NOT BODE WELL FOR TOKYO’S REAL ESTATE MARKET.
Still within the analogy: part of the literary point of Godzilla is that humanity’s efforts to fight it are mostly pretty ineffective. In inter-Godzilla fights, humanity is like an annoying fly buzzing around. The humans just aren’t all that strategically relevant. Sure, humanity’s assistance might add some tiny marginal advantage, but from a Godzilla’s standpoint that advantage is unlikely to be enough to balance the tactical/strategic disadvantages of trying not to step on people.
… and that all seems like it should carry over directly to AI, once AI gets to-or-somewhat-past human level, and definitely by the time we get to strongly superhuman intelligence. Even with just human level, the scaling/coordination/learning advantages of being able to cheaply copy a mind are probably enough for the AIs to reasonably-quickly achieve strategic dominance by enough margin that humanity’s preferences are not particularly relevant. (Assuming that the AI isn’t prohibitively expensive to run—but that seems pretty likely to be true under most plausible paths. For instance, if human-level AI is produced by anything like today’s ML, then training costs will dominate and the systems will be relatively cheap to run or fine-tune.)
(There’s also some alignment-specific problems with this scheme which the Godzilla analogy doesn’t highlight. I’m not going into them here because this post is specifically about the Godzilla issues. But I don’t want to give people the impression that this plan would be fine in a world where humanity has sufficient bargaining power; the lack of bargaining power is only one failure mode.)
So, the analogy here is that there’s hundreds (or more) of Godzillas all running around, doing whatever it is Godzillas want to do. Humanity helps out whatever Godzillas humanity likes best, which in turn creates an incentive for the Godzillas to make humanity like them.
THIS DOES NOT BODE WELL FOR TOKYO’S REAL ESTATE MARKET.
Still within the analogy: part of the literary point of Godzilla is that humanity’s efforts to fight it are mostly pretty ineffective. In inter-Godzilla fights, humanity is like an annoying fly buzzing around. The humans just aren’t all that strategically relevant. Sure, humanity’s assistance might add some tiny marginal advantage, but from a Godzilla’s standpoint that advantage is unlikely to be enough to balance the tactical/strategic disadvantages of trying not to step on people.
… and that all seems like it should carry over directly to AI, once AI gets to-or-somewhat-past human level, and definitely by the time we get to strongly superhuman intelligence. Even with just human level, the scaling/coordination/learning advantages of being able to cheaply copy a mind are probably enough for the AIs to reasonably-quickly achieve strategic dominance by enough margin that humanity’s preferences are not particularly relevant. (Assuming that the AI isn’t prohibitively expensive to run—but that seems pretty likely to be true under most plausible paths. For instance, if human-level AI is produced by anything like today’s ML, then training costs will dominate and the systems will be relatively cheap to run or fine-tune.)
(There’s also some alignment-specific problems with this scheme which the Godzilla analogy doesn’t highlight. I’m not going into them here because this post is specifically about the Godzilla issues. But I don’t want to give people the impression that this plan would be fine in a world where humanity has sufficient bargaining power; the lack of bargaining power is only one failure mode.)