I feel like most of the bad incentives for developing unsafe AGI comes from the positive incentive gradient of marginally stronger AI. I.e. you could model it as a repeated game with hundreds of rounds, where every turn you get resources and power proportional to your total investment in capabilities. This seems to model reality much better (i.e. financial incentives are basically just short-term, and you don’t get to build a 100 person research team if you don’t show anything profitable from it), and also seems like where the real incentive problem stems from.
You might still think the analysis in the post is relevant if there are actors that can shape the incentive gradients you talk about: Google might be able to focus its sub-entities in a particular way while maintaining profit or a government might choose to implement more or less oversight over tech companies.
Even with the above paragraph, it seems like the relative change-over-time in resources and power of the strategic entities would be important to consider, as you point out. In this case, it seems like (known) fast takeoffs might be safer!
I feel like most of the bad incentives for developing unsafe AGI comes from the positive incentive gradient of marginally stronger AI. I.e. you could model it as a repeated game with hundreds of rounds, where every turn you get resources and power proportional to your total investment in capabilities. This seems to model reality much better (i.e. financial incentives are basically just short-term, and you don’t get to build a 100 person research team if you don’t show anything profitable from it), and also seems like where the real incentive problem stems from.
Yes, that seems an important case to consider.
You might still think the analysis in the post is relevant if there are actors that can shape the incentive gradients you talk about: Google might be able to focus its sub-entities in a particular way while maintaining profit or a government might choose to implement more or less oversight over tech companies.
Even with the above paragraph, it seems like the relative change-over-time in resources and power of the strategic entities would be important to consider, as you point out. In this case, it seems like (known) fast takeoffs might be safer!