Categorising the ways that the strategy-stealing assumption can fail:
It is intrinsically easier to gather flexible influence in pursuit of some goals, because
1. It’s easier to build AIs to pursue goals that are easy to check.
3. It’s easier to build institutions to pursue goals that are easy to check.
9. It’s easier to coordinate around simpler goals.
plus 4 and 5 insofar as some values require continuously surviving humans to know what to eventually spend resources on, and some don’t.
plus 6 insofar as humans are otherwise an important part of the strategic environment, such that it’s beneficial to have values that are easy-to-argue.
Jessica Taylor’s argument require that the relevant games are zero sum. Since this isn’t true in the real world:
7. A threat of destroying value (e.g. by threatening extinction) could be used as a bargaining tool, with unpredictable outcomes.
~8. Some groups actively wants other groups to have less resources, in which case they can try to reduce the total amount of resources more or less actively.
~8. Smaller groups have less incentive to contribute to public goods (such as not increasing the probability of extinction), but benefit equally from larger groups’ contributions, which may lead them to getting a disproportionate fraction of resources by defecting in public-goods games.
Humans don’t just care about acquiring flexible long-term influence, because
4. They also want to stay alive.
5 and 6. They want to stay in touch with the rest of the world without going insane.
11. and also they just have a lot of other preferences.
Categorising the ways that the strategy-stealing assumption can fail:
It is intrinsically easier to gather flexible influence in pursuit of some goals, because
1. It’s easier to build AIs to pursue goals that are easy to check.
3. It’s easier to build institutions to pursue goals that are easy to check.
9. It’s easier to coordinate around simpler goals.
plus 4 and 5 insofar as some values require continuously surviving humans to know what to eventually spend resources on, and some don’t.
plus 6 insofar as humans are otherwise an important part of the strategic environment, such that it’s beneficial to have values that are easy-to-argue.
Jessica Taylor’s argument require that the relevant games are zero sum. Since this isn’t true in the real world:
7. A threat of destroying value (e.g. by threatening extinction) could be used as a bargaining tool, with unpredictable outcomes.
~8. Some groups actively wants other groups to have less resources, in which case they can try to reduce the total amount of resources more or less actively.
~8. Smaller groups have less incentive to contribute to public goods (such as not increasing the probability of extinction), but benefit equally from larger groups’ contributions, which may lead them to getting a disproportionate fraction of resources by defecting in public-goods games.
Humans don’t just care about acquiring flexible long-term influence, because
4. They also want to stay alive.
5 and 6. They want to stay in touch with the rest of the world without going insane.
11. and also they just have a lot of other preferences.
(maybe Wei Dai’s point about logical time also goes here)