I’m not worried by the result because there are two very implausible constraints: the number of possible utility functions and the utility of the compromise strategy. Given that there are, in fact, many possible utility functions, it seems really really unlikely that there is a strategy that has 3⁄10 the utility of the optimal strategy for every possible utility function. Additionally, some pairs of utility functions won’t be conducive to high-utility compromise strategies. For example: what if one civilization has paperclip maximization as a value, and another has paperclip minimization as a value?
ETA: Actually, this reasoning is somewhat wrong. The compromise strategy just has to have average utility > x/n, where n is the number of utility functions, and x is the average utility of the optimal strategies for each utility function. (In the context of the original example, n = 10 and x = 10. So the compromise strategy just has to be, on average, > 1 for all utility functions, which makes sense.) I still submit that a compromise strategy having this utility is unlikely, but not as unlikely as I previously argued.
The compromise strategy just has to have average utility > x/n
I’m still not sure this is right. You have to consider not just fi(Si) but all the fi(Sj)‘s as well, i.e. how well each strategy scores under other planets’ utility functions. So I think the relevant cutoff here is 1.9 - a compromise strategy that does better than that under everyone’s utility function would be a win-win-win. The number of possible utility functions isn’t important, just their relative probabilities.
You’re right that it’s far from obvious that such a compromise strategy would exist in real life. It’s worth considering that the utility functions might not be completely arbitrary, as we might expect some of them to be a result of systematizing evolved social norms. We can exclude UFAI disasters from our reference class—we can choose who we want to play PD with, as long as we expect them to choose the same way.
It’s a toy example but doesn’t it still apply if you have an estimate of the expected distribution of instances that will actually be implemented within mind space? The space of possible minds is vast, but the vast majority of those minds will not be implemented (or extremely less often). The math would be much more difficult but couldn’t you still estimate it in principle? I don’t think your criticism actually applies.
I’m not worried by the result because there are two very implausible constraints: the number of possible utility functions and the utility of the compromise strategy. Given that there are, in fact, many possible utility functions, it seems really really unlikely that there is a strategy that has 3⁄10 the utility of the optimal strategy for every possible utility function. Additionally, some pairs of utility functions won’t be conducive to high-utility compromise strategies. For example: what if one civilization has paperclip maximization as a value, and another has paperclip minimization as a value?
ETA: Actually, this reasoning is somewhat wrong. The compromise strategy just has to have average utility > x/n, where n is the number of utility functions, and x is the average utility of the optimal strategies for each utility function. (In the context of the original example, n = 10 and x = 10. So the compromise strategy just has to be, on average, > 1 for all utility functions, which makes sense.) I still submit that a compromise strategy having this utility is unlikely, but not as unlikely as I previously argued.
I’m still not sure this is right. You have to consider not just fi(Si) but all the fi(Sj)‘s as well, i.e. how well each strategy scores under other planets’ utility functions. So I think the relevant cutoff here is 1.9 - a compromise strategy that does better than that under everyone’s utility function would be a win-win-win. The number of possible utility functions isn’t important, just their relative probabilities.
You’re right that it’s far from obvious that such a compromise strategy would exist in real life. It’s worth considering that the utility functions might not be completely arbitrary, as we might expect some of them to be a result of systematizing evolved social norms. We can exclude UFAI disasters from our reference class—we can choose who we want to play PD with, as long as we expect them to choose the same way.
It’s a toy example but doesn’t it still apply if you have an estimate of the expected distribution of instances that will actually be implemented within mind space? The space of possible minds is vast, but the vast majority of those minds will not be implemented (or extremely less often). The math would be much more difficult but couldn’t you still estimate it in principle? I don’t think your criticism actually applies.