But post-FAI, how does anyone except the FAI have anything to offer? Neither anything to offer, nor anything to threaten with. The FAI decides all, does all, rules all. The question is, how should it rule? Since no creature besides the FAI has anything to offer, weighting is out of the equation, and every present, past, and potential creature’s utilities should count the same.
I think an FAI’s values would reflect the programmers’ values (unless it turns out there is Objective Morality or something else unexpected). My understanding now is that if Robin were the FAI’s programmer, the weights he would give to other people in its utility function would depend on how much they helped him create the FAI (and for people who didn’t help, how much the helpers care about them).
Sounds plenty selfish to me. Indeed, no different than might-is-right.
Instead of might-is-right, I’d summarize it as “might-and-the-ability-to-provide-services-to-others-in-exchange-for-what-you-want-is-right” and Robin would presumably emphasize the second part of that.
You can care a lot about other people no matter how much they help you, but should help those who helps you even more for game-theoretic reasons. This doesn’t at all imply “selfishness”.
But post-FAI, how does anyone except the FAI have anything to offer? Neither anything to offer, nor anything to threaten with. The FAI decides all, does all, rules all. The question is, how should it rule? Since no creature besides the FAI has anything to offer, weighting is out of the equation, and every present, past, and potential creature’s utilities should count the same.
I think an FAI’s values would reflect the programmers’ values (unless it turns out there is Objective Morality or something else unexpected). My understanding now is that if Robin were the FAI’s programmer, the weights he would give to other people in its utility function would depend on how much they helped him create the FAI (and for people who didn’t help, how much the helpers care about them).
Sounds plenty selfish to me. Indeed, no different than might-is-right.
Instead of might-is-right, I’d summarize it as “might-and-the-ability-to-provide-services-to-others-in-exchange-for-what-you-want-is-right” and Robin would presumably emphasize the second part of that.
You can care a lot about other people no matter how much they help you, but should help those who helps you even more for game-theoretic reasons. This doesn’t at all imply “selfishness”.