This kind of problem goes back to Bentham and the very beginning of utilitarianism. (Disclaimer: what follows is based on my recollections of an essay by A. J. Ayer on Bentham, read long ago. I cannot find it now online, nor find other discussions of Bentham making the same points, and I am not any kind of expert on the matter. So when I say below “Bentham believed…”, this could be true, or could be true only in Ayer’s interpretation, or even only in my own interpretation of Ayer’s interpretation.)
Bentham believed both that each person pursues only their own happiness, and that the good consists of the creates happiness of the greatest number. (The first if these could correspond in your formulation of the dilemma to saying that agents have utility functions and behave rationally according to them, and the second to a statement of preference utilitarianism.) Then the problem comes up of how are we be utilitarians and try to maximize global happiness if we are psychologically necessitated to care only about our own. Bentham’s solution is to postulate a “lawgiver”, a person whose happiness is greatest when everyone’s happiness is maximized, and say that utilitarianism as a political prescription says that laws should be made by this lawgiver. (In LW-language this could correspond to FAI!)
Translating back Bentham’s solution (or my memory of Ayer’s interpretation of it) back to your question, I think the answer would be that utility functions don’t change; if you are truly a preference utilitarian, then your utility function is already given by the ultimate fixed point of the iteration process you have outlined, so there is no dynamical changing.
Translating back Bentham’s solution (or my memory of Ayer’s interpretation of it) back to your question, I think the answer would be that utility functions don’t change; if you are truly a preference utilitarian, then your utility function is already given by the ultimate fixed point of the iteration process you have outlined, so there is no dynamical changing.
Not sure how this solves anything though. In a society with non-utilitarian defectors, they will still end up determining the fixed points. So the behavior of the society is determined by its least moral members (by the utilitarians’ own lights). Is the response supposed to be that once you do away with the dynamical changing process there is no effective distinction any more between utilitarians and defectors? That would only seem to solve things at the price of massive non-realism. There does seem to be a pretty clear (and morally relevant) real-world distinction between agents who alter their behavior (altruistically) upon learning about the utility functions of others and agents who don’t. I don’t think hand-waving that distinction away in your model is all that helpful.
Alejando1′s point was that Bentham expected everyone to be a “defector”, in your terminology, and but that lawmakers should be given selfish incentives to maximize the sum of everyone’s utility. Although it is unclear to me who could be motivated to ensure that the lawmakers’ incentives are aligned with everyone’s utility if they are all just concerned with maximizing their own utility.
Also, as long as we’re talking about utilitarianism as described by Bentham, it’s worth pointing out that by “utility”, Bentham meant happiness, rather than the modern decision-theory formulation of utility. According to Alejando, if I understand him correctly, Bentham just sort of assumed that personal happiness is all that motivated anyone.
This kind of problem goes back to Bentham and the very beginning of utilitarianism. (Disclaimer: what follows is based on my recollections of an essay by A. J. Ayer on Bentham, read long ago. I cannot find it now online, nor find other discussions of Bentham making the same points, and I am not any kind of expert on the matter. So when I say below “Bentham believed…”, this could be true, or could be true only in Ayer’s interpretation, or even only in my own interpretation of Ayer’s interpretation.)
Bentham believed both that each person pursues only their own happiness, and that the good consists of the creates happiness of the greatest number. (The first if these could correspond in your formulation of the dilemma to saying that agents have utility functions and behave rationally according to them, and the second to a statement of preference utilitarianism.) Then the problem comes up of how are we be utilitarians and try to maximize global happiness if we are psychologically necessitated to care only about our own. Bentham’s solution is to postulate a “lawgiver”, a person whose happiness is greatest when everyone’s happiness is maximized, and say that utilitarianism as a political prescription says that laws should be made by this lawgiver. (In LW-language this could correspond to FAI!)
Translating back Bentham’s solution (or my memory of Ayer’s interpretation of it) back to your question, I think the answer would be that utility functions don’t change; if you are truly a preference utilitarian, then your utility function is already given by the ultimate fixed point of the iteration process you have outlined, so there is no dynamical changing.
Not sure how this solves anything though. In a society with non-utilitarian defectors, they will still end up determining the fixed points. So the behavior of the society is determined by its least moral members (by the utilitarians’ own lights). Is the response supposed to be that once you do away with the dynamical changing process there is no effective distinction any more between utilitarians and defectors? That would only seem to solve things at the price of massive non-realism. There does seem to be a pretty clear (and morally relevant) real-world distinction between agents who alter their behavior (altruistically) upon learning about the utility functions of others and agents who don’t. I don’t think hand-waving that distinction away in your model is all that helpful.
Alejando1′s point was that Bentham expected everyone to be a “defector”, in your terminology, and but that lawmakers should be given selfish incentives to maximize the sum of everyone’s utility. Although it is unclear to me who could be motivated to ensure that the lawmakers’ incentives are aligned with everyone’s utility if they are all just concerned with maximizing their own utility.
Also, as long as we’re talking about utilitarianism as described by Bentham, it’s worth pointing out that by “utility”, Bentham meant happiness, rather than the modern decision-theory formulation of utility. According to Alejando, if I understand him correctly, Bentham just sort of assumed that personal happiness is all that motivated anyone.
Yes, I remember Ayer making explicit this assumption of Bentham and criticizing it as either untrue or vacuous, depending on interpretation.