According to Aumann’s Agreement theorem, such a concurrence provides a tiny amount of Bayesian evidence that you’re onto something.
What? That’s… not AAT at all.
such pleasure-wizards, to put it bluntly, do not exist… But their opposites do.
What possible justification could he have for this? “No one is better at happiness than others, but some people are worse at happiness” is obviously impossible, and if the claim is that there’s a plateau of “normal” people who are all roughly equivalent at converting resources into happiness and then people who are worse than that plateau, that sounds more like wishful thinking than a justified empirical claim.
On closer inspection it was not hard to see why Carter had developed theories so close to those of Eliezer and other members of Less Wrong and SIAI communities.
They really don’t look that similar to me; they’re looking at very different problems and have very different approaches.
The basic problem is that utilitarianism simply doesn’t work.
Carter takes the common critique of total utilitarianism and the common critique of average utilitarianism, and says “well, both critiques go away if we try to maximize a combination of total and average.” But those are just the common critiques, not the most potent ones. The basic problem with utilitarianism is that utility is difficult to measure and impossible to compare- and so both total and average utilitarianism are not things that can actually be calculated.
Eliezer is trying to tackle the problem of what utilities actually cash out as, so that you can build a machine that can perform preference calculations and not get them horribly wrong. Will Alice be happier with an unlimited supply of cookies, or if she has to strive for them? The options satisfy different desires in different amounts, and so fun theory and complexity of value deal with the tradeoffs between different desires. If you just built a machine that knew about our desire to feel happy and didn’t know about our desire to impact the real world, you would get a population of wireheads- something that many of us think would be a bad outcome now, but cannot justify that judgment in terms of average or total ‘happiness.’
I’ve edited it to the version you said, as it’s cleaner for this discussion that way. In general I think I would separate the desire to impact and the desire for the map to match the territory.
The basic problem is that utilitarianism simply doesn’t work. ... Eliezer is trying to tackle the problem of what utilities actually cash out as, so that you can build a machine that can perform preference calculations and not get them horribly wrong. Will Alice be happier with an unlimited supply of cookies, or if she has to strive for them? The options satisfy different desires in different amounts, and so fun theory and complexity of value deal with the tradeoffs between different desires.
Utilitarianism, defined as an ethical theory that only has values involving happiness and suffering, doesn’t work. But values dealing with happiness and suffering are a subset of our values, which also include freedom, challenge, impact on the real world, friendship, novelty, and so on. Carter’s point can be extended fairly easily to these: just as we don’t only value total happiness but also average happiness, we don’t only value happiness but also all those other things. Carter was aiming to solve a problem with utilitarianism, but he managed to find a solution that extends to consequentialism in general.
But values dealing with happiness and suffering are a subset of our values, which also include freedom, challenge, impact on the real world, friendship, novelty, and so on.
Are those values an ideal consequentialist should have? Consider “I want to help people” and “I want people to be helped.”
I don’t think there’s any one “ideal consequentialist”. Some values may be more important to me than others, and I may want to self-modify to care less about some of those things in order to maximize the others, but no terminal values are themselves better or worse. My utility function is what it is.
The issue is that “freedom” and “challenge” aren’t really outcome preferences so much as they are action preferences or decision theory preferences. The consequentialist doesn’t see a difference between me proposing that we trade two of my apples for three of your lemons and a dictator ordering that we trade two of my apples for three of your lemons- the outcome is how many of which fruit each of us ends up with, and if the dictator is better at negotiating and knowing our preferences than we are, the consequentialist suggests that we use the dictator and get over our preference for freedom (which was useful when dictators were bad but isn’t useful when dictators are good).
You can smuggle other moral systems into consequentialism- by, say, including features of the decision tree as part of the outcome- but it’s far cleaner to just discard consequentialism.
I have different subjective experiences when I am making my own decisions and when I am doing something I was ordered to do, even if it’s the same decision and action both times. This suggests “my having freedom” is a real quality of a state of the world, and that therefore I can have consequentialist preferences about its presence vs. its absence. Anything that can distinguish two states of the world is a valid thing consequentialists can have values about.
In the trolley problem, an agent will have different subjective experiences in the case where they do nothing and in the case where they murder someone. Most consequentialist prescriptions count such preferences as insignificant in light of the other outcomes.
I do think that most consequentialists go further, claiming that only the final world state should matter and not how you got there, but I agree with you that consequentialist tools are powerful enough to adopt systems that are typically seen as competing- like deontology- and the reverse is true as well. Because there’s that flexibility in tools, I find conversations are easier if one adopts strict system definitions. If someone uses expected utility theory to pick an action, but their utility is based on the rules they followed in choosing actions, I don’t see the value in calling that consequentialism.
The consequentialist doesn’t see a difference between me proposing that we trade two of my apples for three of your lemons and a dictator ordering that we trade two of my apples for three of your lemons- the outcome is how many of which fruit each of us ends up with, and if the dictator is better at negotiating and knowing our preferences than we are, the consequentialist suggests that we use the dictator and get over our preference for freedom
I think your quandary can be resolved by dividing your example into more than one consequence.
Example 1 has the consequences:
Dictator tells you what to do.
You end up with +3 lemons and −2 apples.
Example 2 has the consequences:
You think hard and make a decision.
You end up with +3 lemons and −2 apples.
I’m making up numbers here, but imagine you assign +10 utility to the consequence “end up with +3 lemons and −2 apples,” +1 utility to the consequence “think hard and make a decision.” and −3 to the consequence “dictator tells me what to do.” Then in Example 1 the two consequences have a combined utility of 7, whereas in Example 2 they have a combined utility of 11.
In the trolley problem, an agent will have different subjective experiences in the case where they do nothing and in the case where they murder someone. Most consequentialist prescriptions count such preferences as insignificant in light of the other outcomes.
I think one reason that subjective experiences don’t matter in the trolly problem is that the stakes are so high. In the trolley problem your desire not to be involved in someone’s death is nothing compared to the desire of six people to not die. If the stakes were much lower, however, your subjective experiences might matter.
For instance, imagine a toned down trolley problem where if you do nothing Alice and Bob will get papercuts on their thumbs, and if you pull a switch Clyde will get a papercut on his thumb. In that case the stakes are low enough that the unpleasant feeling you get from pulling the switch and injuring someone might merit some consideration.
This is actually similar to how the preference for freedom is treated in real life. When the stakes are low freedom is respected more often, even if it sometimes leads to some bad consequences, but when they are higher (during war, viral epidemics, etc) freedom is restricted because the stakes are much higher. (Of course, it goes without saying that in real life treating freedom like this tends to encourage corruption)
My understanding was that if two equally rational people have the same information they will draw the same conclusions. So if someone draws the same conclusion as you, that’s evidence in your favor, but only very mild evidence since you don’t know for sure if they’re as rational as you and if they had the exact same info as you. I do think that might be a weak opening and am thinking of changing it.
What possible justification could he have for this? “No one is better at happiness than others, but some people are worse at happiness” is obviously impossible, and if the claim is that there’s a plateau of “normal” people who are all roughly equivalent at converting resources into happiness and then people who are worse than that plateau, that sounds more like wishful thinking than a justified empirical claim.
It’s perfectly true. People with severe medical problems need huge amount of resources merely to satisfy their preferences of “not being dead” and “not being sick.” A normal person require far less resources to satisfy that preference. So people with severe medical problems are “reverse utility monsters.” This is true regardless of whether you are a preference or happiness utilitarian, since obviously you need to be alive to be happy at all and healthy to reach your full happiness potential (FYI I’m a preference utilitarian).
The basic problem is that utilitarianism simply doesn’t work.
Normal_Anomaly made most of the points I was going to, so I won’t elaborate on this.
My understanding was that if two equally rational people have the same information they will draw the same conclusions.
AAT says that if two people (who may have observed different evidence) have mutual knowledge that they are both perfect epistemic rationalists honestly reporting their posterior probabilities, then they cannot remain in an equalibrium where they both are aware of the other’s posterior probabilities but disagree on their posterior probabilities.
I’m pretty sure they do need to have the same priors.
My intuition is that AAT is basically saying that the perfect epistemic rationalists involved can essentially transfer all of the evidence that they have to the other, so that each one effectively has the same evidence and so should have the same posteriors...except that they’ll still have different posteriors unless they began with the same priors.
If they found that they had different priors, I think that they could just communicate the evidence which led them to form those priors from previous priors and so forth, but I think that if they trace their priors as far back as possible and find that they have different ones, AAT doesn’t work.
I’m not actually super-familiar with it, so update accordingly if I seem to have said something dumb.
Nope, I was wrong. It is the case that agents require equal priors for ATT to hold. AAT is like proving that mixing the same two colors of paint will always result in the same shade or that two equal numbers multiplied by another number will still be equal.
What a worthless theorem!
I guess when I read that AAT required “common priors” I assumed Aumann must mean known priors or knowledge of each others’ priors, since equal priors would constitute both 1) an asinine assumption and, 2) a result not worth reporting. Hanson’s assumption that humans should have a shared prior by virtue of being evolved together is interesting, but more creative than informative.
Good thing I don’t rely on ATT for anything. It’s obvious that disagreeing with most people is rational so updating on people’s posteriors without evidence would be pretty unreasonable. I’m not surprised that ATT would turn out to be so meaningless.
AAT is very specific. Independent invention is evidence for the attractiveness of an idea (and thus, one hopes, its truthfulness) but it’s unrelated to AAT.
People with severe medical problems need huge amount of resources merely to satisfy their preferences of “not being dead” and “not being sick.”
I have edited my post to remove the reference to AAT per your and JGWeissman’s advice.
Sure. But you’re telling me that two healthy individuals are equally efficient at converting resources into happiness? What evidence is there that the Brahmin is not ten times as capable of happiness as the Untouchable?
In Carter’s paper he was discussing a hypothetical pleasure wizard so good at converting resources into happiness that it was better to give it all the resources in the world and let everyone else just have enough for a life barely worth living. It seems unlikely that such an extreme pleasure wizard exists, although it’s quite possible for there to be some variation among people considered “normal.” The psychological unity of humankind provides some evidence against extreme pleasure wizards’ existence, although it’s far from conclusive.
Preference utilitarianism makes things more complicated since someone may be inefficient at producing happiness, but efficient at satisfying some of their other preferences. However, since being alive and healthy are nearly universal preferences, I think that it’s still accurate to call someone with severe illness a reverse pleasure wizard, even if you value preference satisfaction rather than happiness.
Even if you’re right and Carter was mistaken in stating that pleasure wizards don’t exist, that doesn’t alter his main point, which is that equality is valuable for its own sake, and therefore it is immoral to give all resources to pleasure wizards, no matter how efficient they are.
I thought you didn’t need common priors, and I was wrong. Editing. (I might have had in mind Hanson’s result that if you agree on the method to generate priors, then that’s enough.)
What? That’s… not AAT at all.
What possible justification could he have for this? “No one is better at happiness than others, but some people are worse at happiness” is obviously impossible, and if the claim is that there’s a plateau of “normal” people who are all roughly equivalent at converting resources into happiness and then people who are worse than that plateau, that sounds more like wishful thinking than a justified empirical claim.
They really don’t look that similar to me; they’re looking at very different problems and have very different approaches.
The basic problem is that utilitarianism simply doesn’t work.
Carter takes the common critique of total utilitarianism and the common critique of average utilitarianism, and says “well, both critiques go away if we try to maximize a combination of total and average.” But those are just the common critiques, not the most potent ones. The basic problem with utilitarianism is that utility is difficult to measure and impossible to compare- and so both total and average utilitarianism are not things that can actually be calculated.
Eliezer is trying to tackle the problem of what utilities actually cash out as, so that you can build a machine that can perform preference calculations and not get them horribly wrong. Will Alice be happier with an unlimited supply of cookies, or if she has to strive for them? The options satisfy different desires in different amounts, and so fun theory and complexity of value deal with the tradeoffs between different desires. If you just built a machine that knew about our desire to feel happy and didn’t know about our desire to impact the real world, you would get a population of wireheads- something that many of us think would be a bad outcome now, but cannot justify that judgment in terms of average or total ‘happiness.’
Do you really mean this, as opposed to “our desire to impact the real world”?
I’ve edited it to the version you said, as it’s cleaner for this discussion that way. In general I think I would separate the desire to impact and the desire for the map to match the territory.
(nods) That’s fair.
Utilitarianism, defined as an ethical theory that only has values involving happiness and suffering, doesn’t work. But values dealing with happiness and suffering are a subset of our values, which also include freedom, challenge, impact on the real world, friendship, novelty, and so on. Carter’s point can be extended fairly easily to these: just as we don’t only value total happiness but also average happiness, we don’t only value happiness but also all those other things. Carter was aiming to solve a problem with utilitarianism, but he managed to find a solution that extends to consequentialism in general.
Are those values an ideal consequentialist should have? Consider “I want to help people” and “I want people to be helped.”
I don’t think there’s any one “ideal consequentialist”. Some values may be more important to me than others, and I may want to self-modify to care less about some of those things in order to maximize the others, but no terminal values are themselves better or worse. My utility function is what it is.
The issue is that “freedom” and “challenge” aren’t really outcome preferences so much as they are action preferences or decision theory preferences. The consequentialist doesn’t see a difference between me proposing that we trade two of my apples for three of your lemons and a dictator ordering that we trade two of my apples for three of your lemons- the outcome is how many of which fruit each of us ends up with, and if the dictator is better at negotiating and knowing our preferences than we are, the consequentialist suggests that we use the dictator and get over our preference for freedom (which was useful when dictators were bad but isn’t useful when dictators are good).
You can smuggle other moral systems into consequentialism- by, say, including features of the decision tree as part of the outcome- but it’s far cleaner to just discard consequentialism.
I have different subjective experiences when I am making my own decisions and when I am doing something I was ordered to do, even if it’s the same decision and action both times. This suggests “my having freedom” is a real quality of a state of the world, and that therefore I can have consequentialist preferences about its presence vs. its absence. Anything that can distinguish two states of the world is a valid thing consequentialists can have values about.
In the trolley problem, an agent will have different subjective experiences in the case where they do nothing and in the case where they murder someone. Most consequentialist prescriptions count such preferences as insignificant in light of the other outcomes.
I do think that most consequentialists go further, claiming that only the final world state should matter and not how you got there, but I agree with you that consequentialist tools are powerful enough to adopt systems that are typically seen as competing- like deontology- and the reverse is true as well. Because there’s that flexibility in tools, I find conversations are easier if one adopts strict system definitions. If someone uses expected utility theory to pick an action, but their utility is based on the rules they followed in choosing actions, I don’t see the value in calling that consequentialism.
I think your quandary can be resolved by dividing your example into more than one consequence.
Example 1 has the consequences:
Dictator tells you what to do.
You end up with +3 lemons and −2 apples.
Example 2 has the consequences:
You think hard and make a decision.
You end up with +3 lemons and −2 apples.
I’m making up numbers here, but imagine you assign +10 utility to the consequence “end up with +3 lemons and −2 apples,” +1 utility to the consequence “think hard and make a decision.” and −3 to the consequence “dictator tells me what to do.” Then in Example 1 the two consequences have a combined utility of 7, whereas in Example 2 they have a combined utility of 11.
I think one reason that subjective experiences don’t matter in the trolly problem is that the stakes are so high. In the trolley problem your desire not to be involved in someone’s death is nothing compared to the desire of six people to not die. If the stakes were much lower, however, your subjective experiences might matter.
For instance, imagine a toned down trolley problem where if you do nothing Alice and Bob will get papercuts on their thumbs, and if you pull a switch Clyde will get a papercut on his thumb. In that case the stakes are low enough that the unpleasant feeling you get from pulling the switch and injuring someone might merit some consideration.
This is actually similar to how the preference for freedom is treated in real life. When the stakes are low freedom is respected more often, even if it sometimes leads to some bad consequences, but when they are higher (during war, viral epidemics, etc) freedom is restricted because the stakes are much higher. (Of course, it goes without saying that in real life treating freedom like this tends to encourage corruption)
My understanding was that if two equally rational people have the same information they will draw the same conclusions. So if someone draws the same conclusion as you, that’s evidence in your favor, but only very mild evidence since you don’t know for sure if they’re as rational as you and if they had the exact same info as you. I do think that might be a weak opening and am thinking of changing it.
It’s perfectly true. People with severe medical problems need huge amount of resources merely to satisfy their preferences of “not being dead” and “not being sick.” A normal person require far less resources to satisfy that preference. So people with severe medical problems are “reverse utility monsters.” This is true regardless of whether you are a preference or happiness utilitarian, since obviously you need to be alive to be happy at all and healthy to reach your full happiness potential (FYI I’m a preference utilitarian).
Normal_Anomaly made most of the points I was going to, so I won’t elaborate on this.
AAT says that if two people (who may have observed different evidence) have mutual knowledge that they are both perfect epistemic rationalists honestly reporting their posterior probabilities, then they cannot remain in an equalibrium where they both are aware of the other’s posterior probabilities but disagree on their posterior probabilities.
They need to have the same priors too, right?
They need to have the same priors? Wouldn’t that make AAT trivial and vacuous?
I thought the requirement was that priors just weren’t pathologically tuned.
I’m pretty sure they do need to have the same priors.
My intuition is that AAT is basically saying that the perfect epistemic rationalists involved can essentially transfer all of the evidence that they have to the other, so that each one effectively has the same evidence and so should have the same posteriors...except that they’ll still have different posteriors unless they began with the same priors.
If they found that they had different priors, I think that they could just communicate the evidence which led them to form those priors from previous priors and so forth, but I think that if they trace their priors as far back as possible and find that they have different ones, AAT doesn’t work.
I’m not actually super-familiar with it, so update accordingly if I seem to have said something dumb.
Nope, I was wrong. It is the case that agents require equal priors for ATT to hold. AAT is like proving that mixing the same two colors of paint will always result in the same shade or that two equal numbers multiplied by another number will still be equal.
What a worthless theorem!
I guess when I read that AAT required “common priors” I assumed Aumann must mean known priors or knowledge of each others’ priors, since equal priors would constitute both 1) an asinine assumption and, 2) a result not worth reporting. Hanson’s assumption that humans should have a shared prior by virtue of being evolved together is interesting, but more creative than informative.
Good thing I don’t rely on ATT for anything. It’s obvious that disagreeing with most people is rational so updating on people’s posteriors without evidence would be pretty unreasonable. I’m not surprised that ATT would turn out to be so meaningless.
Yes, and even have mutual knowledge that they have the same priors. Which I was thinking, but apparently failed to actually type.
AAT is very specific. Independent invention is evidence for the attractiveness of an idea (and thus, one hopes, its truthfulness) but it’s unrelated to AAT.
Sure. But you’re telling me that two healthy individuals are equally efficient at converting resources into happiness? What evidence is there that the Brahmin is not ten times as capable of happiness as the Untouchable?
I have edited my post to remove the reference to AAT per your and JGWeissman’s advice.
In Carter’s paper he was discussing a hypothetical pleasure wizard so good at converting resources into happiness that it was better to give it all the resources in the world and let everyone else just have enough for a life barely worth living. It seems unlikely that such an extreme pleasure wizard exists, although it’s quite possible for there to be some variation among people considered “normal.” The psychological unity of humankind provides some evidence against extreme pleasure wizards’ existence, although it’s far from conclusive.
Preference utilitarianism makes things more complicated since someone may be inefficient at producing happiness, but efficient at satisfying some of their other preferences. However, since being alive and healthy are nearly universal preferences, I think that it’s still accurate to call someone with severe illness a reverse pleasure wizard, even if you value preference satisfaction rather than happiness.
Even if you’re right and Carter was mistaken in stating that pleasure wizards don’t exist, that doesn’t alter his main point, which is that equality is valuable for its own sake, and therefore it is immoral to give all resources to pleasure wizards, no matter how efficient they are.
You need to have common knowledge of each other’s estimates, common knowledge of each other’s rationality, and common priors.
I thought you didn’t need common priors, and I was wrong. Editing. (I might have had in mind Hanson’s result that if you agree on the method to generate priors, then that’s enough.)