I’m convinced by mathematical arguments that utility should be additive. If the value of N things in the real world is not N times the value of 1 thing, then I handle that in how I assign utility to world states.
I don’t disagree. My choice of slogan wording—“utility is not additive”—doesn’t capture what I mean. I meant only to deny that the value of something happening N times is (N x U) where U is the value of it happening once.
what would you say to a utilitarian who say: “Evolution (memetic evolution, that is) has instilled in me the idea of that I should linearly value the welfare of others regardless of kinship, regardless of what instincts I got from my genes.”
There are two separate issues here. I assume that by “linearly” you are referring to the subject that started this conversation: my claim that utilities “are not additive”, an idea also expressed as “diminishing returns”, or diminishing marginal utility of additional people. I probably would not dispute the memetic evolution claim if it focused on “linearity”.
The second issue is a kind of universality—all people valued equally regardless of kinship or close connectedness in a network of reciprocity. I would probably express skepticism at this claim. I would probe the claim to determine whether the selection operates at the level of the meme, the individual, or the society. And then I would ask how that meme contributes to its own propagation at that level.
By “orthodox position” are you referring to TDT-related ideas?
Mostly, I am referring to views expressed by EY in the sequences and frequently echoed by LW regulars in comments. Some of those ideas were apparently repeated in the TDT writeup (though I may be wrong about that—the write-up was pretty incoherent.)
… the mistake began as soon as we started calling it a “blue-minimizing robot”.
Agreed. But what kind of mistake was that?
Is “This robot is a blue-minimizer” a false statement? I think not. I would classify it as more like the unfortunate selection of the wrong Kuhnian paradigm for explaining the robot’s behavior. A pragmatic mistake. A mistake which does not bode well for discovering the truth, but not a mistake which involves starting from objectively false beliefs.
I think that we should follow Jaynes and insist upon ‘probability’ as the name of the subjective entity. But so-called objective probability should be called ‘propensity’. Frequency is the term for describing actual data. Propensity is objectively expected frequency. Probability is subjectively expected frequency. That is the way I would vote.
As I understand it, EY’s commitment to MWI is a bit more principled than a choice between soccer teams. MWI is the only interpretation that makes sense given Eliezer’s prior metaphysical commitments. Yes rational people can choose a different interpretation of QM, but they probably need to make other metaphysical choices to match in order to maintain consistency.
Does your utility function treat “a life saved by Perplexed” differently from just “a life”?
I’m torn between responding with “Good question!” versus “What difference does it make?”. Since I can’t decide, I’ll make both responses.
Good question! You are correct in surmising that the root justification for much of the value that I attach to other lives is essentially instrumental (via channels of reciprocity). But not all of the justification. Evolution has instilled in me the instinct of valuing the welfare (fitness) of kin at a significant fraction of the value of my own personal welfare. And then there are cases where kinship and reciprocity become connected in serial chains. So the answer is that I discount based on ‘remoteness’ where remoteness is a distance metric reflecting both genetic and social-interactive inverse connectedness.
What difference does it make? This is my utility function we are talking about, and it is only operational in deciding my own actions. So, even if my utility function attached huge value to lives saved by other people, it is not clear how this would change my behavior. The question seems to be whether people ought to have multiple utility functions—one for directing their own rational choices; the others for some other purpose.
I am currently reading Binmore’s two-volume opus Game Theory and the Social Contract. I strongly recommend it to everyone here who is interested in decision theory and ethics. Although Binmore doesn’t put it in these terms, his system does involve two different sets of values, which are used in two different ways. One is the set of values used in the Game of Life—a set of values which may be as egoistic as the agent wishes (or as altruistic). However, although the agent is conceptually free in the Game of Life, as a practical matter, he is coerced by everyone else to adhere to a Social Contract. Due to this coercion, he mostly behaves morally.
But how does the Social Contract arise? In Binmore’s normative fiction, it arises by negotiated consensus of all agents. The negotiation takes place in a Rawlsian Original Position under a Veil of Ignorance. Since the agent-while-negotiating has different self-knowledge than does the agent-while-living, he manifests different values in the two situations—particularly with regard to utilities which accrue indexically. So, according to Binmore, even an agent who is inherently egoistic in the Game of Life will be egalitarian in the Game of Morals where the Social Contract is negotiated. Different values for a different purpose.
That is the concise summary of the ethical system that Binmore is constructing in the two volumes. But he does a marvelously thorough job of ground-clearing—addressing mistakes made by Kant, Rawls, Nozick, Parfit, and others regarding the Prisoner’s Dilemma, Newcomb’s ‘paradox’, whether it is rational to vote (probably wasted), etc. And in the course of doing so, he pretty thoroughly demolishes what I understand to be the orthodox position on these topics here at Less Wrong.
Really, really recommended.
Correct. In fact, I probably confused things here by using the word “discount” for what I am suggesting here. Let me try to summarize the situation with regard to “discounting”.
Time discounting means counting distant future utility as less important than near future utility. EY, in the cited posting, argues against time discounting. (I disagree with EY, for what it is worth.)
“Space discounting” is a locally well-understood idea that utility accruing to people distant from the focal agent is less important than utility accruing to the focal agent’s friends, family, and neighbors. EY presumably disapproves of space discounting. (My position is a bit complicated. Distance in space is not the relevant parameter, but I do approve of discounting using a similar ‘remoteness’ parameter.)
The kind of ‘discounting’ of large utilities that I recommended in the great-grandparent probably shouldn’t be called ‘discounting’. I would sloganize it as “utilities are not additive.” The parent used the phrase ‘diminishing returns’. That is not right either, though it is probably better than ‘discounting’. Another phrase that approximates what I was suggesting is ‘bounded utility’. (I’m pretty sure I disagree with EY on this one too.)
The fact that I disagree with EY on discounting says absolutely nothing about whether I agree with EY on AI risk, reductionism, exercise, and who writes the best SciFi. That shouldn’t need to be said, but sometimes it seems to be necessary in your (XiXiDu’s) case.
What benelliot said.
Sheesh! Please don’t assume that everyone who disagrees with one point you made is doing so because he disagrees with the whole thrust of your thinking.
Baez: … you shouldn’t always maximize expected utility if you only live once.
BenElliot: [Baez is wrong] Expected utilities do not work like that.
XiXiDu: If a mathematician like John Baez can be that wrong …
A mathematician like Baez can indeed be that wrong, when he discusses technical topics that he is insufficiently familiar with. I’m sure Baez is quite capable of understanding the standard position of economists on this topic (the position echoed by BenElliot). But, as it apparently turns out, Baez has not yet done so. No big deal. Treat Baez as an authority on mathematical physics, category theory, and perhaps saving the environment. He is not necessarily an authority on the foundations of microeconomics.
People shouldn’t neglect small probabilities. The math works for small probabilities.
People should discount large utilities. Utilities are not additive. Nothing in economic theory suggests that they are additive.
It is well understood that the utility of two million dollars is not necessarily twice the utility of one million dollars. Yet it is taken here as axiomatic that the utility of two saved lives is twice the utility of one saved life. Two people tortured is taken to be twice the disutility of one person tortured. Why? As far as I can tell, the only answer given here is that our moral intuitions (against simple additivity) are wrong and will be corrected by sufficient reflection.
That is my take on the issue.
You are correct that my comments are missing the mark. Still, there is a sense in which the kinds of non-determinism represented by Born probabilities present problems for Si. I agree that Si definitely does not pretend to generate its predictions based on observation of the whole universe. And it does not pretend to predict everything about the universe. But it does seem to pretend that it is doing something better than making predictions that apply to only one of many randomly selected “worlds”.
Can anyone else—Cousin_it perhaps—explain why deterministic evolution of the wave function seems to be insufficient to place Si on solid ground?
Hmmm. Would you be happier if I changed my last line to read ”… we need to discard the whole Si concept as inappropriate to our imperfectly-observed universe.”?
My implicit point was this: Nesov2006 probably did not realize that Nesov2006 was a fool and Nesov2008 probably did not judge himself to be a crackpot. Therefore, a naive extrapolation (“obvious prediction”) suggests that Nesov2011 has some cognitive flaws which he doesn’t yet recognize; he will recognize this, though, in a few years.
JoshuaZ, as I understood him, was suggesting that one improves ones tolerance by enlarging ones prior for the hypothesis that one is currently flawed oneself. He suggests the time-machine thought experiment as a way of doing so.
You, as I understood you, claimed that JoshuaZ’s self-hack doesn’t work on you. I’m still puzzled as to why not.
Write suggestion-driven fiction.
What is “suggestion-driven fiction”? Googling was unhelpful.
What it sounds like is fiction in which the author has no particular story in mind as (s)he begins the narration, but rather the author generates plot and characters in response to reader suggestions as each chapter is published.
If that is the kind of thing you are talking about, it sounds very intriguing. But I wonder how a beginner captures enough initial readers to generate the suggestions. Reciprocity? If someone wants to organize a circle of three or four novice writers producing serialized fiction on their blogs and providing suggestions to each other, I would like to join the group.
What should I do?
Step 1: Stop being frustrated with them for not knowing/understanding. Instead, direct your frustration at yourself for not successfully communicating.
Step 2: Come to know that the reason for your failure to communicate is not a lack of mastery over your own arguments. It is a lack of understanding of their arguments. Cultivate the skill of listening. Ask which school of martial arts presents the best metaphor for your current disputation habits. Which school best matches the kind of disputation you would like to achieve?
Step 3: In the course of learning to listen, you may also learn other things. The people you are talking to are probably not idiots. Sometimes they will be right and you will be wrong. Notice those occasions. Examine, analyze, and cherish those occasions. Come to see them as winning. After all, you came out of that dispute having gained something useful (knowledge). Your teacher, on the other hand, gained nothing but ephemeral status.
I would easily bite the bullet and say that Nesov2008 was a crackpot and Nesov2006 was a shallow naive fool.
Ah. But would you make the obvious predictions about the opinion Nesov2013 and Nesov2015 will have regarding Nesov2011?
I disagree. Alicorn’s version is more mathematically meaningful, to my mind, than WeiDai’s. But to return to the original problem:
A. Two-boxing yields more money than would be yielded by counterfactually one-boxing.
B. Taboo “counterfactually”. …
“Recommends” is math?
I’m confused. Assuming that I “believe in” the validity of what I have been told of quantum mechanics, I fully expect that a million quantum coin tosses will generate an incompressible string. Are you suggesting that I cannot simultaneously believe in the validity of QM and also believe in the efficacy of Solomonoff induction—when applied to data which is “best explained” as causally random?
Off the top of my head, I am inclined to agree with this suggestion, which in turn suggests that Si is flawed. We need a variant of Si which allows Douglas_Knight’s simple fair coins, without thereby offering a simple explanation of everything. Or, we need to discard the whole Si concept as inappropriate in our non-deterministic universe.
The randomness of a source of information is not an empirical fact which we can discover and test—rather, it is an assumption that we impose upon our model of the data. It is a null hypothesis for which we cannot find Bayesian evidence—we can at best fail to reject it. (I hope the Popper-clippers don’t hear me say that!). Maybe what our revised Si should be looking for is the simplest explanation for data D[0] thru D[n], which explanation is not refuted by data D[n+1] thru D[n+k].
ETA: Whoops. That suggestion doesn’t work. The simplest such explanation will always be that everything is random.
Your just-so story is more complicated than you seem to think. It involves an equilibrium of at least two memes: an evangelical utilitarianism which damages the host but propagates the meme, plus a cryptic egoism which presumably benefits the host but can’t successfully propagate (it repeatedly arises by spontaneous generation, presumably).
I could critique your story on grounds of plausibility (which strategy do crypto-egoists suggest to their own children?) but instead I will ask why someone infected by the evangelical utilitarianism meme would argue as you suggested in the great-grandparent:
Isn’t it more likely that someone realizing that they have been subverted by a selfish meme would be trying to self-modify?