You know, I’ve felt that examining the dust speck vs torture dilemma or stuff like that, finding a way to derive an intuitively false conclusion from intuitively true premises, and thereby concluding that the conclusion must be true after all (rather than there’s some kind of flaw in the proof you can’t see yet) is analogous to seeing a proof that 0 equals 1 or that a hamburger is better than eternal happiness or that no feather is dark, not seeing the mistake in the proof straight away, and thereby concluding that the conclusion must be true. Does anyone else feel the same?
But it’s not like continuing to endorse my intuitions in the absence of any justification for them, on the assumption that all arguments that run counter to my intuitions, however solid they may seem, must be wrong because my intuitions say so, is noticeably more admirable.
When my intuitions point in one direction and my reason points in another, my preference is to endorse neither direction until I’ve thought through the problem more carefully. What I find often happens is that on careful thought, my whole understanding of the problem tends to alter, after which I may end up rejecting both of those directions.
Well, what you should do, is to recognize that such arguments themselves are built entirely out of intuitions, and their validity rest on conjunction of a significant number of often unstated intuitive assumptions. One should not fall for cargo cult imitation of logic.
There’s no fundamental reason why value should be linear in number of dust specks; it’s nothing but an assumption which may be your personal intuition, but it is still intuition that lacks any justification what so ever, and in so much as it is an uncommon intuition, it even lacks the “if it was wrong it would be debunked” sort of justification. There’s always the Dunning Kruger effect. People least capable of moral (or any) reasoning should be expected to think themselves most capable.
There’s no fundamental reason why value should be linear in number of dust specks
Yeah, that has always been my main problem with that scenario.
There are different ways to sum multiple sources of something. Consider linear vs paralel electrical circuits; the total output depends greatly on how you count the individual voltage sources (or resistors or whatever).
When it comes to suffering, well suffering only exists in consciousness, and each point of consciousness—each mind involved—experiences their own dust speck individually. There is no conscious mind in that scenario who is directly experiencing the totality of the dust specks and suffers accordingly. It is in no way obvious to me that the “right” way to consider the totality of that suffering is to just add it up. Perhaps it is. But unless I missed something, no one arguing for torture so far has actually shown it (as opposed to just assuming it).
Suppose we make this about (what starts as) a single person. Suppose that you, yourself, are going to be copied into all that humongous number of copies. And you are given a choice: before that happens, you will be tortured for 50 years. Or you will be unconscious for 50 years, but after copying each of your copies will get a dust speck in the eye. Either way you get copied, that’s not part of the choice. After that, whatever your choice, you will be able to continue with your lives.
EDIT: come to think of it, assume the copying template is taken from you before the 50 years start, so we don’t have to consider memories and lasting psychological effects of torture. My answer remains the same, even if in future I won’t remember the torture, I don’t want to go through it.
As far as I know, TvDS doesn’t assume that value is linear in dust specks. As you say, there are different ways to sum multiple sources of something. In particular, there are many ways to sum the experiences of multiple individuals.
For example, the whole problem evaporates if I decide that people’s suffering only matters to the extent that I personally know those people. In fact, much less ridiculous problems also evaporate… e.g., in that case I also prefer that thousands of people suffer so that I and my friends can live lives of ease, as long as the suffering hordes are sufficiently far away.
It is not obvious to me that I prefer that second way of thinking, though.
e.g., in that case I also prefer that thousands of people suffer so that I and my friends can live lives of ease, as long as the suffering hordes are sufficiently far away.
It is arguable (in terms of revealed preferences) that first-worlders typically do prefer that. This requires a slightly non-normative meaning of “prefer”, but a very useful one.
Oh, absolutely. I chose the example with that in mind.
I merely assert that “but that leads to thousands of people suffering!” is not a ridiculous moral problem for people (like me) who reveal such preferences to consider, and it’s not obvious that a model that causes the problem to evaporate is one that I endorse.
Well, it sure uses linear intuition. 3^^^3 is bigger than number of distinct states, its far past point where you are only increasing exactly-duplicated dust speck experience, so you could reasonably expect it to flat out.
One can go perverse and proclaims that one treats duplicates the same, but then if there’s a button which you press to replace everyone’s mind with mind of happiest person, you should press it.
I think the stupidity of utilitarianism is the belief that the morality is about the state, rather than about dynamic process and state transition. Simulation of pinprick slowed down 1000000 times is not ultra long torture. The ‘murder’ is a form of irreversible state transition. The morality as it exist is about state transitions not about states.
It isn’t clear to me what the phrase “exactly-duplicated” is doing there. Is there a reason to believe that each individual dust-speck-in-eye event is exactly like every other? And if so, what difference does that make? (Relatedly, is there a reason to believe that each individual moment of torture is different from all the others? If it turns out that it’s not, does that imply something relevant?)
In any case, I certainly agree that one could reasonably expect the negvalue of suffering to flatten out no matter how much of it there is. It seems unlikely to me that fifty years of torture is anywhere near the asymptote of that curve, though… for example, I would rather be tortured for fifty years than be tortured for seventy years.
But even if it somehow is at the asymptotic limit, we could recast the problem with ten years of torture instead, or five years, or five months, or some other value that is no longer at that limit, and the same questions would arise.
So, no, I don’t think the TvDS problem depends on intuitions about the linear-additive nature of suffering. (Indeed, the more i think about it the less convinced i am that I have such intuitions, as opposed to approaches-a-limit intuitions. This is perhaps because thinking about it has changed my intuitions.)
I was referring to linear-additive nature of dust speck so called suffering, in the number of people with dust specks.
3^^^3 is far far larger than number of distinct mind states of anything human-like. You can only be dust-speck-ing something like 10^(10^20) distinct human-like entities maximum. I recall i posted about that a while back. You shouldn’t be multiplying anything with 3^^^3 .
TBH, my ‘common sense’ explanation as of why EY chooses to adopt torture > dust specks stance (i say chooses because it is entirely up to grabs here plus his position is fairly incoherent), is because he seriously believes that his work has non negligible chance of influencing lives of an enormous number of people, and subsequently if he can internalize the torture>dust specks, he is free to rationalize any sort of thing he can plausibly do, even if AI extinction risk does not exist.
[edit: this response was to an earlier version of the above comment, before it was edited. Some of it is no longer especially apposite to the comment as it exists now.]
I was referring to linear-additive nature of dust specks.
Well, I agree that 3^^^3 dust specks don’t quite add linearly… long before you reach that ridiculous mass, I expect you get all manner of weird effects that I’m not physicist enough to predict. And I also agree that our intuitions are that dust specks add linearly.
But surely it’s not the dust-specks that we care about here, but the suffering? That is, it seems clear to me that if we eliminated all the dust specks from the scenario and replaced them with something that caused an equally negligible amount of suffering, we would not be changing anything that mattered about the scenario.
And, as I said, it’s not at all clear to me that I intuit linear addition of suffering (whether it’s caused by dust-specks, torture, or something else), or that the scenario depends on assuming linear addition of suffering. It merely depends on assuming that addition of multiple negligible amounts of suffering can lead to an aggregate-suffering result that is commensurable with, and greater than, a single non-negligible amount of suffering.
It’s not clear to me that this assumption holds, but the linear-addition objection seems like a red herring to me.
You can only be dust-speck-ing something like 10^(10^20) distinct human-like entities maximum.
Ah, I see.
Yeah, sure, there’s only X possible ways for a human to be (whether 10^(10^20) or some other vast number doesn’t really matter), and there’s only Y possible ways for a dust speck to be, and there’s only Z possible ways for a given human to experience a given dust speck in their eye. So, sure, we only have (XYZ) distinct dust-speck-in-eye events, and if (XYZ) << 3^^^3 then there’s some duplication. Indeed, there’s vast amounts of duplication, given that (3^^^3/(XYZ)) is still a staggeringly huge number.
Agreed.
I’m still curious about what difference that makes.
Lead to severe discounting of the ‘reasoning method’ that arrived at 3^^^3 dust-specks>torture conclusion without ever coming across the exhaustion of states issue. In all fields where it was employed. And to severely discount anything that came from that process previously. If it failed even though it gone against intuition, it’s even more worthless when it goes along with intuition.
I get the feeling that attempts to ‘logically’ deliberate on morality from some simple principles like “utility” are similar to trying to recognize cats in pictures by reading R,G,B number value array and doing some arithmetic. If someone haven’t got visual cortex they can’t see, even if they do insane amount of reasoning deliberately.
similar to trying to recognize cats in pictures by reading R,G,B number value array and doing some arithmetic
But a computer can recognize cats by reading pixel values in pictures? Maybe not as efficiently and accurately as people, but that’s because brains have a more efficient architecture/algorithms than today’s generic computers.
Yes, it is of course possible in principle (in fact I am using cats as example because Google just did that). The point is that a person can’t do anything equivalent to what human visual cortex does in a fraction of a second by using paper and pencil for multiple lifetimes. The morality and the immorality, just like cat recognition, rely on some innate human ability of connecting symbols with reality.
edit: To clarify. To tell which images are cats and which are dogs, you employ some method that is hopelessly impossible for you to write down. To tell what actions are moral or not, humans employ some method that is likewise hopelessly impossible for them to write down. All you can do is write down guidelines, and add some picture examples of cats and dogs. Various rules like utilitarianism are along the lines of “if the eyes have vertical slits, it’s a cat” which mis-recognize a lizard as a cat but do not recognize the cat that closed the eyes. (There is also the practical matter of law making, where you want to restrict the diversity of moral judgment to something sane, and thus you use principles like ‘if it doesn’t harm anyone else it’s okay’)
To tell which images are cats and which are dogs, you employ some method that is hopelessly impossible for you to write down.
Right, but if/when we get to (partial) brain emulations (in large quantities) we might be able to do the same thing for ‘morality’ that we do today to recognize cats using a computer.
Agreed. We may even see how it is that certain algorithms (very broadly speaking) can feel pain etc, and actually start defining something agreeable from first principles. Meanwhile, all that 3^^^3 people with dustspeck worse than 1 person tortured stuff is to morality as scholasticism is to science. The only value it may have is in highlighting the problem with approximations, and with handwavy reasoning (nobody said that the number of possible people is >3^^^3 (which is false) , even though such statement was a part of reasoning and should have been stated and then rejected invalidating everything that followed. Or a statement that identical instances matter should have been made, which in itself leads to multitude of really dumb decisions whereby the life of a conscious robot that has thicker wires in its computer (or uses otherwise redundant hardware) is worth more)
Or a statement that identical instances matter should have been made
Not many people hold the view that if eternal inflation is true then there is nothing wrong with hitting people with hot pokers, since the relevant brain states exist elsewhere anyway. In Bostrom’s paper he could only find a single backer of the view. In talking to many people, I have seen it expressed more than once, but still only a very small minority of cases. Perhaps not including it in that post looms large for you because you have a strong intuition that it would be OK to torture and kill if the universe were very large, or think it very unlikely that the universe is large, but it’s a niche objection to address.
After all, one could include such a discussion as a rider in every post talking about trying to achieve anything for oneself or others: “well, reading this calculus textbook seems like it could teach you interesting math, but physicists say we might be living in a big universe, in which case there’s no point since brains in all states already exist, if you don’t care about identical copies.”
If there is any nonzero probability that universe is NOT very large (or the copy counting is a bit subtle about the copies which are effectively encoding state onto coordinates), all you did is scaled all the utilities down, which does not affect any decision.
That’s incredibly terrible thing to do for our friends the people who believe themselves to be utilitarian, as those people are going to selectively scale down just some of the utilities and then act, in self interest or otherwise, out of resulting big differences, doing something stupid.
edit: also, the issue with multiple-counting redundant hardware and the thick-wired utility monsters in the utilitarianism that does count extra copies doesn’t go away if the world is big. If you have a solid argument that utilitarianism without counting the extra copies the same does not work, that means utilitarianism does not work. Which I believe is the case. The morals are an engineered / naturally selected solution to problem of peer to peer intellectual and other cooperation, which requires nodes not to model each other in undue detail, which rules out direct straightforward utilitarianism. The utilitarianism is irreparably broken. It’s fake-reductionism where you substitute one irreducible concept for another.
(or the copy counting is a bit subtle about the copies which are effectively encoding state onto coordinates)
That’s an interesting idea, thanks. Maybe caring about anthropic probabilities or measures of conscious experiences directly would make more sense than caring about the number of copies as a proxy.
If you take that idea seriously and assume that all anthropic probabilities of conscious experiences must sum to 1, then torture vs dustspecks seems to lose some of its sting, because the total disutility of dustspecking remains bounded and not very high, no matter how many people you dustspeck. (That’s a little similar to the “proximity argument”, which says faraway people matter less.) And being able to point out the specific person to be tortured means that person doesn’t have too low weight, so torturing that single person would be worse than dustspecking literally everyone else in the multiverse. I don’t remember if anyone made this argument before… Of course there could be any number of holes in it.
Also note that the thicker wires argument is not obviously wrong, because for all we know, thicker wires could affect subjective probabilities. It sounds absurd, sure, but so does the fact that lightspeed is independent of observer speed.
ETA: the first version of this comment mixed up Pascal’s mugging and torture vs dustspecks. Sorry. Though maybe a similar argument could be made for Pascal’s mugging as well.
Thinking about it some more: maybe the key is that it is not enough for something to exist somewhere, just as it is not enough for output tape in Solomonoff induction to contain the desired output string somewhere within it, it should begin with it. (Note that it is a critically important requirement). If you are using Solomonoff induction (suppose you got oracle and suppose universe is computable and so on), then your model contains not only laws of universe but also locator, and my intuition is that one model that has simplest locator is some very huge length shorter than the next simplest model, so all the other models except the one with simplest locator, have to be ignored entirely.
If we require that the locator is present somehow in the whole then the ultra-distant copies are very different while the nearby copies are virtually the same, and Kolmogorov complexity of concatenated strings can be used for count, not counting twice nearby copies (the thick wired monster only weights a teeny tiny bit more).
TBH i feel tho that utilitarianism goes in the wrong direction entirely. Morals can be seen as evolved / engineered solution to peer to peer intellectual and other cooperation, essentially. It relies on trust, not on mutual detailed modeling (which wastes computing power), and the actions are not quite determined by the expected state (which you can’t model), even though it is engineered with some state in mind.
edit: also I think the what ever stuff raises the problem with distant copies or MWI is subjectively disproved by this not saving you from brain damage of any kind (you can get drunk, pass out, wake up with a little bit fewer neurons). So we basically know something’s screwed up with naive counting for probabilities, or the world is small.
Lead to severe discounting of the ‘reasoning method’ that arrived at 3^^^3 dust-specks>torture conclusion without ever coming across the exhaustion of states issue.
This is mistaken. E.g. see this post which discusses living in a Big World, as in eternal inflation theories where the universe extends infinitely and has random variation so that somewhere in the universe every possible galaxy or supercluster will be realized, and all the human brain states will be explored.
Or see Bostrom’s paper on this issue, which is very widely read around here. Many people think that our actions can still matter in such a world, e.g. that it’s better to try to give people chocolate than to torture them here on Earth, even if in ludicrously distant region there are brains that have experienced all the variations of chocolate and torture.
Lead to severe discounting of the ‘reasoning method’ that arrived at 3^^^3 dust-specks>torture conclusion without ever coming across the exhaustion of states issue.
Even better, to my mind, is to think about the scenario from the ground up and form my own conclusions, rather than start with some intuitive judgment about someone else’s writeup about it and then update that judgment based on things they didn’t mention in that writeup.
If someone haven’t got visual cortex they can’t see, even if they do insane amount of reasoning deliberately
It’s not clear to me that I correctly understand what you mean here, but given my current understanding, I disagree. All my visual cortex is doing is performing computations on the output of my eyes; if that’s seeing, then anything else that performs the same computations can see just as well.
Even better, to my mind, is to think about the scenario from the ground up and form my own conclusions, rather than start with some intuitive judgment about someone else’s writeup about it and then update that judgment based on things they didn’t mention in that writeup.
The point is that the approach is flawed; one should always learn on mistakes. The issue here is in building an argument which is superficially logical—conforms to the structure of something a logical rational person might say—something you might have a logical character in a movie say—but is fundamentally a string of very shaky intuitions which are only correct if nothing outside the argument interferes, rather than solid steps.
It’s not clear to me that I correctly understand what you mean here, but given my current understanding, I disagree. All my visual cortex is doing is performing computations on the output of my eyes; if that’s seeing, then anything else that performs the same computations can see just as well.
In theory. In practice it takes ridiculous number of operations, and you can’t chinese-room vision without slow-down by factor of billions. Decades for single-cat recognition versus fraction of a second, and that’s if you got an algorithm for it.
I disagree with pretty much all of this, as well as with most of what seem to be the ideas underlying it, and don’t see any straightforward way to achieve convergence rather than infinitely ramified divergence, so I suppose it’s best for me to drop the thread here.
I think the stupidity of utilitarianism is the belief that the morality is about the state, rather than about dynamic process and state transition.
“State” doesn’t have to mean “frozen state” or something similar, it could mean “state of the world/universe”. E.g. “a state of the universe” in which many people are being tortured includes the torture process in it’s description. I think this is how it’s normally used.
Well, if you are to coherently take it that the transitions have value, rather than states, then you arrive at morality that regulates the transitions that the agent should try to make happen, ending up with morality that is more about means than about ends.
I think it’s simply that the pain feels like a state rather than dynamic process, and so utilitarianism treats it as state, while doing something feels like a dynamic process, so utilitarianism doesn’t treat it as state and is only concerned with difference in utilities.
Agreed that all of these sorts of arguments ultimately rest on different intuitions about morality, which sometimes conflict, or seem to conflict.
Agreed that value needn’t add linearly, and indeed my intuition is that it probably doesn’t.
It seems clear to me that if I negatively value something happening, I also negatively value it happening more more. That is, for any X I don’t want to have happen, it seems I would rather have X happen than have X happen twice. I can’t imagine an X where I don’t want X to happen and would prefer to have X happen twice than once. (Barring silly examples like “the power switch for the torture device gets flipped”.)
Can anyone explain what goes wrong if you say something like, “The marginal utility of my terminal values increases asymtotically, and u(Torture) approaches a much higher asymptote than u(Dust speck)” (or indeed whether it goes wrong at all)?
Thank you. I had expected the bottom to drop out of it somehow.
EDIT: Although come to think of it I’m not sure the objections presented in that paper are so deadly after all if you takes TDT-like considerations into account (i.e. there would not be a difference between “kill 1 person, prevent 1000 mutilations” + “kill 1 person, prevent 1000 mutilations” and “kill 2 people, prevent 2000 mutilations”.) Will have to think on it some more.
Can anyone explain what goes wrong if you say something like, “The marginal utility of my terminal values increases asymtotically, and u(Torture) approaches a much higher asymptote than u(Dust speck)” (or indeed whether it goes wrong at all)?
Nothing, iif that happens to be be what your actual preferences are. If your preferences did not happen to be as you describe but instead you are confused by an inconsistency in your intuitions then you will make incorrect decisions.
The challenge is not to construct a utility function such that you can justify it to others in the face of opposition. The challenge is to work out what your actual preferences are and implement them.
The challenge is to work out what your actual preferences are and implement them.
Ayup. Also, it may be worth saying explicitly that a lot of the difficulty comes in working out a model of my actual preferences that is internally consistent and can be extended to apply to novel situations. If I give up those constraints, it’s easier to come up with propositions that seem to model my preferences, because they approximate particular aspects of my preferences well enough that in certain situations I can’t tell the difference. And if I don’t ever try to make decisions outside of that narrow band of situations, that can be enough to satisfy me.
The challenge is to work out what your actual preferences are and implement them.
[Edited to separate from quote]
But doesn’t that beg the question? Don’t you have to ask a the meta question “what kinds of preferences are reasonable to have?” Why should we shape ethics the way evolution happened to set up our values? That’s why I favor hedonistic utiltiarianism that is about actual states of the world that can in itself be bad (--> suffering).
Note that markup requires a blank line between your quote and the rest of the topic.
It does beg a question: specifically, the question of whether I ought to implement my preferences (or some approximation of them) in the first place. If, for example, my preferences are instead irrelevant to what I ought to do, then time spent working out my preferences is time that could better have been spent doing something else.
All of that said, it sounds like you’re suggesting that suffering is somehow unrelated to the way evolution set up our values. If that is what you’re suggesting, then I’m completely at a loss to understand either your model of what suffering is, or how evolution works.
The fact that suffering feels awful is about the very thing, and nothing else. There’s no valuing required, no being ask itself “should I dislike this experience” when it is in suffering. It wouldn’t be suffering otherwise.
My position implies that in a world without suffering (or happiness, if I were not a negative utiltiarian), nothing would matter.
If I make that assumption, then it follows that given enough Torture to approach its limit, I choose any number of Dust Specks rather than that amount of Torture.
If my goal is to come up with an algorithm that leads to that choice, then I’ve succeeded.
(I think talking about Torture and Dust Specks as terminal values is silly, but it isn’t necessary for what I think you’re trying to get at.)
Yes. The known unreliability of my own thought processes tempers my confidence in any prima facie absurd conclusion I come to. All the more so when it’s a conclusion I didn’t come to, but merely followed along with someone else’s argument to.
I feel this way. The linear theories are usually nothing but first order approximations.
Also, the very idea of summing of individual agent utilities… that’s, frankly, nothing but pseudomathematics. Each agent’s utility function can be modified without changing agent’s behaviour in any way. The utility function is a phantom. It isn’t so defined that you could add two of them together. You can map same agent’s preferences (whenever they are well ordered) to infinite variety of real valued ‘utility functions’.
Yes. The trouble with “shut up and multiply” - beyond assuming that humans have a utility function at all—is assuming that utility works like conventional arithmetic and that you can in fact multiply.
There’s also measuring and shut-up-and-multiplying the wrong thing: e.g., seeing people willing to pay about the same in total to save 2000 birds or 20,000 birds and claiming this constitutes “scope insensitivity.” The error is assuming this means that people are scope-insensitive, rather than to realise that people aren’t buying saved birds at all, but are paying what they’re willing to pay for warm fuzzies in general—a constant amount.
The attraction of utilitarianism is that calculating actions would be so much simpler if utility functions existed and their output could be added with the same sort of rules as conventional arithmetic. This does not, however, constitute non-negligible evidence that any of the required assumptions hold.
This does not, however, constitute non-negligible evidence that any of the required assumptions hold.
It even tends to count against it, by the A+B rule. If items are selected by a high enough combined score on two criteria A and B, then among the selected items, there will tend to be a negative correlation between A and B.
There’s also measuring and shut-up-and-multiplying the wrong thing: e.g., seeing people willing to pay about the same in total to save 2000 birds or 20,000 birds and claiming this constitutes “scope insensitivity.” The error is assuming this means that people are scope-insensitive, rather than to realise that people aren’t buying saved birds at all, but are paying what they’re willing to pay for warm fuzzies in general—a constant amount.
I don’t know who’s making that error. Seems like scope insensitivity and purchasing of warm fuzzies are usually discussed together around here.
Anyway, if there’s an error here then it isn’t about utilitarianism vs something else, but about declared vs revealed preference. The people believe that they care about the birds. They don’t act as if they cared about the birds. For those who accept deliberative reasoning as an expression of human values it’s a failure of decision-making intuitions and it’s called scope insensitivity. For those who believe that true preference is revealed through behavior it’s a failure of reflection. None of those positions seems inconsistent with utilitarianism. In fact it might be easier to be a total utilitarian if you go all the way and conclude that humans really care only about power and sex. Just give everybody nymphomania and megalomania, prohibit birth control and watch that utility counter go. ;)
I don’t think it’s even linearly combinable. Suppose there were 4 copies of me total, pair doing some identical thing, other pair doing 2 different things. The second pair is worth more. When I see someone go linear on morals, that strikes me as evidence of poverty of moral value and/or poverty of mathematical language they have available.
Then the consequentialism. The consequences are hard to track—got to model the worlds resulting from uncertain initial state. Really really computationally expensive. Everything is going to use heuristics, even jupiter brains.
There’s also measuring and shut-up-and-multiplying the wrong thing: e.g., seeing people willing to pay about the same in total to save 2000 birds or 20,000 birds and claiming this constitutes “scope insensitivity.” The error is assuming this means that people are scope-insensitive, rather than to realise that people aren’t buying saved birds at all, but are paying what they’re willing to pay for warm fuzzies in general—a constant amount.
Well, “willing to pay for warm fuzzies” is a bad way to put it IMO. There’s limited amount of money available in the first place, if you care about birds rather than warm fuzzies that doesn’t make you a billionaire.
Well, “willing to pay for warm fuzzies” is a bad way to put it IMO. There’s limited amount of money available in the first place, if you care about birds rather than warm fuzzies that doesn’t make you a billionaire.
The figures people would pay to save 2000, 20,000, or 200,000 birds were $80, $78 and $88 respectively, which oughtn’t be so much that the utility of money for most WEIRD people would be significantly non-linear. (A much stronger effect IMO could be people taking—possibly subconsiously—the “2000” or the “20,000” as evidence about the total population of that bird species.)
Utilitarians don’t have to sum different utility functions. An utilitarian has an utility function that happens to be defined as a sum of intermediate values assigned to each individual. Those intermediate values are also (confusingly) referred to as utility but they don’t come from evaluating any of the infinite variety of ‘true’ utility functions of every individual. They come from evaluating the total utilitarian’s model of individual preference satisfaction (or happiness or whatever).
Or at least it seems to me that it should be that way. If I see a simple technical problem that doesn’t really affect the spirit of the argument then the best thing to do is to fix the problem and move on. If total utilitarianism really is commonly defined as summing every individual’s utility function then that is silly but it’s a problem of confused terminology and not really a strong argument against utilitarianism.
Well and then you can have model where the model of individual is sad when the real individual is happy and vice versa, and there would be no problem with that.
You got to ground the symbols somewhere. The model has to be defined to approximate reality for it to make sense, and for the model to approximate reality it has to somehow process individual’s internal state.
Yes. The error is that humans aren’t good at utilitarianism.
private_messaging has given an example elsewhere: the trouble with utilitarians is that they think they are utilitarians. They then use numbers to convince themselves to do something they would otherwise consider evil.
The Soviet Union was an attempt to build a Friendly government based on utilitarianism. They quickly reached “shoot someone versus dust specks” and went for shooting people.
They weren’t that good at lesser utilitarian decisions either, tending to ignore how humans actually behaved in favour of taking their theories and shutting-up-and-multiplying. Then when that didn’t work, they did it harder.
I’m sure someone objecting to the Soviet Union example as non-negligible evidence can come up with examples that worked out much better, of course.
The key difference between a normal utilitarian and a Leninist: When a normal utilitarian concludes that mass murder would maximize social utility, he checks his work! He goes over his calculations with a fine-tooth comb, hoping to discover a way to implement beneficial policy changes without horrific atrocities. The Leninist, in contrast, reasons backwards from the atrocities that emotionally inspire him to the utilitarian argument that morally justifies his atrocities.
If this seems woefully uncharitable, compare the amount of time a proto-Leninist like Raskolnikov spends lovingly reviewing the mere conceivability of morally justified bloodbaths to the amount of time he spends (a) empirically evaluating the effects of policies or (b) searching for less brutal ways to implement whatever policies he wants. These ratios are typical for the entire Russian radical tradition; it’s what they imagined to be “profound.” When men like this gained power in Russia, they did precisely what you’d expect: treat mass murder like a panacea. This is the banality of Leninism.
As I have noted, when you’ve repeatedly emphasised “shut up and multiply”, tacking “btw don’t do anything weird” on the end strikes me as susceptible to your readers not heeding it, particularly when they really need to. If arithmetical utilitarianism works so well, it would work in weird territory.
Caplan does have a cultural point on the Soviet Union example. OTOH, it does seem a bit “no true utilitarian”.
If arithmetical utilitarianism works so well, it would work in weird territory.
Note the bank robbery thread below. Someone claims that “the utilitarian math” shows that robbing banks and donating to charity would have the best consequences. But they don’t do any math or look up basic statistics to do a Fermi calculation. A few minutes of effort shows that bank robbery actually pays much worse than working as a bank teller over the course of a career (including jail time, etc).
In Giving What We Can there are several people who donate half their income (or all income above a Western middle class standard of living) to highly efficient charities helping people in the developing world. They expect to donate millions of dollars over their careers, and to have large effects on others through their examples and reputations, both as individuals and via their impact on organizations like Giving What We Can. They do try to actually work things out, and basic calculations easily show that running around stealing organs or robbing banks would have terrible consequences, thanks to strong empirical regularities:
Crime mostly doesn’t pay. Bank robbers, drug dealers, and the like make less than legitimate careers. They also spend a big chunk of time imprisoned, and ruin their employability for the future. Very talented people who might do better than the average criminal can instead go to Wall Street or Silicon Valley and make far more.
Enormous amounts of good can be done through a normal legitimate career. Committing violent crime, or other hated acts close off such opportunities very rapidly.
Really dedicated do-gooders hope to have most of their influence through example, encouraging others to do good. Becoming a hated criminal, and associating their ethical views with such, should be expected to have huge negative effects by staunching the flow of do-gooders to exploit the vast legitimate opportunities to help people.
If some criminal scheme looks easy and low-risk, consider that law enforcement uses many techniques which are not made public, and are very hard for a lone individual to learn. There are honey-pots, confederates, and so forth. In the market for nuclear materials, most of the buyers and sellers are law enforcement agents trying to capture any real criminal participants. In North America terrorist cells are now regularly infiltrated long before they act, with government informants insinuated into the cell, phone and internet activities monitored, etc.
It is hard to keep a crime secret over time. People feel terrible guilt, and often are caught after they confess to others. In the medium term there is some chance of more effective neuroscience-based lie detectors, which goes still higher long-term.
The broader society, over time, could punish utilitarian villainy by reducing its support for the things utilitarians seek as they are associated with villains, or even by producing utilitarian evils. If animal rights terrorists tried to kill off humanity, it might lead to angry people eating more meat or creating anti-utilitronium (by the terrorists’ standards, not so much the broader society, focused on animals, say) in anger. The 9/11 attacks were not good for Osama bin Laden’s ambitions of ruling Saudi Arabia.
There are other considerations, but these are enough to dispense with the vast bestiary of supposedly utility-boosting sorts of wrongdoing. Arithmetical utilitarianism does say you should not try to become a crook. But unstable or vicious people (see the Caplan Leninist link) sometimes do like to take the idea of “the end justifies the means” as an excuse to go commit crimes without even trying to work out how the means are related to the end, and to alternatives.
Disclaimer: I do not value total welfare to the exclusion of other ethical and personal concerns. My moral feelings oppose deontological nastiness aside from aggregate welfare. But I am tired of straw-manning “estimating consequences” and “utilitarian math” by giving examples where these aren’t used and would have prevented the evil conclusion supposedly attributed to them.
I’m confused. Your comment paints a picture of a super-efficient police force that infiltrates criminal groups long before they act. But the Internet seems to say that many gangs in the US operate openly for years, control whole neighborhoods, and have their own Wikipedia pages...
Gang membership still doesn’t pay relative to regular jobs
The police largely know who is in the gangs, and can crack down if this becomes a higher priority
Terrorism is such a priority, to a degree way out of line with the average historical damage, because of 9/11; many have critiqued the diversion of law enforcement resources to terrorism
Such levels of gang control are concentrated in poor areas with less police funding, and areas where the police are estranged from the populace, limiting police activity.
Gang violence is heavily directed at other criminal gangs, reducing the enthusiasm of law enforcement, relative to more photogenic victims
The other side is that robbing banks at gunpoint isn’t the most effective way to redistribute wealth from those who have it to those to whom it should go.
I suspect that the most efficient way to do that is government seizure- declare that the privately held assets of the bank now belong to the charities. That doesn’t work, because the money isn’t value, it’s a signifier of value, and rewriting the map does not change the territory- if money is forcibly redistributed too much, it loses too much value and the only way to enforce the tax collection is by using the threat of prison and execution- but the jailors and executioners can only be paid by the taxes. Effectively robbing banks to give the money to charity harms everyone significantly, and fails to be better than doing nothing.
It may have been better if CarlShulman used a different word—perhaps ‘Evil’ - to represent the ‘ethical injunctions’ idea. That seems to better represent the whole “deliberately subvert consequentialist reasoning in certain areas due to acknowledgement of corrupted and bounded hardware”. ‘Weird’ seems to be exactly the sort of thing Eliezer might advocate. For example “make yourself into a corpsicle” and “donate to SingInst”.
And “weird” includes many things Eliezer advocates, but I would be very surprised if it did not include things that Eliezer most certainly would not advocate.
And “weird” includes many things Eliezer advocates, but I would be very surprised if it did not include things that Eliezer most certainly would not advocate.
Of course it does. For example: dressing up as a penguin and beating people to death with a live fish. But that’s largely irrelevant. Rejecting ‘weird’ as the class of things that must never be done is not the same thing as saying that all things in that class must be done. Instead, weirdness is just ignored.
I’ve always felt that post was very suspect. Because, if you do the utilitarian math, robbing banks and giving them to charity is still a good deal, even if there’s a very low chance of it working. Your own welfare simply doesn’t play a factor, given the size of the variables you’re playing with. It seems to be that there is a deeper moral reason not to murder organ donors or steal food for the hungry than ‘it might end poorly for you.’
Because, if you do the utilitarian math, robbing banks and giving them to charity is still a good deal
Bank robbery is actually unprofitable. Even setting aside reputation (personal and for one’s ethos), “what if others reasoned similarly,” the negative consequences of the robbery, and so forth you’d generate more expected income working an honest job. This isn’t a coincidence. Bank robbery hurts banks, insurers, and ultimately bank customers, and so they are willing to pay to make it unprofitable.
According to a study by British researchers Barry Reilly, Neil Rickman and Robert Witt written up in this month’s issue of the journal Significance, the average take from a U.S. bank robbery is $4,330. To put that in perspective, PayScale.com says bank tellers can earn as much as $28,205 annually. So, a bank robber would have to knock over more than six banks, facing increasing risk with each robbery, in a year to match the salary of the tellers he’s holding up.
That was a somewhat lazy example, I admit, but consider the most inconvenient possible world. Let’s say you could expect to take a great deal more from a bank robbery. Would it then be valid utilitarian ethics to rob (indirectly) from the rich (us) to give to the poor?
My whole point in the comments on this post has been that it’s a pernicious practice to use such false examples. They leave erroneous impressions and associations. A world where bank-robbery is super-profitable, so profitable as to outweigh the effects of reputation and the like, is not very coherent.
A better example would be something like: “would utilitarians support raising taxes to fund malaria eradication,” or “would a utilitarian who somehow inherited swoopo.com (a dollar auction site) shut down the site or use the revenue to save kids from malaria” or “if a utilitarian inherited the throne in a monarchy like Oman (without the consent of the people) would he spend tax revenues on international good causes or return them to the taxpayers?”
if you do the utilitarian math, robbing banks and giving them to charity is still a good deal
Only if you’re bad at math. Banks aren’t just piggybanks to smash, they perform a useful function in the economy, and to disrupt it has consequences.
Of course I prefer to defeat bad utilitarian math with better utilitarian math rather than with ethical injunctions. But hey, that’s the woe of bounded reason, even without going into the whole corrupted hardware problem: your model is only so good, and heuristics that serve as warning signals have their place.
Yes. The error is that humans aren’t good at utilitarianism.
Why would that be an error? It’s not a requirement for an ethical theory that Homo sapiens must be good at it. If we notice that humans are bad at it, maybe we should make AI or posthumans that are better at it, if we truly view this as the best ethical theory. Besides, if the outcome of people following utilitarianism is really that bad, then utilitarianism would demand (it gets meta now) that people should follow some other theory that overall has better outcomes (see also Parfit’s Reasons and Persons). Another solution is Hare’s proposed “Two-Level Utilitarianism”.
From Wikipedia:
Hare proposed that on a day to day basis, one should think and act like a rule utilitarian and follow a set of intuitive prima facie rules, in order to avoid human error and bias influencing one’s decision-making, and thus avoiding the problems that affected act utilitarianism.
The error is that it’s humans who are attempting to implement the utilitarianism. I’m not talking about hypothetical non-human intelligences, and I don’t think they were implied in the context.
See also Ends Don’t Justify Means (Among Humans): having non-consequentialist rules (e.g. “Thou shalt not murder, even if it seems like a good idea”) can be consequentially desirable since we’re not capable of being ideal consequentialists.
Oh, indeed. But when you’ve repeatedly emphasised “shut up and multiply”, tacking “btw don’t do anything weird” on the end strikes me as susceptible to your readers not heeding it, particularly when they really need to.
I don’t think hypothetical superhuman would be dramatically different in their ability to employ predictive models upon uncertainty. If you increase power so it is to mankind as mankind is to 1 amoeba, you only double anything that is fundamentally logarithmic. While in many important cases there are faster approximations, it’s magical thinking to expect them everywhere; and there are problems where the errors inherently grow exponentially with time even if the model is magically perfect (butterfly effect). Plus, of course, models of other intelligences rapidly get unethical as you try to improve fidelity (if it is emulating people and putting them through torture and dust speck experience to compare values).
Well, those examples would have a lot of “okay we can’t calculate utility here, so we’ll use a principle” and far less faith in direct utilitarianism.
With the torture and dust specks, see, it arrives at counter intuitive conclusion, but it is not proof grade reasoning by any means. Who knows, maybe the correct algorithm for evaluation of torture vs dust specks must have BusyBeaver(10) for the torture, and BusyBeaver(9) for dust specks, or something equally outrageously huge (after all, thought, which is being screwed with by torture, is turing-complete). The 3^^^3 is not a very big number. There are numbers which are big like you wouldn’t believe.
edit: also I think even vastly superhuman entities wouldn’t be very good at consequence evaluation, especially from uncertain start state. In any case, some sorta morality oracle would have to be able to, at very least, take in full specs of human brain and then spit out the understanding of how to trade off the extreme pain of that individual, for dust speck of that individual (at task which may well end up in ultra long computations BusyBeaver(1E10) style. Forget the puny up arrow). That’s an enormously huge problem which the torture-choosers obviously a: haven’t done and b: didn’t even comprehend that something like this would be needed. Which brings us to the final point: the utilitarians are the people whom haven’t slightest clue what it might take to make an utilitarian decision, but are unaware of that deficiency. edit: and also, I would likely take 1/3^^^3 chance of torture over a dust speck. Why? Because dust speck may result in an accident leading up to decades of torturous existence. Dust speck’s own value is still non comparable, it only bothers me because it creates the risk.
edit: note, the busy beaver reference is just an example. Before you can be additively operating on dust specks and pain, and start doing some utilitarian math there, you have to at least understand how the hell is it that an algorithm can be feeling pain, what is the pain exactly (in reductionist terms).
and also, I would likely take 1/3^^^3 chance of torture over a dust speck. Why? Because dust speck may result in an accident leading up to decades of torturous existence
IIRC, in the original torture vs specks post EY specified that none of the dust specks would have any long-term consequence.
I know. Just wanted to point out where the personal preference (easily demonstrable when people e.g. neglect to take inconvenient safety measures) of small chance of torture vs definite dust speck comes from.
You know, I’ve felt that examining the dust speck vs torture dilemma or stuff like that, finding a way to derive an intuitively false conclusion from intuitively true premises, and thereby concluding that the conclusion must be true after all (rather than there’s some kind of flaw in the proof you can’t see yet) is analogous to seeing a proof that 0 equals 1 or that a hamburger is better than eternal happiness or that no feather is dark, not seeing the mistake in the proof straight away, and thereby concluding that the conclusion must be true. Does anyone else feel the same?
Sure.
But it’s not like continuing to endorse my intuitions in the absence of any justification for them, on the assumption that all arguments that run counter to my intuitions, however solid they may seem, must be wrong because my intuitions say so, is noticeably more admirable.
When my intuitions point in one direction and my reason points in another, my preference is to endorse neither direction until I’ve thought through the problem more carefully. What I find often happens is that on careful thought, my whole understanding of the problem tends to alter, after which I may end up rejecting both of those directions.
Well, what you should do, is to recognize that such arguments themselves are built entirely out of intuitions, and their validity rest on conjunction of a significant number of often unstated intuitive assumptions. One should not fall for cargo cult imitation of logic.
There’s no fundamental reason why value should be linear in number of dust specks; it’s nothing but an assumption which may be your personal intuition, but it is still intuition that lacks any justification what so ever, and in so much as it is an uncommon intuition, it even lacks the “if it was wrong it would be debunked” sort of justification. There’s always the Dunning Kruger effect. People least capable of moral (or any) reasoning should be expected to think themselves most capable.
Yeah, that has always been my main problem with that scenario.
There are different ways to sum multiple sources of something. Consider linear vs paralel electrical circuits; the total output depends greatly on how you count the individual voltage sources (or resistors or whatever).
When it comes to suffering, well suffering only exists in consciousness, and each point of consciousness—each mind involved—experiences their own dust speck individually. There is no conscious mind in that scenario who is directly experiencing the totality of the dust specks and suffers accordingly. It is in no way obvious to me that the “right” way to consider the totality of that suffering is to just add it up. Perhaps it is. But unless I missed something, no one arguing for torture so far has actually shown it (as opposed to just assuming it).
Suppose we make this about (what starts as) a single person. Suppose that you, yourself, are going to be copied into all that humongous number of copies. And you are given a choice: before that happens, you will be tortured for 50 years. Or you will be unconscious for 50 years, but after copying each of your copies will get a dust speck in the eye. Either way you get copied, that’s not part of the choice. After that, whatever your choice, you will be able to continue with your lives.
In that case, I don’t care about doing the “right” math that will make people call me rational, I care about being the agent who is happily NOT writhing in pain with 50 years more of it ahead of him.
EDIT: come to think of it, assume the copying template is taken from you before the 50 years start, so we don’t have to consider memories and lasting psychological effects of torture. My answer remains the same, even if in future I won’t remember the torture, I don’t want to go through it.
As far as I know, TvDS doesn’t assume that value is linear in dust specks. As you say, there are different ways to sum multiple sources of something. In particular, there are many ways to sum the experiences of multiple individuals.
For example, the whole problem evaporates if I decide that people’s suffering only matters to the extent that I personally know those people. In fact, much less ridiculous problems also evaporate… e.g., in that case I also prefer that thousands of people suffer so that I and my friends can live lives of ease, as long as the suffering hordes are sufficiently far away.
It is not obvious to me that I prefer that second way of thinking, though.
It is arguable (in terms of revealed preferences) that first-worlders typically do prefer that. This requires a slightly non-normative meaning of “prefer”, but a very useful one.
Oh, absolutely. I chose the example with that in mind.
I merely assert that “but that leads to thousands of people suffering!” is not a ridiculous moral problem for people (like me) who reveal such preferences to consider, and it’s not obvious that a model that causes the problem to evaporate is one that I endorse.
Well, it sure uses linear intuition. 3^^^3 is bigger than number of distinct states, its far past point where you are only increasing exactly-duplicated dust speck experience, so you could reasonably expect it to flat out.
One can go perverse and proclaims that one treats duplicates the same, but then if there’s a button which you press to replace everyone’s mind with mind of happiest person, you should press it.
I think the stupidity of utilitarianism is the belief that the morality is about the state, rather than about dynamic process and state transition. Simulation of pinprick slowed down 1000000 times is not ultra long torture. The ‘murder’ is a form of irreversible state transition. The morality as it exist is about state transitions not about states.
It isn’t clear to me what the phrase “exactly-duplicated” is doing there. Is there a reason to believe that each individual dust-speck-in-eye event is exactly like every other? And if so, what difference does that make? (Relatedly, is there a reason to believe that each individual moment of torture is different from all the others? If it turns out that it’s not, does that imply something relevant?)
In any case, I certainly agree that one could reasonably expect the negvalue of suffering to flatten out no matter how much of it there is. It seems unlikely to me that fifty years of torture is anywhere near the asymptote of that curve, though… for example, I would rather be tortured for fifty years than be tortured for seventy years.
But even if it somehow is at the asymptotic limit, we could recast the problem with ten years of torture instead, or five years, or five months, or some other value that is no longer at that limit, and the same questions would arise.
So, no, I don’t think the TvDS problem depends on intuitions about the linear-additive nature of suffering. (Indeed, the more i think about it the less convinced i am that I have such intuitions, as opposed to approaches-a-limit intuitions. This is perhaps because thinking about it has changed my intuitions.)
I was referring to linear-additive nature of dust speck so called suffering, in the number of people with dust specks.
3^^^3 is far far larger than number of distinct mind states of anything human-like. You can only be dust-speck-ing something like 10^(10^20) distinct human-like entities maximum. I recall i posted about that a while back. You shouldn’t be multiplying anything with 3^^^3 .
TBH, my ‘common sense’ explanation as of why EY chooses to adopt torture > dust specks stance (i say chooses because it is entirely up to grabs here plus his position is fairly incoherent), is because he seriously believes that his work has non negligible chance of influencing lives of an enormous number of people, and subsequently if he can internalize the torture>dust specks, he is free to rationalize any sort of thing he can plausibly do, even if AI extinction risk does not exist.
[edit: this response was to an earlier version of the above comment, before it was edited. Some of it is no longer especially apposite to the comment as it exists now.]
Well, I agree that 3^^^3 dust specks don’t quite add linearly… long before you reach that ridiculous mass, I expect you get all manner of weird effects that I’m not physicist enough to predict. And I also agree that our intuitions are that dust specks add linearly.
But surely it’s not the dust-specks that we care about here, but the suffering? That is, it seems clear to me that if we eliminated all the dust specks from the scenario and replaced them with something that caused an equally negligible amount of suffering, we would not be changing anything that mattered about the scenario.
And, as I said, it’s not at all clear to me that I intuit linear addition of suffering (whether it’s caused by dust-specks, torture, or something else), or that the scenario depends on assuming linear addition of suffering. It merely depends on assuming that addition of multiple negligible amounts of suffering can lead to an aggregate-suffering result that is commensurable with, and greater than, a single non-negligible amount of suffering.
It’s not clear to me that this assumption holds, but the linear-addition objection seems like a red herring to me.
Ah, I see.
Yeah, sure, there’s only X possible ways for a human to be (whether 10^(10^20) or some other vast number doesn’t really matter), and there’s only Y possible ways for a dust speck to be, and there’s only Z possible ways for a given human to experience a given dust speck in their eye. So, sure, we only have (XYZ) distinct dust-speck-in-eye events, and if (XYZ) << 3^^^3 then there’s some duplication. Indeed, there’s vast amounts of duplication, given that (3^^^3/(XYZ)) is still a staggeringly huge number.
Agreed.
I’m still curious about what difference that makes.
Well, some difference that it should make:
Lead to severe discounting of the ‘reasoning method’ that arrived at 3^^^3 dust-specks>torture conclusion without ever coming across the exhaustion of states issue. In all fields where it was employed. And to severely discount anything that came from that process previously. If it failed even though it gone against intuition, it’s even more worthless when it goes along with intuition.
I get the feeling that attempts to ‘logically’ deliberate on morality from some simple principles like “utility” are similar to trying to recognize cats in pictures by reading R,G,B number value array and doing some arithmetic. If someone haven’t got visual cortex they can’t see, even if they do insane amount of reasoning deliberately.
But a computer can recognize cats by reading pixel values in pictures? Maybe not as efficiently and accurately as people, but that’s because brains have a more efficient architecture/algorithms than today’s generic computers.
Yes, it is of course possible in principle (in fact I am using cats as example because Google just did that). The point is that a person can’t do anything equivalent to what human visual cortex does in a fraction of a second by using paper and pencil for multiple lifetimes. The morality and the immorality, just like cat recognition, rely on some innate human ability of connecting symbols with reality.
edit: To clarify. To tell which images are cats and which are dogs, you employ some method that is hopelessly impossible for you to write down. To tell what actions are moral or not, humans employ some method that is likewise hopelessly impossible for them to write down. All you can do is write down guidelines, and add some picture examples of cats and dogs. Various rules like utilitarianism are along the lines of “if the eyes have vertical slits, it’s a cat” which mis-recognize a lizard as a cat but do not recognize the cat that closed the eyes. (There is also the practical matter of law making, where you want to restrict the diversity of moral judgment to something sane, and thus you use principles like ‘if it doesn’t harm anyone else it’s okay’)
Right, but if/when we get to (partial) brain emulations (in large quantities) we might be able to do the same thing for ‘morality’ that we do today to recognize cats using a computer.
Agreed. We may even see how it is that certain algorithms (very broadly speaking) can feel pain etc, and actually start defining something agreeable from first principles. Meanwhile, all that 3^^^3 people with dustspeck worse than 1 person tortured stuff is to morality as scholasticism is to science. The only value it may have is in highlighting the problem with approximations, and with handwavy reasoning (nobody said that the number of possible people is >3^^^3 (which is false) , even though such statement was a part of reasoning and should have been stated and then rejected invalidating everything that followed. Or a statement that identical instances matter should have been made, which in itself leads to multitude of really dumb decisions whereby the life of a conscious robot that has thicker wires in its computer (or uses otherwise redundant hardware) is worth more)
Not many people hold the view that if eternal inflation is true then there is nothing wrong with hitting people with hot pokers, since the relevant brain states exist elsewhere anyway. In Bostrom’s paper he could only find a single backer of the view. In talking to many people, I have seen it expressed more than once, but still only a very small minority of cases. Perhaps not including it in that post looms large for you because you have a strong intuition that it would be OK to torture and kill if the universe were very large, or think it very unlikely that the universe is large, but it’s a niche objection to address.
After all, one could include such a discussion as a rider in every post talking about trying to achieve anything for oneself or others: “well, reading this calculus textbook seems like it could teach you interesting math, but physicists say we might be living in a big universe, in which case there’s no point since brains in all states already exist, if you don’t care about identical copies.”
If there is any nonzero probability that universe is NOT very large (or the copy counting is a bit subtle about the copies which are effectively encoding state onto coordinates), all you did is scaled all the utilities down, which does not affect any decision.
That’s incredibly terrible thing to do for our friends the people who believe themselves to be utilitarian, as those people are going to selectively scale down just some of the utilities and then act, in self interest or otherwise, out of resulting big differences, doing something stupid.
edit: also, the issue with multiple-counting redundant hardware and the thick-wired utility monsters in the utilitarianism that does count extra copies doesn’t go away if the world is big. If you have a solid argument that utilitarianism without counting the extra copies the same does not work, that means utilitarianism does not work. Which I believe is the case. The morals are an engineered / naturally selected solution to problem of peer to peer intellectual and other cooperation, which requires nodes not to model each other in undue detail, which rules out direct straightforward utilitarianism. The utilitarianism is irreparably broken. It’s fake-reductionism where you substitute one irreducible concept for another.
That’s an interesting idea, thanks. Maybe caring about anthropic probabilities or measures of conscious experiences directly would make more sense than caring about the number of copies as a proxy.
If you take that idea seriously and assume that all anthropic probabilities of conscious experiences must sum to 1, then torture vs dustspecks seems to lose some of its sting, because the total disutility of dustspecking remains bounded and not very high, no matter how many people you dustspeck. (That’s a little similar to the “proximity argument”, which says faraway people matter less.) And being able to point out the specific person to be tortured means that person doesn’t have too low weight, so torturing that single person would be worse than dustspecking literally everyone else in the multiverse. I don’t remember if anyone made this argument before… Of course there could be any number of holes in it.
Also note that the thicker wires argument is not obviously wrong, because for all we know, thicker wires could affect subjective probabilities. It sounds absurd, sure, but so does the fact that lightspeed is independent of observer speed.
ETA: the first version of this comment mixed up Pascal’s mugging and torture vs dustspecks. Sorry. Though maybe a similar argument could be made for Pascal’s mugging as well.
Thinking about it some more: maybe the key is that it is not enough for something to exist somewhere, just as it is not enough for output tape in Solomonoff induction to contain the desired output string somewhere within it, it should begin with it. (Note that it is a critically important requirement). If you are using Solomonoff induction (suppose you got oracle and suppose universe is computable and so on), then your model contains not only laws of universe but also locator, and my intuition is that one model that has simplest locator is some very huge length shorter than the next simplest model, so all the other models except the one with simplest locator, have to be ignored entirely.
If we require that the locator is present somehow in the whole then the ultra-distant copies are very different while the nearby copies are virtually the same, and Kolmogorov complexity of concatenated strings can be used for count, not counting twice nearby copies (the thick wired monster only weights a teeny tiny bit more).
TBH i feel tho that utilitarianism goes in the wrong direction entirely. Morals can be seen as evolved / engineered solution to peer to peer intellectual and other cooperation, essentially. It relies on trust, not on mutual detailed modeling (which wastes computing power), and the actions are not quite determined by the expected state (which you can’t model), even though it is engineered with some state in mind.
edit: also I think the what ever stuff raises the problem with distant copies or MWI is subjectively disproved by this not saving you from brain damage of any kind (you can get drunk, pass out, wake up with a little bit fewer neurons). So we basically know something’s screwed up with naive counting for probabilities, or the world is small.
This is mistaken. E.g. see this post which discusses living in a Big World, as in eternal inflation theories where the universe extends infinitely and has random variation so that somewhere in the universe every possible galaxy or supercluster will be realized, and all the human brain states will be explored.
Or see Bostrom’s paper on this issue, which is very widely read around here. Many people think that our actions can still matter in such a world, e.g. that it’s better to try to give people chocolate than to torture them here on Earth, even if in ludicrously distant region there are brains that have experienced all the variations of chocolate and torture.
Even better, to my mind, is to think about the scenario from the ground up and form my own conclusions, rather than start with some intuitive judgment about someone else’s writeup about it and then update that judgment based on things they didn’t mention in that writeup.
It’s not clear to me that I correctly understand what you mean here, but given my current understanding, I disagree. All my visual cortex is doing is performing computations on the output of my eyes; if that’s seeing, then anything else that performs the same computations can see just as well.
The point is that the approach is flawed; one should always learn on mistakes. The issue here is in building an argument which is superficially logical—conforms to the structure of something a logical rational person might say—something you might have a logical character in a movie say—but is fundamentally a string of very shaky intuitions which are only correct if nothing outside the argument interferes, rather than solid steps.
In theory. In practice it takes ridiculous number of operations, and you can’t chinese-room vision without slow-down by factor of billions. Decades for single-cat recognition versus fraction of a second, and that’s if you got an algorithm for it.
I disagree with pretty much all of this, as well as with most of what seem to be the ideas underlying it, and don’t see any straightforward way to achieve convergence rather than infinitely ramified divergence, so I suppose it’s best for me to drop the thread here.
“State” doesn’t have to mean “frozen state” or something similar, it could mean “state of the world/universe”. E.g. “a state of the universe” in which many people are being tortured includes the torture process in it’s description. I think this is how it’s normally used.
Well, if you are to coherently take it that the transitions have value, rather than states, then you arrive at morality that regulates the transitions that the agent should try to make happen, ending up with morality that is more about means than about ends.
I think it’s simply that the pain feels like a state rather than dynamic process, and so utilitarianism treats it as state, while doing something feels like a dynamic process, so utilitarianism doesn’t treat it as state and is only concerned with difference in utilities.
Agreed that all of these sorts of arguments ultimately rest on different intuitions about morality, which sometimes conflict, or seem to conflict.
Agreed that value needn’t add linearly, and indeed my intuition is that it probably doesn’t.
It seems clear to me that if I negatively value something happening, I also negatively value it happening more more. That is, for any X I don’t want to have happen, it seems I would rather have X happen than have X happen twice. I can’t imagine an X where I don’t want X to happen and would prefer to have X happen twice than once. (Barring silly examples like “the power switch for the torture device gets flipped”.)
Can anyone explain what goes wrong if you say something like, “The marginal utility of my terminal values increases asymtotically, and u(Torture) approaches a much higher asymptote than u(Dust speck)” (or indeed whether it goes wrong at all)?
That’s been done in this paper, secion VI “The Asymptotic Gambit”.
Thank you. I had expected the bottom to drop out of it somehow.
EDIT: Although come to think of it I’m not sure the objections presented in that paper are so deadly after all if you takes TDT-like considerations into account (i.e. there would not be a difference between “kill 1 person, prevent 1000 mutilations” + “kill 1 person, prevent 1000 mutilations” and “kill 2 people, prevent 2000 mutilations”.) Will have to think on it some more.
Nothing, iif that happens to be be what your actual preferences are. If your preferences did not happen to be as you describe but instead you are confused by an inconsistency in your intuitions then you will make incorrect decisions.
The challenge is not to construct a utility function such that you can justify it to others in the face of opposition. The challenge is to work out what your actual preferences are and implement them.
Ayup. Also, it may be worth saying explicitly that a lot of the difficulty comes in working out a model of my actual preferences that is internally consistent and can be extended to apply to novel situations. If I give up those constraints, it’s easier to come up with propositions that seem to model my preferences, because they approximate particular aspects of my preferences well enough that in certain situations I can’t tell the difference. And if I don’t ever try to make decisions outside of that narrow band of situations, that can be enough to satisfy me.
[Edited to separate from quote] But doesn’t that beg the question? Don’t you have to ask a the meta question “what kinds of preferences are reasonable to have?” Why should we shape ethics the way evolution happened to set up our values? That’s why I favor hedonistic utiltiarianism that is about actual states of the world that can in itself be bad (--> suffering).
Note that markup requires a blank line between your quote and the rest of the topic.
It does beg a question: specifically, the question of whether I ought to implement my preferences (or some approximation of them) in the first place. If, for example, my preferences are instead irrelevant to what I ought to do, then time spent working out my preferences is time that could better have been spent doing something else.
All of that said, it sounds like you’re suggesting that suffering is somehow unrelated to the way evolution set up our values. If that is what you’re suggesting, then I’m completely at a loss to understand either your model of what suffering is, or how evolution works.
The fact that suffering feels awful is about the very thing, and nothing else. There’s no valuing required, no being ask itself “should I dislike this experience” when it is in suffering. It wouldn’t be suffering otherwise.
My position implies that in a world without suffering (or happiness, if I were not a negative utiltiarian), nothing would matter.
Depends on what I’m trying to do.
If I make that assumption, then it follows that given enough Torture to approach its limit, I choose any number of Dust Specks rather than that amount of Torture.
If my goal is to come up with an algorithm that leads to that choice, then I’ve succeeded.
(I think talking about Torture and Dust Specks as terminal values is silly, but it isn’t necessary for what I think you’re trying to get at.)
Nope! Some proofs are better-supported than others.
Yes. The known unreliability of my own thought processes tempers my confidence in any prima facie absurd conclusion I come to. All the more so when it’s a conclusion I didn’t come to, but merely followed along with someone else’s argument to.
I feel this way. The linear theories are usually nothing but first order approximations.
Also, the very idea of summing of individual agent utilities… that’s, frankly, nothing but pseudomathematics. Each agent’s utility function can be modified without changing agent’s behaviour in any way. The utility function is a phantom. It isn’t so defined that you could add two of them together. You can map same agent’s preferences (whenever they are well ordered) to infinite variety of real valued ‘utility functions’.
Yes. The trouble with “shut up and multiply” - beyond assuming that humans have a utility function at all—is assuming that utility works like conventional arithmetic and that you can in fact multiply.
There’s also measuring and shut-up-and-multiplying the wrong thing: e.g., seeing people willing to pay about the same in total to save 2000 birds or 20,000 birds and claiming this constitutes “scope insensitivity.” The error is assuming this means that people are scope-insensitive, rather than to realise that people aren’t buying saved birds at all, but are paying what they’re willing to pay for warm fuzzies in general—a constant amount.
The attraction of utilitarianism is that calculating actions would be so much simpler if utility functions existed and their output could be added with the same sort of rules as conventional arithmetic. This does not, however, constitute non-negligible evidence that any of the required assumptions hold.
It even tends to count against it, by the A+B rule. If items are selected by a high enough combined score on two criteria A and B, then among the selected items, there will tend to be a negative correlation between A and B.
I don’t know who’s making that error. Seems like scope insensitivity and purchasing of warm fuzzies are usually discussed together around here.
Anyway, if there’s an error here then it isn’t about utilitarianism vs something else, but about declared vs revealed preference. The people believe that they care about the birds. They don’t act as if they cared about the birds. For those who accept deliberative reasoning as an expression of human values it’s a failure of decision-making intuitions and it’s called scope insensitivity. For those who believe that true preference is revealed through behavior it’s a failure of reflection. None of those positions seems inconsistent with utilitarianism. In fact it might be easier to be a total utilitarian if you go all the way and conclude that humans really care only about power and sex. Just give everybody nymphomania and megalomania, prohibit birth control and watch that utility counter go. ;)
An explanatory reply from the downvoter would be useful. I’d like to think I could learn.
I don’t think it’s even linearly combinable. Suppose there were 4 copies of me total, pair doing some identical thing, other pair doing 2 different things. The second pair is worth more. When I see someone go linear on morals, that strikes me as evidence of poverty of moral value and/or poverty of mathematical language they have available.
Then the consequentialism. The consequences are hard to track—got to model the worlds resulting from uncertain initial state. Really really computationally expensive. Everything is going to use heuristics, even jupiter brains.
Well, “willing to pay for warm fuzzies” is a bad way to put it IMO. There’s limited amount of money available in the first place, if you care about birds rather than warm fuzzies that doesn’t make you a billionaire.
The figures people would pay to save 2000, 20,000, or 200,000 birds were $80, $78 and $88 respectively, which oughtn’t be so much that the utility of money for most WEIRD people would be significantly non-linear. (A much stronger effect IMO could be people taking—possibly subconsiously—the “2000” or the “20,000” as evidence about the total population of that bird species.)
Utilitarians don’t have to sum different utility functions. An utilitarian has an utility function that happens to be defined as a sum of intermediate values assigned to each individual. Those intermediate values are also (confusingly) referred to as utility but they don’t come from evaluating any of the infinite variety of ‘true’ utility functions of every individual. They come from evaluating the total utilitarian’s model of individual preference satisfaction (or happiness or whatever).
Or at least it seems to me that it should be that way. If I see a simple technical problem that doesn’t really affect the spirit of the argument then the best thing to do is to fix the problem and move on. If total utilitarianism really is commonly defined as summing every individual’s utility function then that is silly but it’s a problem of confused terminology and not really a strong argument against utilitarianism.
But the spirit of the argument is ungrounded in anything. What evidence is there that you can do this stuff at all using actual numbers without repeatedly bumping into “don’t do non-normative things even if you got that answer from a shut-up-and-multiply”?
Well and then you can have model where the model of individual is sad when the real individual is happy and vice versa, and there would be no problem with that.
You got to ground the symbols somewhere. The model has to be defined to approximate reality for it to make sense, and for the model to approximate reality it has to somehow process individual’s internal state.
Yes. The error is that humans aren’t good at utilitarianism.
private_messaging has given an example elsewhere: the trouble with utilitarians is that they think they are utilitarians. They then use numbers to convince themselves to do something they would otherwise consider evil.
The Soviet Union was an attempt to build a Friendly government based on utilitarianism. They quickly reached “shoot someone versus dust specks” and went for shooting people.
They weren’t that good at lesser utilitarian decisions either, tending to ignore how humans actually behaved in favour of taking their theories and shutting-up-and-multiplying. Then when that didn’t work, they did it harder.
I’m sure someone objecting to the Soviet Union example as non-negligible evidence can come up with examples that worked out much better, of course.
See Eliezer’s Ethical Injunctions post.
Also Bryan Caplan:
As I have noted, when you’ve repeatedly emphasised “shut up and multiply”, tacking “btw don’t do anything weird” on the end strikes me as susceptible to your readers not heeding it, particularly when they really need to. If arithmetical utilitarianism works so well, it would work in weird territory.
Caplan does have a cultural point on the Soviet Union example. OTOH, it does seem a bit “no true utilitarian”.
Note the bank robbery thread below. Someone claims that “the utilitarian math” shows that robbing banks and donating to charity would have the best consequences. But they don’t do any math or look up basic statistics to do a Fermi calculation. A few minutes of effort shows that bank robbery actually pays much worse than working as a bank teller over the course of a career (including jail time, etc).
In Giving What We Can there are several people who donate half their income (or all income above a Western middle class standard of living) to highly efficient charities helping people in the developing world. They expect to donate millions of dollars over their careers, and to have large effects on others through their examples and reputations, both as individuals and via their impact on organizations like Giving What We Can. They do try to actually work things out, and basic calculations easily show that running around stealing organs or robbing banks would have terrible consequences, thanks to strong empirical regularities:
Crime mostly doesn’t pay. Bank robbers, drug dealers, and the like make less than legitimate careers. They also spend a big chunk of time imprisoned, and ruin their employability for the future. Very talented people who might do better than the average criminal can instead go to Wall Street or Silicon Valley and make far more.
Enormous amounts of good can be done through a normal legitimate career. Committing violent crime, or other hated acts close off such opportunities very rapidly.
Really dedicated do-gooders hope to have most of their influence through example, encouraging others to do good. Becoming a hated criminal, and associating their ethical views with such, should be expected to have huge negative effects by staunching the flow of do-gooders to exploit the vast legitimate opportunities to help people.
If some criminal scheme looks easy and low-risk, consider that law enforcement uses many techniques which are not made public, and are very hard for a lone individual to learn. There are honey-pots, confederates, and so forth. In the market for nuclear materials, most of the buyers and sellers are law enforcement agents trying to capture any real criminal participants. In North America terrorist cells are now regularly infiltrated long before they act, with government informants insinuated into the cell, phone and internet activities monitored, etc.
It is hard to keep a crime secret over time. People feel terrible guilt, and often are caught after they confess to others. In the medium term there is some chance of more effective neuroscience-based lie detectors, which goes still higher long-term.
The broader society, over time, could punish utilitarian villainy by reducing its support for the things utilitarians seek as they are associated with villains, or even by producing utilitarian evils. If animal rights terrorists tried to kill off humanity, it might lead to angry people eating more meat or creating anti-utilitronium (by the terrorists’ standards, not so much the broader society, focused on animals, say) in anger. The 9/11 attacks were not good for Osama bin Laden’s ambitions of ruling Saudi Arabia.
There are other considerations, but these are enough to dispense with the vast bestiary of supposedly utility-boosting sorts of wrongdoing. Arithmetical utilitarianism does say you should not try to become a crook. But unstable or vicious people (see the Caplan Leninist link) sometimes do like to take the idea of “the end justifies the means” as an excuse to go commit crimes without even trying to work out how the means are related to the end, and to alternatives.
Disclaimer: I do not value total welfare to the exclusion of other ethical and personal concerns. My moral feelings oppose deontological nastiness aside from aggregate welfare. But I am tired of straw-manning “estimating consequences” and “utilitarian math” by giving examples where these aren’t used and would have prevented the evil conclusion supposedly attributed to them.
I’m confused. Your comment paints a picture of a super-efficient police force that infiltrates criminal groups long before they act. But the Internet seems to say that many gangs in the US operate openly for years, control whole neighborhoods, and have their own Wikipedia pages...
The gangs do well, and the rare criminals who become successful gang leaders may sometimes do well, but does the average gangster do well?
Gang membership still doesn’t pay relative to regular jobs
The police largely know who is in the gangs, and can crack down if this becomes a higher priority
Terrorism is such a priority, to a degree way out of line with the average historical damage, because of 9/11; many have critiqued the diversion of law enforcement resources to terrorism
Such levels of gang control are concentrated in poor areas with less police funding, and areas where the police are estranged from the populace, limiting police activity.
Gang violence is heavily directed at other criminal gangs, reducing the enthusiasm of law enforcement, relative to more photogenic victims
The other side is that robbing banks at gunpoint isn’t the most effective way to redistribute wealth from those who have it to those to whom it should go.
I suspect that the most efficient way to do that is government seizure- declare that the privately held assets of the bank now belong to the charities. That doesn’t work, because the money isn’t value, it’s a signifier of value, and rewriting the map does not change the territory- if money is forcibly redistributed too much, it loses too much value and the only way to enforce the tax collection is by using the threat of prison and execution- but the jailors and executioners can only be paid by the taxes. Effectively robbing banks to give the money to charity harms everyone significantly, and fails to be better than doing nothing.
It may have been better if CarlShulman used a different word—perhaps ‘Evil’ - to represent the ‘ethical injunctions’ idea. That seems to better represent the whole “deliberately subvert consequentialist reasoning in certain areas due to acknowledgement of corrupted and bounded hardware”. ‘Weird’ seems to be exactly the sort of thing Eliezer might advocate. For example “make yourself into a corpsicle” and “donate to SingInst”.
But, of course, “weird” versus “evil” is not even broadly agreed upon.
And “weird” includes many things Eliezer advocates, but I would be very surprised if it did not include things that Eliezer most certainly would not advocate.
Of course it does. For example: dressing up as a penguin and beating people to death with a live fish. But that’s largely irrelevant. Rejecting ‘weird’ as the class of things that must never be done is not the same thing as saying that all things in that class must be done. Instead, weirdness is just ignored.
I’ve always felt that post was very suspect. Because, if you do the utilitarian math, robbing banks and giving them to charity is still a good deal, even if there’s a very low chance of it working. Your own welfare simply doesn’t play a factor, given the size of the variables you’re playing with. It seems to be that there is a deeper moral reason not to murder organ donors or steal food for the hungry than ‘it might end poorly for you.’
Bank robbery is actually unprofitable. Even setting aside reputation (personal and for one’s ethos), “what if others reasoned similarly,” the negative consequences of the robbery, and so forth you’d generate more expected income working an honest job. This isn’t a coincidence. Bank robbery hurts banks, insurers, and ultimately bank customers, and so they are willing to pay to make it unprofitable.
That was a somewhat lazy example, I admit, but consider the most inconvenient possible world. Let’s say you could expect to take a great deal more from a bank robbery. Would it then be valid utilitarian ethics to rob (indirectly) from the rich (us) to give to the poor?
My whole point in the comments on this post has been that it’s a pernicious practice to use such false examples. They leave erroneous impressions and associations. A world where bank-robbery is super-profitable, so profitable as to outweigh the effects of reputation and the like, is not very coherent.
A better example would be something like: “would utilitarians support raising taxes to fund malaria eradication,” or “would a utilitarian who somehow inherited swoopo.com (a dollar auction site) shut down the site or use the revenue to save kids from malaria” or “if a utilitarian inherited the throne in a monarchy like Oman (without the consent of the people) would he spend tax revenues on international good causes or return them to the taxpayers?”
Only if you’re bad at math. Banks aren’t just piggybanks to smash, they perform a useful function in the economy, and to disrupt it has consequences.
Of course I prefer to defeat bad utilitarian math with better utilitarian math rather than with ethical injunctions. But hey, that’s the woe of bounded reason, even without going into the whole corrupted hardware problem: your model is only so good, and heuristics that serve as warning signals have their place.
Why would that be an error? It’s not a requirement for an ethical theory that Homo sapiens must be good at it. If we notice that humans are bad at it, maybe we should make AI or posthumans that are better at it, if we truly view this as the best ethical theory. Besides, if the outcome of people following utilitarianism is really that bad, then utilitarianism would demand (it gets meta now) that people should follow some other theory that overall has better outcomes (see also Parfit’s Reasons and Persons). Another solution is Hare’s proposed “Two-Level Utilitarianism”. From Wikipedia:
The error is that it’s humans who are attempting to implement the utilitarianism. I’m not talking about hypothetical non-human intelligences, and I don’t think they were implied in the context.
See also Ends Don’t Justify Means (Among Humans): having non-consequentialist rules (e.g. “Thou shalt not murder, even if it seems like a good idea”) can be consequentially desirable since we’re not capable of being ideal consequentialists.
Oh, indeed. But when you’ve repeatedly emphasised “shut up and multiply”, tacking “btw don’t do anything weird” on the end strikes me as susceptible to your readers not heeding it, particularly when they really need to.
I don’t think hypothetical superhuman would be dramatically different in their ability to employ predictive models upon uncertainty. If you increase power so it is to mankind as mankind is to 1 amoeba, you only double anything that is fundamentally logarithmic. While in many important cases there are faster approximations, it’s magical thinking to expect them everywhere; and there are problems where the errors inherently grow exponentially with time even if the model is magically perfect (butterfly effect). Plus, of course, models of other intelligences rapidly get unethical as you try to improve fidelity (if it is emulating people and putting them through torture and dust speck experience to compare values).
Well, those examples would have a lot of “okay we can’t calculate utility here, so we’ll use a principle” and far less faith in direct utilitarianism.
With the torture and dust specks, see, it arrives at counter intuitive conclusion, but it is not proof grade reasoning by any means. Who knows, maybe the correct algorithm for evaluation of torture vs dust specks must have BusyBeaver(10) for the torture, and BusyBeaver(9) for dust specks, or something equally outrageously huge (after all, thought, which is being screwed with by torture, is turing-complete). The 3^^^3 is not a very big number. There are numbers which are big like you wouldn’t believe.
edit: also I think even vastly superhuman entities wouldn’t be very good at consequence evaluation, especially from uncertain start state. In any case, some sorta morality oracle would have to be able to, at very least, take in full specs of human brain and then spit out the understanding of how to trade off the extreme pain of that individual, for dust speck of that individual (at task which may well end up in ultra long computations BusyBeaver(1E10) style. Forget the puny up arrow). That’s an enormously huge problem which the torture-choosers obviously a: haven’t done and b: didn’t even comprehend that something like this would be needed. Which brings us to the final point: the utilitarians are the people whom haven’t slightest clue what it might take to make an utilitarian decision, but are unaware of that deficiency. edit: and also, I would likely take 1/3^^^3 chance of torture over a dust speck. Why? Because dust speck may result in an accident leading up to decades of torturous existence. Dust speck’s own value is still non comparable, it only bothers me because it creates the risk.
edit: note, the busy beaver reference is just an example. Before you can be additively operating on dust specks and pain, and start doing some utilitarian math there, you have to at least understand how the hell is it that an algorithm can be feeling pain, what is the pain exactly (in reductionist terms).
IIRC, in the original torture vs specks post EY specified that none of the dust specks would have any long-term consequence.
I know. Just wanted to point out where the personal preference (easily demonstrable when people e.g. neglect to take inconvenient safety measures) of small chance of torture vs definite dust speck comes from.