Agree, having lived in chronic pain supposedly worse than untrained childbirth, I’d say that even an hour has a really seriously different possibility in terms of capacity for suffering than a day, and a day different from a week. For me it breaks down somewhere, even when multiplying between the 10^15 for 1 day and 10^21 for one minute. You can’t really feel THAT much pain in a minute that is comparable to a day, even orders of magnitude? Its just qualitatively different. Interested to hear pushback on this
We could go from a day to a minute more slowly; for example, by increasing the number of people by a factor of a googolplex every time the torture time decreases by 1 second.
I absolutely agree that the length of torture increases how bad it is in nonlinear ways, but this doesn’t mean we can’t find exponential factors that dominate it at every point at least along the “less than 50 years” range.
Absolutely. We’re bad at anything that we can’t easily imagine. Probably, for many people, intuition for “torture vs. dust specks” imagines a guy with a broken arm on one side, and a hundred people saying ‘ow’ on the other.
The consequences of our poor imagination for large numbers of people (i.e. scope insensitivity) are well-studied. We have trouble doing charity effectively because our intuition doesn’t take the number of people saved by an intervention into account; we just picture the typical effect on a single person.
What, I wonder, are the consequence of our poor imagination for extremity of suffering? For me, the prison system comes to mind: I don’t know how bad being in prison is, but it probably becomes much worse than I imagine if you’re there for 50 years, and we don’t think about that at all when arguing (or voting) about prison sentences.
My heuristic for dealing with such situations is somewhat reminiscent of Hofstadter’s Law: however bad you imagine it to be, it’s worse than that, even when you take the preceding statement into account. In principle, this recursion should go on forever and lead to you regarding any sufficiently unimaginably bad situation as infinitely bad, but in practice, I’ve yet to have it overflow, probably because your judgment spontaneously regresses back to your original (inaccurate) representation of the situation unless consciously corrected for.
My feeling is that situations like being caught for doing something horrendous might or might not be subject to psychological adjustment—that many situations of suffering are subject to psychological adjustment and so might actually be not as bad as we though. But chronic intense pain, is literally unadjustable to some degree—you can adjust to being in intense suffering but that doesn’t make the intense suffering go away. That’s why I think its a special class of states of being—one that invokes action. What do people think?
That strikes me as a deliberate set up for a continuum fallacy.
Also, why are you so sure that the number of people increases suffering in a linear way for even very large numbers? What is a number of people anyway?
I’d much prefer to have a [large number of exact copies of me] experience 1 second of headache than for one me to suffer it for a whole day. Because those copies they don’t have any mechanism which could compound their suffering. They aren’t even different subjectivities. I don’t see any reason why a hypothetical mind upload of me running on multiple redundant hardware should be an utility monster, if it can’t even tell subjectively how redundant it’s hardware is.
Some anaesthetics do something similar, preventing any new long term memories, people have no problem with taking those for surgery. Something’s still experiencing pain but it’s not compounding into anything really bad (unless the drugs fail to work, or unless some form of long term memory still works). A real example of a very strong preference for N independent experiences of 30 seconds of pain over 1 experience of 30*N seconds of pain.
It’s not a continuum fallacy because I would accept “There is some pair (N,T) such that (N people tortured for T seconds) is worse than (10^100 N people tortured for T-1 seconds), but I don’t know the exact values of N and T” as an answer. If, on the other hand, the comparison goes the other way for any values of N and T, then you have to accept the transitive closure of those comparisons as well.
Also, why are you so sure that the number of people increases suffering in a linear way for even very large numbers? What is a number of people anyway?
I’m not sure what you mean by this. I don’t believe in linearity of suffering: that would be the claim that 2 people tortured for 1 day is the same as 1 person tortured for 2 days, and that’s ridiculous. I believe in comparability of suffering, which is the claim that for some value of N, N people tortured for 1 day is worse than 1 person tortured for 2 days.
Regarding anaesthetics: I would prefer a memory inhibitor for a painful surgery to the absence of one, but I would still strongly prefer to feel less pain during the surgery even if I know I will not remember it one way or the other. Is this preference unusual?
I believe in comparability of suffering, which is the claim that for some value of N, N people tortured for 1 day is worse than 1 person tortured for 2 days.
This is where the argument for choosing torture falls apart for me, really. I don’t think there is any number of people getting dust specks in their eyes that would be worse than torturing one person for fifty years. I have to assume my utility function over other people is asymptotic; the amount of disutility of choosing to let even an infinity of people get dust specks in their eyes is still less than the disutility of one person getting tortured for fifty years.
I’m not sure what you mean by this. I don’t believe in linearity of suffering: that would be the claim that 2 people tortured for 1 day is the same as 1 person tortured for 2 days, and that’s ridiculous.
I think he’s questioning the idea that two people getting dust specks in their eyes is twice the disutility of one person getting dust specks, and that is the linearity he’s referring to.
Personally, I think the problem stems from dust specks being such a minor inconvenience that it’s basically below the noise threshold. I’d almost be indifferent between choosing for nothing to happen or choosing for everyone on Earth to get dust specks (assuming they don’t cause crashes or anything).
There’s the question of linearity- but if you use big enough numbers you can brute force any nonlinear relationship, as Yudkowsky correctly pointed out some years ago. Take Kindly’s statement:
“There is some pair (N,T) such that (N people tortured for T seconds) is worse than (10^100 N people tortured for T-1 seconds), but I don’t know the exact values of N and T”
We can imagine a world where this statement is true (probably for a value of T really close to 1). And we can imagine knowing the correct values of N and T in that world. But even then, if a critical condition is met, it will be true that
“For all values of N, and for all T>1, there exists a value of A such that torturing N people for T seconds is better than torturing A*N people for T-1 seconds.”
Sure, the value of A may be larger than 10^100… But then, 3^^^3 is already vastly larger than 10^100. And if it weren’t big enough we could just throw a bigger number at the problem; there is no upper bound on the size of conceivable real numbers. So if we grant the critical condition in question, as Yudkowsky does/did in the original post…
Well, you basically have to concede that “torture” wins the argument, because even if you say that [hugenumber] of dust specks does not equate to a half-century of torture, that is NOT you winning the argument. That is just you trying to bid up the price of half a century of torture.
The critical condition that must be met here is simple, and is an underlying assumption of Yudkowsky’s original post: All forms of suffering and inconvenience are represented by some real number quantity, with commensurate units to all other forms of suffering and inconvenience.
In other words, the “torture one person rather than allow 3^^^3 dust specks” wins, quite predictably, if and only if it is true that that the ‘pain’ component of the utility function is measured in one and only one dimension.
So the question is, basically, do you measure your utility function in terms of a single input variable?
If you do, then either you bury your head in the sand and develop a severe case of scope insensitivity… or you conclude that there has to be some number of dust specks worse than a single lifetime of torture.
If you don’t, it raises a large complex of additional questions- but so far as I know, there may well be space to construct coherent, rational systems of ethics in that realm of ideas.
It occurred to me to add something to my previous comments about the idea of harm being nonlinear, or something that we compute in multiple dimensions that are not commensurate.
One is that any deontological system of ethics automatically has at least two dimensions. One for general-purpose “utilons,” and one for… call them “red flags.” As soon as you accumulate even one red flag you are doing something capital-w Wrong in that system of ethics, regardless of the number of utilons you’ve accumulated.
The main argument justifying this is, of course, that you may think you have found a clever way to accumulate 3^^^3 utilons in exchange for a trivial amount of harm (torture ONLY one scapegoat!)… but the overall weighted average of all human moral reasoning suggests that people who think they’ve done this are usually wrong. Therefore, best to red-flag such methods, because they usually only sound clever.
Obviously, one may need to take this argument with a grain of salt, or 3^^^3 grains of salt. It depends on how strongly you feel bound to honor conclusions drawn by looking at the weighted average of past human decision-making.
The other observation that occurred to me is unrelated. It is about the idea of harm being nonlinear, which as I noted above is just plain not enough to invalidate the torture/specks argument by itself due to the ability to keep thwacking a nonlinear relationship with bigger numbers until it collapses.
Take as a thought-experiment an alternate Earth where, in the year 1000, population growth has stabilized at an equilibrium level, and will rise back to that equilibrium level in response to sudden population decrease. The equilibrium level is assumed to be stable in and of itself.
Imagine aliens arriving and killing 50% of all humans, chosen apparently at random. Then they wait until the population has returned to equilibrium (say, 150 years) and do it again. Then they repeat the process twice more.
The world population circa 1000 was about 300 million (roughly,) so we estimate that this process would kill 600 million people.
Now consider as an alternative, said aliens simply killing everyone, all at once. 300 million dead.
Which outcome is worse?
If harm is strictly linear, we would expect that one death plus one death is exactly as bad as two deaths. By the same logic, 300 megadeaths is only half as bad as 600 megadeaths, and if we inoculate ourselves against hyperbolic discounting...
Well, the “linear harm” theory smacks into a wall. Because it is very credible to claim that the extinction of the human species is much worse than merely twice as bad as the extinction of exactly half the human species. Many arguments can be presented, and no doubt have been presented on this very site. The first that comes to mind is that human extinction means the loss of all potential future value associated with humans, not just the loss of present value, or even the loss of some portion of the potential future.
We are forced to conclude that there is a “total extinction” term in our calculation of harm, one that rises very rapidly in an ‘inflationary’ way. And it would do this as the destruction wrought upon humanity reaches and passes a level beyond which the species could not recover- the aliens killing all humans except one is not noticeably better than killing all of them, nor is sparing any population less than a complete breeding population, but once a breeding population is spared, there is a fairly sudden drop in the total quantity of harm.
Now, again, in itself this does not strictly invalidate the Torture/Specks argument. Assuming that the harm associated with human extinction (or torturing one person) is any finite amount that could conceivably be equalled by adding up a finite number of specks in eyes, then by definition there is some “big enough” number of specks that the aliens would rationally decide to wipe out humanity rather than accept that many specks in that many eyes.
But I can’t recall a similar argument for nonlinear harm measurement being presented in any of the comments I’ve sampled, so I wanted to mention it.
But I thought it was interesting and couldn’t recall seeing it elsewhere.
I mentioned duplication. That in 3^^^3 people, most have to be exact duplicates of one another birth to death.
In your extinction example, once you have substantially more than the breeding population, extra people duplicate some aspects of your population (ability to breed) which causes you to find it less bad.
The other observation that occurred to me is unrelated. It is about the idea of harm being nonlinear, which as I noted above is just plain not enough to invalidate the torture/specks argument by itself due to the ability to keep thwacking a nonlinear relationship with bigger numbers until it collapses.
Not every non-linear relationship can be thwacked with bigger and bigger numbers...
For one thing N=1 T=1 trivially satisfies your condition…
I’m not sure what you mean by this.
I mean, suppose that you got yourself a function that takes in a description of what’s going on in a region of spacetime, and it spits out a real number of how bad it is.
Now, that function can do all sorts of perfectly reasonable things that could make it asymptotic for large numbers of people, for example it could be counting distinct subjective experiences in there (otherwise a mind upload on very multiple redundant hardware is an utility monster, despite having an identical subjective experience to same upload running one time. That’s much sillier than the usual utility monster, which feels much stronger feelings). This would impose a finite limit (for brains of finite complexity).
One thing that function can’t do, is to have a general property that f(a union b)=f(a)+f(b) , because then we just subdivide our space into individual atoms none of which are feeling anything.
Agree, having lived in chronic pain supposedly worse than untrained childbirth, I’d say that even an hour has a really seriously different possibility in terms of capacity for suffering than a day, and a day different from a week. For me it breaks down somewhere, even when multiplying between the 10^15 for 1 day and 10^21 for one minute. You can’t really feel THAT much pain in a minute that is comparable to a day, even orders of magnitude? Its just qualitatively different. Interested to hear pushback on this
We could go from a day to a minute more slowly; for example, by increasing the number of people by a factor of a googolplex every time the torture time decreases by 1 second.
I absolutely agree that the length of torture increases how bad it is in nonlinear ways, but this doesn’t mean we can’t find exponential factors that dominate it at every point at least along the “less than 50 years” range.
Obviously. Just important to remember that extremity of suffering is something we frequently fail to think well about.
Absolutely. We’re bad at anything that we can’t easily imagine. Probably, for many people, intuition for “torture vs. dust specks” imagines a guy with a broken arm on one side, and a hundred people saying ‘ow’ on the other.
The consequences of our poor imagination for large numbers of people (i.e. scope insensitivity) are well-studied. We have trouble doing charity effectively because our intuition doesn’t take the number of people saved by an intervention into account; we just picture the typical effect on a single person.
What, I wonder, are the consequence of our poor imagination for extremity of suffering? For me, the prison system comes to mind: I don’t know how bad being in prison is, but it probably becomes much worse than I imagine if you’re there for 50 years, and we don’t think about that at all when arguing (or voting) about prison sentences.
My heuristic for dealing with such situations is somewhat reminiscent of Hofstadter’s Law: however bad you imagine it to be, it’s worse than that, even when you take the preceding statement into account. In principle, this recursion should go on forever and lead to you regarding any sufficiently unimaginably bad situation as infinitely bad, but in practice, I’ve yet to have it overflow, probably because your judgment spontaneously regresses back to your original (inaccurate) representation of the situation unless consciously corrected for.
Obligatory xkcd.
That would have been a better comic without the commentary in the last panel.
But the alt text is great X-)
My feeling is that situations like being caught for doing something horrendous might or might not be subject to psychological adjustment—that many situations of suffering are subject to psychological adjustment and so might actually be not as bad as we though. But chronic intense pain, is literally unadjustable to some degree—you can adjust to being in intense suffering but that doesn’t make the intense suffering go away. That’s why I think its a special class of states of being—one that invokes action. What do people think?
That strikes me as a deliberate set up for a continuum fallacy.
Also, why are you so sure that the number of people increases suffering in a linear way for even very large numbers? What is a number of people anyway?
I’d much prefer to have a [large number of exact copies of me] experience 1 second of headache than for one me to suffer it for a whole day. Because those copies they don’t have any mechanism which could compound their suffering. They aren’t even different subjectivities. I don’t see any reason why a hypothetical mind upload of me running on multiple redundant hardware should be an utility monster, if it can’t even tell subjectively how redundant it’s hardware is.
Some anaesthetics do something similar, preventing any new long term memories, people have no problem with taking those for surgery. Something’s still experiencing pain but it’s not compounding into anything really bad (unless the drugs fail to work, or unless some form of long term memory still works). A real example of a very strong preference for N independent experiences of 30 seconds of pain over 1 experience of 30*N seconds of pain.
It’s not a continuum fallacy because I would accept “There is some pair (N,T) such that (N people tortured for T seconds) is worse than (10^100 N people tortured for T-1 seconds), but I don’t know the exact values of N and T” as an answer. If, on the other hand, the comparison goes the other way for any values of N and T, then you have to accept the transitive closure of those comparisons as well.
I’m not sure what you mean by this. I don’t believe in linearity of suffering: that would be the claim that 2 people tortured for 1 day is the same as 1 person tortured for 2 days, and that’s ridiculous. I believe in comparability of suffering, which is the claim that for some value of N, N people tortured for 1 day is worse than 1 person tortured for 2 days.
Regarding anaesthetics: I would prefer a memory inhibitor for a painful surgery to the absence of one, but I would still strongly prefer to feel less pain during the surgery even if I know I will not remember it one way or the other. Is this preference unusual?
This is where the argument for choosing torture falls apart for me, really. I don’t think there is any number of people getting dust specks in their eyes that would be worse than torturing one person for fifty years. I have to assume my utility function over other people is asymptotic; the amount of disutility of choosing to let even an infinity of people get dust specks in their eyes is still less than the disutility of one person getting tortured for fifty years.
I think he’s questioning the idea that two people getting dust specks in their eyes is twice the disutility of one person getting dust specks, and that is the linearity he’s referring to.
Personally, I think the problem stems from dust specks being such a minor inconvenience that it’s basically below the noise threshold. I’d almost be indifferent between choosing for nothing to happen or choosing for everyone on Earth to get dust specks (assuming they don’t cause crashes or anything).
There’s the question of linearity- but if you use big enough numbers you can brute force any nonlinear relationship, as Yudkowsky correctly pointed out some years ago. Take Kindly’s statement:
“There is some pair (N,T) such that (N people tortured for T seconds) is worse than (10^100 N people tortured for T-1 seconds), but I don’t know the exact values of N and T”
We can imagine a world where this statement is true (probably for a value of T really close to 1). And we can imagine knowing the correct values of N and T in that world. But even then, if a critical condition is met, it will be true that
“For all values of N, and for all T>1, there exists a value of A such that torturing N people for T seconds is better than torturing A*N people for T-1 seconds.”
Sure, the value of A may be larger than 10^100… But then, 3^^^3 is already vastly larger than 10^100. And if it weren’t big enough we could just throw a bigger number at the problem; there is no upper bound on the size of conceivable real numbers. So if we grant the critical condition in question, as Yudkowsky does/did in the original post…
Well, you basically have to concede that “torture” wins the argument, because even if you say that [hugenumber] of dust specks does not equate to a half-century of torture, that is NOT you winning the argument. That is just you trying to bid up the price of half a century of torture.
The critical condition that must be met here is simple, and is an underlying assumption of Yudkowsky’s original post: All forms of suffering and inconvenience are represented by some real number quantity, with commensurate units to all other forms of suffering and inconvenience.
In other words, the “torture one person rather than allow 3^^^3 dust specks” wins, quite predictably, if and only if it is true that that the ‘pain’ component of the utility function is measured in one and only one dimension.
So the question is, basically, do you measure your utility function in terms of a single input variable?
If you do, then either you bury your head in the sand and develop a severe case of scope insensitivity… or you conclude that there has to be some number of dust specks worse than a single lifetime of torture.
If you don’t, it raises a large complex of additional questions- but so far as I know, there may well be space to construct coherent, rational systems of ethics in that realm of ideas.
It occurred to me to add something to my previous comments about the idea of harm being nonlinear, or something that we compute in multiple dimensions that are not commensurate.
One is that any deontological system of ethics automatically has at least two dimensions. One for general-purpose “utilons,” and one for… call them “red flags.” As soon as you accumulate even one red flag you are doing something capital-w Wrong in that system of ethics, regardless of the number of utilons you’ve accumulated.
The main argument justifying this is, of course, that you may think you have found a clever way to accumulate 3^^^3 utilons in exchange for a trivial amount of harm (torture ONLY one scapegoat!)… but the overall weighted average of all human moral reasoning suggests that people who think they’ve done this are usually wrong. Therefore, best to red-flag such methods, because they usually only sound clever.
Obviously, one may need to take this argument with a grain of salt, or 3^^^3 grains of salt. It depends on how strongly you feel bound to honor conclusions drawn by looking at the weighted average of past human decision-making.
The other observation that occurred to me is unrelated. It is about the idea of harm being nonlinear, which as I noted above is just plain not enough to invalidate the torture/specks argument by itself due to the ability to keep thwacking a nonlinear relationship with bigger numbers until it collapses.
Take as a thought-experiment an alternate Earth where, in the year 1000, population growth has stabilized at an equilibrium level, and will rise back to that equilibrium level in response to sudden population decrease. The equilibrium level is assumed to be stable in and of itself.
Imagine aliens arriving and killing 50% of all humans, chosen apparently at random. Then they wait until the population has returned to equilibrium (say, 150 years) and do it again. Then they repeat the process twice more.
The world population circa 1000 was about 300 million (roughly,) so we estimate that this process would kill 600 million people.
Now consider as an alternative, said aliens simply killing everyone, all at once. 300 million dead.
Which outcome is worse?
If harm is strictly linear, we would expect that one death plus one death is exactly as bad as two deaths. By the same logic, 300 megadeaths is only half as bad as 600 megadeaths, and if we inoculate ourselves against hyperbolic discounting...
Well, the “linear harm” theory smacks into a wall. Because it is very credible to claim that the extinction of the human species is much worse than merely twice as bad as the extinction of exactly half the human species. Many arguments can be presented, and no doubt have been presented on this very site. The first that comes to mind is that human extinction means the loss of all potential future value associated with humans, not just the loss of present value, or even the loss of some portion of the potential future.
We are forced to conclude that there is a “total extinction” term in our calculation of harm, one that rises very rapidly in an ‘inflationary’ way. And it would do this as the destruction wrought upon humanity reaches and passes a level beyond which the species could not recover- the aliens killing all humans except one is not noticeably better than killing all of them, nor is sparing any population less than a complete breeding population, but once a breeding population is spared, there is a fairly sudden drop in the total quantity of harm.
Now, again, in itself this does not strictly invalidate the Torture/Specks argument. Assuming that the harm associated with human extinction (or torturing one person) is any finite amount that could conceivably be equalled by adding up a finite number of specks in eyes, then by definition there is some “big enough” number of specks that the aliens would rationally decide to wipe out humanity rather than accept that many specks in that many eyes.
But I can’t recall a similar argument for nonlinear harm measurement being presented in any of the comments I’ve sampled, so I wanted to mention it.
But I thought it was interesting and couldn’t recall seeing it elsewhere.
I mentioned duplication. That in 3^^^3 people, most have to be exact duplicates of one another birth to death.
In your extinction example, once you have substantially more than the breeding population, extra people duplicate some aspects of your population (ability to breed) which causes you to find it less bad.
Not every non-linear relationship can be thwacked with bigger and bigger numbers...
For one thing N=1 T=1 trivially satisfies your condition…
I mean, suppose that you got yourself a function that takes in a description of what’s going on in a region of spacetime, and it spits out a real number of how bad it is.
Now, that function can do all sorts of perfectly reasonable things that could make it asymptotic for large numbers of people, for example it could be counting distinct subjective experiences in there (otherwise a mind upload on very multiple redundant hardware is an utility monster, despite having an identical subjective experience to same upload running one time. That’s much sillier than the usual utility monster, which feels much stronger feelings). This would impose a finite limit (for brains of finite complexity).
One thing that function can’t do, is to have a general property that f(a union b)=f(a)+f(b) , because then we just subdivide our space into individual atoms none of which are feeling anything.
Obviously I only meant to consider values of T and N that actually occur in the argument we were both talking about.
Well I’m not sure what’s the point then. What you’re trying to induct from it.