Having disagreed with Zack many times in the past, it is a pleasure to say: I think this is absolutely right (except that I think I’d replace “pleasure and pain” with “something pleasure-like and something pain-like”); that bit of “Value is Fragile” is surely wrong, and the intuitions that drove the relevant bits of “Three Worlds Collide” are more reflective of how actual human value systems work.
I think I’d want to distinguish two related but separate issues here. (1) Should we expect that (some) other intelligent agents are things whose welfare we value? (Whether they are might depend on whether we think they have internal mechanisms that resemble our mechanisms of pleasure, pain, hope, fear, etc.) (2) Should we expect that (some) other intelligent agents share some of our values? (Whether they do would depend on how far the structure of their thinking has converged with ours.) If there are other intelligent species out there, then whether they’re “animal-like organisms that feel pleasure and pain” addresses #1 and whether “the idealized values of those organisms are not a random utility function” addresses #2.
(Of course, how much we care about their welfare may depend on how much we think they share our values, for internalized-game-theory-ish reasons. And presumably they’re likely to share more of our values if their motivational systems work similarly to ours. So the issues are not only related but interdependent.)
Suppose that (evolved/uplifted/otherwise-advanced-enough-for-sapience) octopuses share some of our values. Now suppose that humans go extinct, and these Octopus sapiens create an advanced civilization, whose products instantiate some values we would recognize, like art, music, science, etc.
Does this future contain anything of value? I say it does not, because there are no humans around to value it. There are octopuses, and that’s great for the octopuses, but as far as human values go, this future ended with humanity’s extinction. Whatever happens afterwards is irrelevant.
EDIT: Mind you—this is not quite the point Eliezer was making, I don’t think; I am responding to gjm’s comment, here. This comment should not necessarily be taken to constitute part of a defense of the point made in the OP (and quoted by Zack upthread).
When I consider this possible universe, I find that I do attach some value to the welfare of these sapient octopuses, and I do consider that it’s a universe that contains plenty of value. (It depends somewhat on whether they have, as well as values resembling ours, something I can recognize as welfare; see my last couple of paragraphs above.) If there were a magic switch I could control, where one setting is “humans go extinct, no other advanced civilization ever exists” and the other is “humans go extinct, the sapient octopus civilization arises”, I would definitely put it on the second setting, and if sufficiently convinced that the switch would really do what it says then I think I would pay a nonzero amount, or put up with nonzero effort or inconvenience, to put it there.
Of course my values are mine and your values are yours, and if we disagree there may be no way for either of us to persuade the other. But I’ll at least try to explain why I feel the way I do. (So far as I can; introspection is difficulty and unreliable.)
First, consider two possible futures. 1: Humanity continues for millions of years, substantially unchanged from how we are now. (I take it we agree that in this case the future universe contains much of value.) 2: Humanity continues for millions of years, gradually evolving (in the Darwinian sense or otherwise) but always somewhat resembling us, and always retaining something like our values. It seems to me that here, too, the future universe contains much of value.
The sapient octopuses, I am taking it, do somewhat resemble us and have something like our values. Perhaps as much so as our descendants in possible future 2. So why should I care much less about them? I can see only one plausible reason: because our descendants are, in fact, our descendants: they are biologically related to us. How plausible is that reason?
Possible future 3: at some point in that future history of humanity, our descendants decide to upload themselves into computers and continue their lives virtually. Possible future 4: at some point in that virtual existence they decide they’d like to be embodied again, and arrange for it to happen. Their new bodies are enough like original-human bodies for them to feel at home in them, but they use some freshly-invented genetic material rather than DNA, and many of the internal organs are differently designed.
I don’t find that the loss of biological continuity in these possible futures makes me not care about the welfare of our kinda-sorta-descendants there. I don’t see any reason why it should, either. So if I should care much less about the octopuses, what matters must be some more generalized sort of continuity: the future-kinda-humans are our “causal descendants” or something, even if not our biological descendants.
At that point I think I stop; I can see how someone might find that relationship super-important, and care about “causal descendants” but not about other beings, physically and mentally indistinguishable, who happen not to be our “causal descendants”; but I don’t myself feel much inclination to see that as super-important, and I don’t see any plausible way to change anyone’s mind on the matter by argument.
One can construct all sorts of hypothetical scenarios, but I am far from convinced of their usefulness in teasing out our “true” values (as contrasted with “confabulating some plausible-sounding, but not reflectively stable, set of values”). That said, it seems to me that how much I value (and should value) any given future depends on the degree of that future’s resemblance to my current values. So, to take the examples:
1: Humanity continues for millions of years, substantially unchanged from how we are now. (I take it we agree that in this case the future universe contains much of value.)
Indeed, we agree.
2: Humanity continues for millions of years, gradually evolving (in the Darwinian sense or otherwise) but always somewhat resembling us, and always retaining something like our values. It seems to me that here, too, the future universe contains much of value.
Well, it depends: it seems to me that the further from my current values this future humanity drifts, the less I value this future.
Crucially, it seems to me that the degree of difference (at any given future time period) will depend (and how can it not?) on the starting point. Start with current humans, and you get one degree of resemblance; start with octopuses, on the other hand…
Possible future 3: at some point in that future history of humanity, our descendants decide to upload themselves into computers and continue their lives virtually.
I would not like for this to happen, personally. I value this future substantially less, thereby.
Possible future 4: at some point in that virtual existence they decide they’d like to be embodied again, and arrange for it to happen. Their new bodies are enough like original-human bodies for them to feel at home in them, but they use some freshly-invented genetic material rather than DNA, and many of the internal organs are differently designed.
The impact of this biological re-invention on how valuable the future is, will depend on what impact it has on observable and experiential traits of this new humanity—I care about the interface, so to speak, not the implementation details. (After all, suppose that, while I slept, you replaced my liver, kidneys, pancreas, and some other internal organs with a different set of organs—which, however, performed all the same functions, allowing me to continue living my life as before. I do not see what difference this would make to… well, almost anything, really. Perhaps I couldn’t even tell that this had been done! Would this matter in any moral calculus? I think not…)
So if I should care much less about the octopuses, what matters must be some more generalized sort of continuity: the future-kinda-humans are our “causal descendants” or something, even if not our biological descendants.
Causal descendancy is something, certainly; but, again, for me it is a question of degree of resemblance. Perhaps another way of putting it is: could I inhabit this future? Would I, personally, find it… fun? Would I, living inside it, consider it to be awesome, amazing, wonderful? Or would I find it to be alien and bizarre? It is all well and good to “expect weirdtopia”, but there is no law of morality that says I have to want weirdtopia…
Having disagreed with Zack many times in the past, it is a pleasure to say: I think this is absolutely right (except that I think I’d replace “pleasure and pain” with “something pleasure-like and something pain-like”); that bit of “Value is Fragile” is surely wrong, and the intuitions that drove the relevant bits of “Three Worlds Collide” are more reflective of how actual human value systems work.
I think I’d want to distinguish two related but separate issues here. (1) Should we expect that (some) other intelligent agents are things whose welfare we value? (Whether they are might depend on whether we think they have internal mechanisms that resemble our mechanisms of pleasure, pain, hope, fear, etc.) (2) Should we expect that (some) other intelligent agents share some of our values? (Whether they do would depend on how far the structure of their thinking has converged with ours.) If there are other intelligent species out there, then whether they’re “animal-like organisms that feel pleasure and pain” addresses #1 and whether “the idealized values of those organisms are not a random utility function” addresses #2.
(Of course, how much we care about their welfare may depend on how much we think they share our values, for internalized-game-theory-ish reasons. And presumably they’re likely to share more of our values if their motivational systems work similarly to ours. So the issues are not only related but interdependent.)
Suppose that (evolved/uplifted/otherwise-advanced-enough-for-sapience) octopuses share some of our values. Now suppose that humans go extinct, and these Octopus sapiens create an advanced civilization, whose products instantiate some values we would recognize, like art, music, science, etc.
Does this future contain anything of value? I say it does not, because there are no humans around to value it. There are octopuses, and that’s great for the octopuses, but as far as human values go, this future ended with humanity’s extinction. Whatever happens afterwards is irrelevant.
EDIT: Mind you—this is not quite the point Eliezer was making, I don’t think; I am responding to gjm’s comment, here. This comment should not necessarily be taken to constitute part of a defense of the point made in the OP (and quoted by Zack upthread).
When I consider this possible universe, I find that I do attach some value to the welfare of these sapient octopuses, and I do consider that it’s a universe that contains plenty of value. (It depends somewhat on whether they have, as well as values resembling ours, something I can recognize as welfare; see my last couple of paragraphs above.) If there were a magic switch I could control, where one setting is “humans go extinct, no other advanced civilization ever exists” and the other is “humans go extinct, the sapient octopus civilization arises”, I would definitely put it on the second setting, and if sufficiently convinced that the switch would really do what it says then I think I would pay a nonzero amount, or put up with nonzero effort or inconvenience, to put it there.
Of course my values are mine and your values are yours, and if we disagree there may be no way for either of us to persuade the other. But I’ll at least try to explain why I feel the way I do. (So far as I can; introspection is difficulty and unreliable.)
First, consider two possible futures. 1: Humanity continues for millions of years, substantially unchanged from how we are now. (I take it we agree that in this case the future universe contains much of value.) 2: Humanity continues for millions of years, gradually evolving (in the Darwinian sense or otherwise) but always somewhat resembling us, and always retaining something like our values. It seems to me that here, too, the future universe contains much of value.
The sapient octopuses, I am taking it, do somewhat resemble us and have something like our values. Perhaps as much so as our descendants in possible future 2. So why should I care much less about them? I can see only one plausible reason: because our descendants are, in fact, our descendants: they are biologically related to us. How plausible is that reason?
Possible future 3: at some point in that future history of humanity, our descendants decide to upload themselves into computers and continue their lives virtually. Possible future 4: at some point in that virtual existence they decide they’d like to be embodied again, and arrange for it to happen. Their new bodies are enough like original-human bodies for them to feel at home in them, but they use some freshly-invented genetic material rather than DNA, and many of the internal organs are differently designed.
I don’t find that the loss of biological continuity in these possible futures makes me not care about the welfare of our kinda-sorta-descendants there. I don’t see any reason why it should, either. So if I should care much less about the octopuses, what matters must be some more generalized sort of continuity: the future-kinda-humans are our “causal descendants” or something, even if not our biological descendants.
At that point I think I stop; I can see how someone might find that relationship super-important, and care about “causal descendants” but not about other beings, physically and mentally indistinguishable, who happen not to be our “causal descendants”; but I don’t myself feel much inclination to see that as super-important, and I don’t see any plausible way to change anyone’s mind on the matter by argument.
One can construct all sorts of hypothetical scenarios, but I am far from convinced of their usefulness in teasing out our “true” values (as contrasted with “confabulating some plausible-sounding, but not reflectively stable, set of values”). That said, it seems to me that how much I value (and should value) any given future depends on the degree of that future’s resemblance to my current values. So, to take the examples:
Indeed, we agree.
Well, it depends: it seems to me that the further from my current values this future humanity drifts, the less I value this future.
Crucially, it seems to me that the degree of difference (at any given future time period) will depend (and how can it not?) on the starting point. Start with current humans, and you get one degree of resemblance; start with octopuses, on the other hand…
I would not like for this to happen, personally. I value this future substantially less, thereby.
The impact of this biological re-invention on how valuable the future is, will depend on what impact it has on observable and experiential traits of this new humanity—I care about the interface, so to speak, not the implementation details. (After all, suppose that, while I slept, you replaced my liver, kidneys, pancreas, and some other internal organs with a different set of organs—which, however, performed all the same functions, allowing me to continue living my life as before. I do not see what difference this would make to… well, almost anything, really. Perhaps I couldn’t even tell that this had been done! Would this matter in any moral calculus? I think not…)
Causal descendancy is something, certainly; but, again, for me it is a question of degree of resemblance. Perhaps another way of putting it is: could I inhabit this future? Would I, personally, find it… fun? Would I, living inside it, consider it to be awesome, amazing, wonderful? Or would I find it to be alien and bizarre? It is all well and good to “expect weirdtopia”, but there is no law of morality that says I have to want weirdtopia…