Another thing to worry about with CEV is that the nonperson predicates that whoever writes it decides on will cover things that you consider people, or would not like to see be destroyed at the end of an instrumental simulation.
Humans probably have no built-in intuitions for the details of distinction of things that deserve ethical consideration at the precision required for a nonperson predicate that can flag things as nonpersons that will be useful for instrumental simulations, and yet not flag a fully-detailed simulation of you or me as a nonperson. We don’t have detailed enough introspection to know what “sentience” (whatever that means) is at a mechanical level. How can we care about the arrangement of parts that make “sentience,” when we don’t know what that arrangement is?
I think the process by which some people come to care about animals and others do not is probably highly dependent on which thought experiments they considered in which order, which label they first used for the category in their mind of “things that shouldn’t be hurt.”
The most memorable occasion when my person predicate changed was when I used to think that people could only exist in a basement universe. Simulations were automatically nonpersons. I thought to myself “if they aren’t real I don’t care.” What changed my mind was the thought “If you ran a simulated version of me, and informed it that it was in a simulation, would it stop simulatedly caring about itself?” (The answer was no). But what if I had read LessWrong first, and become accustomed to thinking of myself as an insane (objectively speaking, not by human standards) biased ape, and said “No, but that’s because I’m only human and sometimes have feelings that are contrary to my true ideal utility function. The simulated version may not alieve that he was not real, but he really wouldn’t be real, so he Ideal_Mestroyer::should stop caring about himself.” That thought isn’t factually incorrect. If I had thought it back then, I might still care about “realness” in the same sense. But thinking about it now, it is too late, my terminal values have already changed, perhaps because of a misstep in my reasoning back then, and I am glad they have. But maybe the introduction of “real” (being directly made of physics and not in a simulation) as an important factor was originally based on mistaken reasoning too.
I think most of the features of our nonperson predicates are decided in the same way, partly randomly, based on reasoning mistakes and thought experiments considered first, (more randomly the more philosophical the person is), and partly through absorption from family and peers, which means it doesn’t make sense for there to be a coherent extrapolated nonperson predicate for humanity (though you can still superpose a bunch of different ones).
Even if you don’t really care about animals, your “person” category (or just “I care about this being” category) might be broader than SIAI’s, and if it is, you should be afraid that vast numbers of people will be killed by terminating instrumental simulations.
Even so, if your person predicate is part of the CEV of humanity, perhaps an FAI could self-modify (after running some number of simulations using the old one that wasn’t really that big compared to the number of people that would exist in a post-friendly-foom world)
So those people might not be that important to you, compared to what else is at stake. But if your nonperson predicate is not in humanity’s CEV, and is uncommon enough that it’s not worth it to humanity to accommodate you, and you disvalue death (and not just suffering) CEV might cause you to spend billions of years screaming.
Interesting story. Yes, I think our intuitions about what kinds of computations we want to care about are easily bent and twisted depending on the situation at hand. In analogy with Dennett’s “intentional stance,” humans have a “compassionate stance” that we apply to some physical operations and don’t apply to others. It’s not too hard to manipulate these intuitions by thought experiments. So, yes, I do fear that other people may differ (perhaps quite a bit) in their views about what kinds of computations are suffering that we should avoid.
Another thing to worry about with CEV is that the nonperson predicates that whoever writes it decides on will cover things that you consider people, or would not like to see be destroyed at the end of an instrumental simulation.
Humans probably have no built-in intuitions for the details of distinction of things that deserve ethical consideration at the precision required for a nonperson predicate that can flag things as nonpersons that will be useful for instrumental simulations, and yet not flag a fully-detailed simulation of you or me as a nonperson. We don’t have detailed enough introspection to know what “sentience” (whatever that means) is at a mechanical level. How can we care about the arrangement of parts that make “sentience,” when we don’t know what that arrangement is?
I think the process by which some people come to care about animals and others do not is probably highly dependent on which thought experiments they considered in which order, which label they first used for the category in their mind of “things that shouldn’t be hurt.”
The most memorable occasion when my person predicate changed was when I used to think that people could only exist in a basement universe. Simulations were automatically nonpersons. I thought to myself “if they aren’t real I don’t care.” What changed my mind was the thought “If you ran a simulated version of me, and informed it that it was in a simulation, would it stop simulatedly caring about itself?” (The answer was no). But what if I had read LessWrong first, and become accustomed to thinking of myself as an insane (objectively speaking, not by human standards) biased ape, and said “No, but that’s because I’m only human and sometimes have feelings that are contrary to my true ideal utility function. The simulated version may not alieve that he was not real, but he really wouldn’t be real, so he Ideal_Mestroyer::should stop caring about himself.” That thought isn’t factually incorrect. If I had thought it back then, I might still care about “realness” in the same sense. But thinking about it now, it is too late, my terminal values have already changed, perhaps because of a misstep in my reasoning back then, and I am glad they have. But maybe the introduction of “real” (being directly made of physics and not in a simulation) as an important factor was originally based on mistaken reasoning too.
I think most of the features of our nonperson predicates are decided in the same way, partly randomly, based on reasoning mistakes and thought experiments considered first, (more randomly the more philosophical the person is), and partly through absorption from family and peers, which means it doesn’t make sense for there to be a coherent extrapolated nonperson predicate for humanity (though you can still superpose a bunch of different ones).
Even if you don’t really care about animals, your “person” category (or just “I care about this being” category) might be broader than SIAI’s, and if it is, you should be afraid that vast numbers of people will be killed by terminating instrumental simulations.
Even so, if your person predicate is part of the CEV of humanity, perhaps an FAI could self-modify (after running some number of simulations using the old one that wasn’t really that big compared to the number of people that would exist in a post-friendly-foom world)
So those people might not be that important to you, compared to what else is at stake. But if your nonperson predicate is not in humanity’s CEV, and is uncommon enough that it’s not worth it to humanity to accommodate you, and you disvalue death (and not just suffering) CEV might cause you to spend billions of years screaming.
Interesting story. Yes, I think our intuitions about what kinds of computations we want to care about are easily bent and twisted depending on the situation at hand. In analogy with Dennett’s “intentional stance,” humans have a “compassionate stance” that we apply to some physical operations and don’t apply to others. It’s not too hard to manipulate these intuitions by thought experiments. So, yes, I do fear that other people may differ (perhaps quite a bit) in their views about what kinds of computations are suffering that we should avoid.