To me it seems like destroying a bunch of sentient minds and then recreating an equivalent number later
Destroying, minus all of the experiential bad things we mentally associate with “destroying”!
I mean, why wouldn’t it? Weird differences at that low a level having profound differences on something high-level like utility functions feels normal to me?
I don’t object to the idea of people having utility functions over various unconscious physical states—e.g., maybe I assign some aesthetic value to having my upload run on cool shiny-looking computronium, rather than something tacky, even if no one’s ever going to see my hardware.
The three things that make me suspicious of your “this is a huge part of my utility function” claim are:
The fact that the properties you’re attached to are things that (for all you know) you never had in the first place. The idea of humans experiencing and enjoying beauty, and then desiring that things be beautiful even in observed parts of the universe, makes total sense to me. The idea of humans coming up with a weird metaphysical hypothesis, and assigning ultimate value to the answer to this hypothesis even though its truth or falsehood has never once made a perceptible difference in anyone’s life, seems far weirder to me.
This alleged utility function component happens to coincide with a powerful and widespread cognitive illusion (the idea of a Soul or Self or Homunculus existing outside determinism / physics / the Cartesian theater).
The alleged importance of this component raises major alarm bells. I wouldn’t be skeptical if you said that you slightly aesthetically prefer bodily continuity, all else equal; but treating this metaphysical stuff as though it were a matter of life or death, or as though it were crucial in the way that the distinction between “me” and “a random stranger” is important, seems like a flag that there’s something wrong here. The “it’s a matter of life or death” thing only makes sense, AFAICT, if you’re subject to a soul-style illusion about what “you” are, and about what your experience of personal continuity currently consists in.
You see this person, you interact with him (albeit in weird, perfectly symmetrical ways), would you have no preference for your life and happiness over his?
To my eye, this is isomorphic to the following hypothetical (modulo the fact that two copies of me existing may not have the same moral value as one copy of me existing):
I’m playing a VR game.
Iff a VR avatar of mine is killed, I die in real life.
When I use the controls, it causes two different avatars to move. They move in exactly the same way, and since they’re symmetrically positioned in a symmetrical room, there’s no difference between whether I’m seeing through the eyes of one avatar or the other. Through my VR helmet, I am in fact looking through the eyes of just one of the two avatars—I can’t see both of their experiences at once—but since the experiences are identical, it doesn’t matter which one.
It doesn’t really matter whether I value “my” life over “his”, because we’re going to have the same experiences going forward regardless. But if something breaks this symmetry (note that at that point we’d need to clarify that there are two separate streams of consciousness here, not just one), my preference before the symmetry is broken should be indifferent between which of the copies gets the reward, because both versions of me bear the correct memory-relationship to my present self.
(So, I should “anticipate” being both copies of me, for the same reason I should “anticipate” being all of my MWI selves, and anticipate having new experiences after stepping through a teleporter.)
This alleged utility function component happens to coincide with a powerful and widespread cognitive illusion
So what? Ones you know that the illusion is wrong, rescuing your utility function even to “the universe is now sad forever” is perfectly fine. So in what way it doesn’t make sense to value some approximation of soul that is more problematic than patternist one? There is no law that says you must use the same reasoning for ethically navigating ontology shifts, that you use for figuring the world out.
So in what way it doesn’t make sense to value some approximation of soul that is more problematic than patternist one?
Because the reason why people make such an approximation is because they wish to save the continuity of their consciousness. But there is nothing else than patternism that does that.
They don’t terminally value being made of the same substance, or having the same body, etc. They terminally value their consciousness persisting, and the common decision to care about, let’s say, preserving the substance of their body, is made instrumentally to protect that.
Once they someday understand that their consciousness persists independently of that, they will stop valuing it.
Persistence of your consciousness is not something you (only) understand—it’s a value-laden concept, because it depends on what you define as “you”. If your identity includes your body then the consciousness in new body is not yours so your consciousness didn’t persist.
But there is nothing else than patternism that does that.
Sure there is—“soul dies when body dies” preserves the continuity of consciousness until body dies. Or what do you mean by “save”?
They don’t terminally value being made of the same substance, or having the same body, etc. They terminally value their consciousness persisting, and the common decision to care about, let’s say, preserving the substance of their body, is made instrumentally to protect that.
And how did you decide that? Why they couldn’t have been wrong about preserving body just for the soul and then figure out that no, they actually do terminally care about not destroying specific instantiations of their pattern?
Persistence of your consciousness is not something you (only) understand—it’s a value-laden concept, because it depends on what you define as “you”. If your identity includes your body then the consciousness in new body is not yours so your consciousness didn’t persist.
There is only one consciousness. If I define myself as myself + my house and someone demolishes my house, there is no consciousness that has just been destroyed, in reality or in the model. Similarly, if I include my body in the definition of me and then destroy my body, no consciousness has been destroyed. Etc.
We can choose to call my surviving consciousness someone else’s, but that doesn’t make it so. This purely definitional going out of existence isn’t what those people have in mind. What they believe is that they actually black out forever, like after a car accident, and a copy of them who falsely feels like them will actually take their place.
(If I told you one day that I terminally cared about never leaving my house because I defined myself as my body + my house, and if my body leaves my house, I cease to exist, would you think that I’m holding a terminal belief, or would you consider some deep confusion, possibly caused by schizophrenia? I don’t admire the self-determination of people who choose to define themselves as their body—instead, I can see they’re making an error of judgment as to what actually destroys/doesn’t destroy their consciousness (not because of schizophrenia, but nevertheless an error)).
There is only one universe. Everything else, including singling out consciousness, is already definitional (and I don’t see how physical locality, for example, is more definitional than “not blacking out forever”). Thinking overwise is either smuggling epistemic methods to ethics (“but it’s simpler if we model you and house as separate entities!” or something) which is unjustified personal meta-preference and certainly bad without explicitly marking it as such. Or just plain forgetting about subjectivity of high-level concepts such as consciousness. Calling something “consciousness” also doesn’t change underling reality of your brain continuing to exist after a car accident.
What they believe is that they actually black out forever, like after a car accident, and a copy of them who falsely feels like them will actually take their place.
In the case of teleportation, for example, how “this body will no longer instantiate my pattern of consciousness” is not correct description of reality and how “I will black out forever” is not a decent approximation of it? There will be instantiation of blacking out forever—decision to model it as continuing somewhere else is, like you say, definitional.
If I told you one day that I terminally cared about never leaving my house because I defined myself as my body + my house, and if my body leaves my house, I cease to exist, would you think that I’m holding a terminal belief, or would you consider some deep confusion, possibly caused by schizophrenia?
Wait, why do you think schizophrenia doesn’t change terminal values? I would hope it’s just confusion, but if t’s your values then it’s your values.
I don’t admire the self-determination of people who choose to define themselves as their body—instead, I can see they’re making an error of judgment as to what actually destroys/doesn’t destroy their consciousness
You didn’t demonstrate what fact about reality in low-level terms they get wrong.
I don’t see how physical locality, for example, is more definitional than “not blacking out forever”
If by “definitional” you mean “just a matter of semantics,” that would be a bizarre position to hold (it would mean, for example, that if someone shoots you in the head, it’s just a matter of semantics whether you black out forever or not). If by “definitional” you mean something else, please, clarify what.
In the case of teleportation, for example, how “this body will no longer instantiate my pattern of consciousness” is not correct description of reality and how “I will black out forever” is not a decent approximation of it?
The former is correct, while the latter is completely wrong. Since you actually black out forever iff your consciousness is destroyed, and your consciousness isn’t destroyed (it just moves somewhere else).
Saying that you will black out forever after the teleportation is as much of a decent approximation of the truth as saying that you will black out forever after going to sleep.
it would mean, for example, that if someone shoots you in the head, it’s just a matter of semantics whether you black out forever or not
Yes, it is just a matter of semantics, unless you already specified what “you black out forever” means in low-level terms. There are no additional facts about reality that force you to describe your head in a shot state as blacking out forever or even talk about “you” at all.
Since you actually black out forever iff your consciousness is destroyed
And that’s an inference from what law? Defining “destroyed” as “there are no other instantiations of a pattern of your consciousness” is just begging the question.
Saying that you will black out forever after the teleportation is as much of a decent approximation of the truth as saying that you will black out forever after going to sleep
I mean, there are detectable physical differences, but sure, what’s your objection to “sleep is death” preference?
No, it’s not. The low-level terms have to be inferred, not made correct by definition.
Otherwise, you could survive your death by defining yourself to be a black hole (those are extremely long-lived).
And that’s an inference from what law?
There is, of course, no other possibility. If your consciousness survives but you don’t black out forever, that would be a contradiction—similarly, if the latter is true but the former false.
what’s your objection to “sleep is death” preference
Otherwise, you could survive your death by defining yourself to be a black hole (those are extremely long-lived).
That’s the point? There are no physical laws that force you to care about your pattern instead of caring about black hole. And there are no laws that force you to define death one way or another. And no unique value-free way to infer low-level description from high-level concept—that’s ontological shift problem. You can’t “know”, that death means your pattern is not instantiated anywhere, ones you know about atoms or whatever. You can only know that there are atoms and that you can model yourself as using “death” concept in some situations. You still need to define death afterwards. You can motivate high-level concepts by epistemic values—“modeling these atoms as a chair allows me to cheaply predict it”. But there is no fundamental reason why your epistemic values must dictate what high-level concepts you use for your other values.
If your consciousness survives but you don’t black out forever, that would be a contradiction
It’s only contradiction if you define “you blacking out” as your destruction of consciousness. Like I said, it’s begging the question. Nothing is forcing you to equate ethically significant blacking out with “there are no instantiations of your pattern” instead of “current instantiation is destroyed.
That’s the point? There are no physical laws that force you to care about your pattern instead of caring about black hole.
There, you are confusing not caring about what you are with what you are being a matter of semantics.
And there are no laws that force you to define death one way or another.
There you are confusing not being forced by the laws of physics to define death correctly with death being a matter of semantics.
And no unique value-free way to infer low-level description from high-level concept—that’s ontological shift problem.
That’s potentially a good objection, but a high-level concept already has a set of properties, explanatorily prior to it being reduced. They don’t follow from our choice of reduction, rather, our choice of reduction is fixed by them.
Nothing is forcing you to equate ethically significant blacking out with “there are no instantiations of your pattern” instead of “current instantiation is destroyed.
(I assume that means having to keep the same matter.)
You’re conflating two things there—both options being equally correct and nothing forcing us to pick the other option. It’s true nothing forces us to pick the other option, but once we make an analysis of what consciousness, the continuity of consciousness and qualia are, it turns out the correct reduction is to the pattern, and not to the pattern+substance. People who pick other reductions made a mistake in their reasoning somewhere along the way.
This isn’t purely a matter of who wins in a philosophy paper. We’ll have mind uploading relatively soon, and the deaths of people who decide, to show off their wit, to define their substance as a part of them, will be as needless as someone’s who refuses to leave a burning bus because they’re defining it as a part of them.
I pretty much agree with your restatement of my position, but you didn’t present arguments for yours. Yes, I’m saying that all high-level concepts are value-laden. You didn’t specify what do you mean by “correct”, but for “corresponds to reality” it just doesn’t make sense for the definition of death to be incorrect. What do you even mean that it is incorrect, when it describes the same reality?
That’s potentially a good objection, but a high-level concept already has a set of properties, explanatorily prior to it being reduced.
Yeah, they are called “preferences” and the problem is that they are in different language.
I mean, it’s not exactly inconsistent to have a system of preferences about how you resolve ontological shifts and to call that subset of preferences “correct” and nothing is really low-level anyway, but… you do realize that mistakes in reduction are not the same things as mistakes about reality?
This isn’t purely a matter of who wins in a philosophy paper.
So is killing someone by separating them from their bus.
Yes, I’m saying that all high-level concepts are value-laden.
No, concepts are value-neutral. Caring about a concept is distinct from whether a given reduction of a concept is correct, incorrect or arbitrary.
it just doesn’t make sense for the definition of death to be incorrect
You’re confusing semantics and ontology. While all definitions are arbitrary, the ontology of any given existing thing (like consciousness) is objective. (So far it seems that in your head, the arbitrariness of semantics somehow spills over into arbitrariness of ontology, so you think you can just say that to preserve your consciousness, you need to keep the same matter, and it will really be that way).)
you do realize that mistakes in reduction are not the same things as mistakes about reality?
They are a subset of them (because by making a mistake in the first one, we’ll end up mistakenly believing incorrect things about reality (namely, that whatever we reduced our high-level concept to will behave the same way we expect our high-level concept to behave)).
While all definitions are arbitrary, the ontology of any given existing thing (like consciousness) is objective.
How does this work? There is only one objective ontology—true physics. That we don’t know it complicates things somewhat, but on our current understanding (ethically significant) consciousness is not an ontological primitive. Everything is just quantum amplitude. Nothing really changes whether you call some part of universe “consciousness” or “chair” or whatever. Your reduction can’t say things that contradict real ontology, of course—you can’t say “chair is literally these atoms and also it teleports faster than light”. But there is nothing that contradict real ontology in “I am my body”.
No, concepts are value-neutral. Caring about a concept is distinct from whether a given reduction of a concept is correct, incorrect or arbitrary.
There is no objective justification for a concept of a chair. AIXI doesn’t need to think about chairs. Like, really, try to specify correctness of a reduction of chair without appeal to usefulness.
we’ll end up mistakenly believing incorrect things about reality
Wait, but we already assumed that we are using “I am my body” definition that is correct about reality. Well, I assumed. Being incorrect means there must be some atoms in your model that are not in there real place. But “I am my body” doesn’t mean you forget that there will be another body with the same pattern at the destination of a teleporter. Or any other physical consequence. You still haven’t specified what incorrect things about atoms “I am my body” entails.
Is it about you thinking that consciousness specifically is ontologically primitive or that “blacking out” can’t be reduced to whatever you want or something and you would agree if we only talked about chairs? Otherwise I really want to see your specification of what does “correct reduction” means.
How does this work? There is only one objective ontology—true physics. That we don’t know it complicates things somewhat, but on our current understanding (ethically significant) consciousness is not an ontological primitive.
It doesn’t have to be. Aspects of the pattern are the correct ontology of consciousness. They’re not ontologically primitive, but that’s the correct ontology of consciousness. Someone else can use the word consciousness to denote something else, like a chair, but then they are no longer talking about consciousness. They didn’t start talking about chairs (instead of consciousness) because they care about chairs. They started talking about chairs because they mistakenly believe that what we call consciousness reduces to chair-shaped collections of atoms that people can sit in, and if some random good-doer found for them the mistake they made in reducing the concept of consciousness, they would agree that they made a mistake, stop caring about chair-shaped collections of atoms that people can sit in and start caring about consciousness.
Otherwise I really want to see your specification of what does “correct reduction” means.
I don’t have an explicit specification. Maybe it could be something like a process that maps a high-level concept to a lowest-level one while preserving the implicit and explicit properties of the concept.
“Different bodies have different consciousness” is an implicit property of the concept. The whole problem of changing ontology is that you can’t keep all the properties. And there is nothing except your current preferences that can decide which ones you keep.
They didn’t start talking about chairs (instead of consciousness) because they care about chairs.
In what parts (the reduction of) your concept of being mistaken is not isomorphic to caring? Or, if someone just didn’t talk about high-level concepts at all, what are your explicit and implicit properties of correctness that are not satisfied by knowing where all the atoms are and still valuing your body?
“Different bodies have different consciousness” is an implicit property of the concept.
No, it’s not, you just think it is. If you could reflect on all your beliefs, you’d come to the conclusion you were wrong. (For example, transplanting a brain to another body is something you’d (hopefully) agree preserves you. Etc.)
The whole problem of changing ontology is that you can’t keep all the properties.
Why not? All properties are reducible, so you can reduce them along with the concept.
In what parts (the reduction of) your concept of being mistaken is not isomorphic to caring?
All of them. Those are two different concepts. Being mistaken in the reduction means making a logical error at some step that a computer could point at. Caring means having a high-level (or a low-level) concept in your utility function.
Or, if someone just didn’t talk about high-level concepts at all, what are your explicit and implicit properties of correctness that are not satisfied by knowing where all the atoms are and still valuing your body?
If you mean not talking about them in the sense of not referring to them, I’d want to know how they reduced consciousness (or their personal survival) if they couldn’t refer to those concepts in the first place. If they were a computer who already started out as only referring to low-level concepts, they might not be making any mistakes, but they don’t care about the survival of anyone’s consciousness. No human is like that.
Wait, “logical” error? Like, you believe that “transplanting a brain to another body preserves you” is a theorem of QFT + ZFC or something? That… doesn’t make sense—there is no symbol for “brain” in QFT + ZFC.
No, it’s not, you just think it is. If you could reflect on all your beliefs, you’d come to the conclusion you were wrong.
How does it make that conclusion correct?
If you mean not talking about them in the sense of not referring to them, I’d want to know how they reduced consciousness (or their personal survival) if they couldn’t refer to those concepts in the first place.
I mean after they stopped believing in (and valuing) soul they switched to valuing physically correct description of their body without thinking whether it was correct reduction of a soul. And not being a computer they are not perfectly correct in their description, but the point is why not help them correct their description and make them more like a computer valuing the body? Where is mistake in that?
Or even, can you give an example of just one correct step in the reasoning about what are the real properties of a concept (of a chair or consciousness or whatever)?
Wait, “logical” error? Like, you believe that “transplanting a brain to another body preserves you” is a theorem of QFT + ZFC or something? That… doesn’t make sense—there is no symbol for “brain” in QFT + ZFC.
Why would ZFC have to play a role there? By a logical error, I had in mind committing a contradiction, an invalid implication, etc.
In other words, if you consider the explicit properties you believe the concept of yourself to have, and then you compare them against other beliefs you already hold, you’ll discover a contradiction which can be only remedied by accepting the reduction of “you” into a substanceless pattern. There is no other way.
I mean after they stopped believing in (and valuing) soul they switched to valuing physically correct description of their body without thinking whether it was correct reduction of a soul.
That’s not possible. There must’ve been some step in between, even if it wasn’t made explicit, in their decision chain. (For example, they were seeking what low-level concept would fit their high-level concept of themselves (since a soul can no longer fit the bill) and didn’t do the reduction correctly.)
Or even, can you give an example of just one correct step in the reasoning about what are the real properties of a concept (of a chair or consciousness or whatever)?
For example, that step could be imagining slowly replacing your neurons by mechanical ones performing the same function. (Then there would be subsequent steps, which would end with concluding the only possible reduction is to a substanceless pattern.)
By a logical error, I had in mind committing a contradiction, an invalid implication, etc.
Implication from what? There is no chain of implications that starts with “I think I value me” and “everything is atoms” and ends with “transplanting a brain to another body preserves me”. Unless you already have laws for how you reduce things.
In other words, if you consider the explicit properties you believe the concept of yourself to have, and then you compare them against other beliefs you already hold, you’ll discover a contradiction which can be only remedied by accepting the reduction of “you” into a substanceless pattern. There is no other way.
Your beliefs and explicit properties are in different ontologies—there is no law for comparing them. If your current reduction of yourself contradicts your beliefs, you can change your reduction. Yes, a substanceless pattern is a valid change of reduction. But a vast space of other reductions is also contradiction-free (physically possible, in other words) - “I am my body” doesn’t require atoms to be in wrong places. You didn’t present an example of where it does, so you agree, right?
If by “beliefs” you mean high-level approximations, like in addition to “I am my body”, you have “I remain myself after sleep” and then you figure out atoms and start to use “the body after sleep is not really the same”, then obviously there are many other ways to resolve this instead of “I am substanceless pattern”. There is nothing preventing you from saying that “body” should mean different things in “I am my body” and “the body after sleep”, you can conclude that you are not you after sleep—why is one explicit property is better than another if excluding either solves the contradiction? Like I said, is it about consciousness specifically, where you think people can’t be wrong about what way point to when they think about blacking out? Because it’s totally possible to be wrong about your consciousness.
“To be sure, Fading Qualia may be logically possible. Arguably, there is no contradiction in the notion of a system that is so wrong about its experiences.” So, by “correct” you mean “doesn’t feel implausible”? Or what else makes imagining slowly replacing your neurons “correct”?
I mean, where did you even got the idea that it is possible to derive anything ethical using only correctness? That’s is/ought distinction, isn’t it?
There is no chain of implications that starts with “I think I value me” and “everything is atoms” and ends with “transplanting a brain to another body preserves me”.
Right, you need more than those two statements. (Also, the first one doesn’t actually help—it doesn’t matter to the conclusion if you value yourself or not.)
contradiction-free (physically possible, in other words)
Contradiction-free doesn’t mean physically possible.
“I am my body” doesn’t require atoms to be in wrong places
Right. The contradiction is in your brain in the form of the data encoded there. It’s not an incorrect belief about where atoms are.
in addition to “I am my body”, you have “I remain myself after sleep” and then you figure out atoms and start to use “the body after sleep is not really the same”, then obviously there are many other ways to resolve this instead of “I am substanceless pattern”.
There are. The problem is that there is more than one (“I remain myself after sleep”) statement and if you consider all of them together, there is no longer another way.
you can conclude that you are not you after sleep
You can’t. Nobody can actually believe that.
Like I said, is it about consciousness specifically, where you think people can’t be wrong about what way point to when they think about blacking out? Because it’s totally possible to be wrong about your consciousness.
People can be wrong when doing this sort of reasoning, but the solution isn’t to postulate the answer by an axiom. The solution is to be really careful about the reasoning.
“To be sure, Fading Qualia may be logically possible. Arguably, there is no contradiction in the notion of a system that is so wrong about its experiences.” So, by “correct” you mean “doesn’t feel implausible”?
That would require some extremely convoluted theory of consciousness that nobody could believe. (For example, it would contradict one of the things you said previously, where a consciousness belongs to a macroscopic, spatially extended object (like a human body), and that’s what makes the object experience that consciousness. (That wouldn’t be possible on the theory of Fading Qualia (because Joe from the thought experiment doesn’t have fully functioning consciousness even though both the consciousness and his body function correctly, etc.).))
I mean, where did you even got the idea that it is possible to derive anything ethical using only correctness?
The problem is that there is more than one (“I remain myself after sleep”) statement and if you consider all of them together, there is no longer another way.
Well, yes, there are other statements—“I am my body” and “I remain myself after sleep” are among them. If your way allows contradicting “I am my body” then it’s not the only contradiction-free way, and other ways (that contradict other initial statements) are on the same footing. At least as far as logic goes.
The contradiction is in your brain in the form of the data encoded there. It’s not an incorrect belief about where atoms are.
Then patternist identity encodes contradiction to “I am my body” in the same way. And if your choice of statements to contradict is not determined by either logic or beliefs about atoms, then it is determined by your preferences. There is just not much other kinds of stuff in the universe.
That would require some extremely convoluted theory of consciousness that nobody could believe.
So like I said, requirement for a theory of consciousness to not be convoluted is just your preference. Just like any definition of what it means for someone to actually believe something—it’s not logic that forces you, because as long as you have a contradiction anyway, you can say that someone was wrong about themselves not believing in convoluted theory of consciousness—and not knowledge about reality. That’s why it’s about ethics. Or why do you thing someone should prefer non-convoluted theory of consciousness?
For example, it would contradict one of the things you said previously, where a consciousness belongs to a macroscopic, spatially extended object (like a human body), and that’s what makes the object experience that consciousness.
Nah, you can just always make it more convoluted^^. For example I could say that usually microscopic changes are safe, but changing neurons into silicon is too much and destroys consciousness.
Well, yes, there are other statements—“I am my body” and “I remain myself after sleep” are among them.
The first one can’t be there. If you put it there and then we add everything else, there will be some statements we’re psychologically incapable of disbelieving, and us being our body isn’t among them, so it is that statement that will have to go.
And if your choice of statements to contradict is not determined by either logic or beliefs about atoms, then it is determined by your preferences. There is just not much other kinds of stuff in the universe.
There is a fourth kind of data—namely, what our psychological makeup determines we’re capable of believing. (Those aren’t our preferences.)
So like I said, requirement for a theory of consciousness to not be convoluted is just your preference.
Right, but the key part there isn’t that it’s convoluted, but that we’re incapable of believing it.
Caring about what our psychological makeup determines we’re capable of believing, instead of partially operating only on surface reasoning until you change your psychological makeup, is a preference. It’s not a law that you must believe things in whatever sense you mean it for these things to matter. It may be useful for acquiring knowledge, but it’s not “correct” to always do everything that maximally helps your brain know true things. It’s not avoiding mistakes—it’s just selling your soul for knowledge.
Caring about what our psychological makeup determines we’re capable of believing, instead of partially operating only on surface reasoning until you change your psychological makeup, is a preference.
You can’t change your psychological makeup to allow you to hold a self-consistent system of beliefs that would include the belief that you are your body. Even if you could (which you can’t), you haven’t done it yet, so you can’t currently hold such a system of beliefs.
It’s not a law that you must believe things in whatever sense you mean it for these things to matter.
If you don’t believe any system of statements that includes that you are your body, then you have no reason to avoid a mind upload or a teleporter.
If you want to declare that you have null beliefs about what you are and say that you only care about your physical body (instead of believing that that is you), that’s not possible. Humans don’t psychologically work like that.
it’s not “correct” to always do everything that maximally helps your brain know true things
I assume you also, then, think it would be horrible news to think you’re in a simulation?
I don’t see why this follows? I would find it sad if the substrate forming my self ceased to exist, but nothing about a simulation implies that—my “body” wouldn’t exist, but there would be something somewhere that hosted the signals forming me.
Like, the mere fact that we don’t know whether we’re being destroyed and recreated innumerable times every second, seems to rule out the idea that there’s an experiential difference here.
We don’t know in the present, yeah. But in my model it’s like if Omega decided to torture all of us in half the Everett branches leading forward—to the the ones in the remaining branches, there wouldn’t be any experiential difference. But we’d still value negatively that happening to copies of ourselves.
It’s less intuitive in a case where Omega instead decides to kill us in those branches because death by definition isn’t experiential—and I think that might lead to some ontological confusion? Like, I don’t really see there being much of a difference between the two apart from that in one we suffer, and in the other we die. In the case where us being destroyed and recreated constantly is how reality works, I would think that’s sad for the same reason it’s sad if it were Omega making a copy of us constantly and torturing it instead.
I think it’s possible we might agree on some of the practical points if “I anticipate dying as one of the experiences if I use a teletransporter” is valid in your eyes? My disagreement past that would only be that for a given body / substrate my thoughts are currently hosted on, I anticipate just the experiences those thoughts undergo, which I’ll address in the last paragraph.
I can see how parts of this resemble soul-theory behaviourally (but not completely I think? Unless this is just my being bad at modelling soul-theorists, I don’t think they’d consider a copy of themselves to be a perfectly normal sentient being in all ways), but I don’t think they’re the same internally (while this might explain why I believed it in the first place if the object-level arguments were wrong, I’m not convinced of that, so it doesn’t feel that way to me on reflection).
It doesn’t really matter whether I value “my” life over “his”, because we’re going to have the same experiences going forward regardless. But if something breaks this symmetry (note that at that point we’d need to clarify that there are two separate streams of consciousness here, not just one), my preference before the symmetry is broken should be indifferent between which of the copies gets the reward, because both versions of me bear the correct memory-relationship to my present self.
I’m somewhat sceptical about this. If a copy of yourself appeared before your eyes right now, and Omega riddled him with bullets until he died, would you at that moment assign that the same amount of negative utility as you having gotten shot yourself? Insofar as we assign the pain we experience a unique utility value relative to knowing someone else is experiencing it, it’s (at least partly) because we’re feeling it, and the you in that body wouldn’t have. In that view, preferences before the symmetry is broken should still care about the specific body one copy is in. That’s the difference I guess I’m talking about?
(I spent way too long writing this before deciding to just post it, so apologies if it seems janky).
Destroying, minus all of the experiential bad things we mentally associate with “destroying”!
I don’t object to the idea of people having utility functions over various unconscious physical states—e.g., maybe I assign some aesthetic value to having my upload run on cool shiny-looking computronium, rather than something tacky, even if no one’s ever going to see my hardware.
The three things that make me suspicious of your “this is a huge part of my utility function” claim are:
The fact that the properties you’re attached to are things that (for all you know) you never had in the first place. The idea of humans experiencing and enjoying beauty, and then desiring that things be beautiful even in observed parts of the universe, makes total sense to me. The idea of humans coming up with a weird metaphysical hypothesis, and assigning ultimate value to the answer to this hypothesis even though its truth or falsehood has never once made a perceptible difference in anyone’s life, seems far weirder to me.
This alleged utility function component happens to coincide with a powerful and widespread cognitive illusion (the idea of a Soul or Self or Homunculus existing outside determinism / physics / the Cartesian theater).
The alleged importance of this component raises major alarm bells. I wouldn’t be skeptical if you said that you slightly aesthetically prefer bodily continuity, all else equal; but treating this metaphysical stuff as though it were a matter of life or death, or as though it were crucial in the way that the distinction between “me” and “a random stranger” is important, seems like a flag that there’s something wrong here. The “it’s a matter of life or death” thing only makes sense, AFAICT, if you’re subject to a soul-style illusion about what “you” are, and about what your experience of personal continuity currently consists in.
To my eye, this is isomorphic to the following hypothetical (modulo the fact that two copies of me existing may not have the same moral value as one copy of me existing):
I’m playing a VR game.
Iff a VR avatar of mine is killed, I die in real life.
When I use the controls, it causes two different avatars to move. They move in exactly the same way, and since they’re symmetrically positioned in a symmetrical room, there’s no difference between whether I’m seeing through the eyes of one avatar or the other. Through my VR helmet, I am in fact looking through the eyes of just one of the two avatars—I can’t see both of their experiences at once—but since the experiences are identical, it doesn’t matter which one.
It doesn’t really matter whether I value “my” life over “his”, because we’re going to have the same experiences going forward regardless. But if something breaks this symmetry (note that at that point we’d need to clarify that there are two separate streams of consciousness here, not just one), my preference before the symmetry is broken should be indifferent between which of the copies gets the reward, because both versions of me bear the correct memory-relationship to my present self.
(So, I should “anticipate” being both copies of me, for the same reason I should “anticipate” being all of my MWI selves, and anticipate having new experiences after stepping through a teleporter.)
So what? Ones you know that the illusion is wrong, rescuing your utility function even to “the universe is now sad forever” is perfectly fine. So in what way it doesn’t make sense to value some approximation of soul that is more problematic than patternist one? There is no law that says you must use the same reasoning for ethically navigating ontology shifts, that you use for figuring the world out.
Because the reason why people make such an approximation is because they wish to save the continuity of their consciousness. But there is nothing else than patternism that does that.
They don’t terminally value being made of the same substance, or having the same body, etc. They terminally value their consciousness persisting, and the common decision to care about, let’s say, preserving the substance of their body, is made instrumentally to protect that.
Once they someday understand that their consciousness persists independently of that, they will stop valuing it.
Persistence of your consciousness is not something you (only) understand—it’s a value-laden concept, because it depends on what you define as “you”. If your identity includes your body then the consciousness in new body is not yours so your consciousness didn’t persist.
Sure there is—“soul dies when body dies” preserves the continuity of consciousness until body dies. Or what do you mean by “save”?
And how did you decide that? Why they couldn’t have been wrong about preserving body just for the soul and then figure out that no, they actually do terminally care about not destroying specific instantiations of their pattern?
I think we need to go through this first:
There is only one consciousness. If I define myself as myself + my house and someone demolishes my house, there is no consciousness that has just been destroyed, in reality or in the model. Similarly, if I include my body in the definition of me and then destroy my body, no consciousness has been destroyed. Etc.
We can choose to call my surviving consciousness someone else’s, but that doesn’t make it so. This purely definitional going out of existence isn’t what those people have in mind. What they believe is that they actually black out forever, like after a car accident, and a copy of them who falsely feels like them will actually take their place.
(If I told you one day that I terminally cared about never leaving my house because I defined myself as my body + my house, and if my body leaves my house, I cease to exist, would you think that I’m holding a terminal belief, or would you consider some deep confusion, possibly caused by schizophrenia? I don’t admire the self-determination of people who choose to define themselves as their body—instead, I can see they’re making an error of judgment as to what actually destroys/doesn’t destroy their consciousness (not because of schizophrenia, but nevertheless an error)).
There is only one universe. Everything else, including singling out consciousness, is already definitional (and I don’t see how physical locality, for example, is more definitional than “not blacking out forever”). Thinking overwise is either smuggling epistemic methods to ethics (“but it’s simpler if we model you and house as separate entities!” or something) which is unjustified personal meta-preference and certainly bad without explicitly marking it as such. Or just plain forgetting about subjectivity of high-level concepts such as consciousness. Calling something “consciousness” also doesn’t change underling reality of your brain continuing to exist after a car accident.
In the case of teleportation, for example, how “this body will no longer instantiate my pattern of consciousness” is not correct description of reality and how “I will black out forever” is not a decent approximation of it? There will be instantiation of blacking out forever—decision to model it as continuing somewhere else is, like you say, definitional.
Wait, why do you think schizophrenia doesn’t change terminal values? I would hope it’s just confusion, but if t’s your values then it’s your values.
You didn’t demonstrate what fact about reality in low-level terms they get wrong.
If by “definitional” you mean “just a matter of semantics,” that would be a bizarre position to hold (it would mean, for example, that if someone shoots you in the head, it’s just a matter of semantics whether you black out forever or not). If by “definitional” you mean something else, please, clarify what.
The former is correct, while the latter is completely wrong. Since you actually black out forever iff your consciousness is destroyed, and your consciousness isn’t destroyed (it just moves somewhere else).
Saying that you will black out forever after the teleportation is as much of a decent approximation of the truth as saying that you will black out forever after going to sleep.
Yes, it is just a matter of semantics, unless you already specified what “you black out forever” means in low-level terms. There are no additional facts about reality that force you to describe your head in a shot state as blacking out forever or even talk about “you” at all.
And that’s an inference from what law? Defining “destroyed” as “there are no other instantiations of a pattern of your consciousness” is just begging the question.
I mean, there are detectable physical differences, but sure, what’s your objection to “sleep is death” preference?
No, it’s not. The low-level terms have to be inferred, not made correct by definition.
Otherwise, you could survive your death by defining yourself to be a black hole (those are extremely long-lived).
There is, of course, no other possibility. If your consciousness survives but you don’t black out forever, that would be a contradiction—similarly, if the latter is true but the former false.
That it’s a mental illness, I suppose.
That’s the point? There are no physical laws that force you to care about your pattern instead of caring about black hole. And there are no laws that force you to define death one way or another. And no unique value-free way to infer low-level description from high-level concept—that’s ontological shift problem. You can’t “know”, that death means your pattern is not instantiated anywhere, ones you know about atoms or whatever. You can only know that there are atoms and that you can model yourself as using “death” concept in some situations. You still need to define death afterwards. You can motivate high-level concepts by epistemic values—“modeling these atoms as a chair allows me to cheaply predict it”. But there is no fundamental reason why your epistemic values must dictate what high-level concepts you use for your other values.
It’s only contradiction if you define “you blacking out” as your destruction of consciousness. Like I said, it’s begging the question. Nothing is forcing you to equate ethically significant blacking out with “there are no instantiations of your pattern” instead of “current instantiation is destroyed.
So it is a matter of semantics.
There are 2 confusions there:
There, you are confusing not caring about what you are with what you are being a matter of semantics.
There you are confusing not being forced by the laws of physics to define death correctly with death being a matter of semantics.
That’s potentially a good objection, but a high-level concept already has a set of properties, explanatorily prior to it being reduced. They don’t follow from our choice of reduction, rather, our choice of reduction is fixed by them.
(I assume that means having to keep the same matter.)
You’re conflating two things there—both options being equally correct and nothing forcing us to pick the other option. It’s true nothing forces us to pick the other option, but once we make an analysis of what consciousness, the continuity of consciousness and qualia are, it turns out the correct reduction is to the pattern, and not to the pattern+substance. People who pick other reductions made a mistake in their reasoning somewhere along the way.
This isn’t purely a matter of who wins in a philosophy paper. We’ll have mind uploading relatively soon, and the deaths of people who decide, to show off their wit, to define their substance as a part of them, will be as needless as someone’s who refuses to leave a burning bus because they’re defining it as a part of them.
I pretty much agree with your restatement of my position, but you didn’t present arguments for yours. Yes, I’m saying that all high-level concepts are value-laden. You didn’t specify what do you mean by “correct”, but for “corresponds to reality” it just doesn’t make sense for the definition of death to be incorrect. What do you even mean that it is incorrect, when it describes the same reality?
Yeah, they are called “preferences” and the problem is that they are in different language.
I mean, it’s not exactly inconsistent to have a system of preferences about how you resolve ontological shifts and to call that subset of preferences “correct” and nothing is really low-level anyway, but… you do realize that mistakes in reduction are not the same things as mistakes about reality?
So is killing someone by separating them from their bus.
No, concepts are value-neutral. Caring about a concept is distinct from whether a given reduction of a concept is correct, incorrect or arbitrary.
You’re confusing semantics and ontology. While all definitions are arbitrary, the ontology of any given existing thing (like consciousness) is objective. (So far it seems that in your head, the arbitrariness of semantics somehow spills over into arbitrariness of ontology, so you think you can just say that to preserve your consciousness, you need to keep the same matter, and it will really be that way).)
They are a subset of them (because by making a mistake in the first one, we’ll end up mistakenly believing incorrect things about reality (namely, that whatever we reduced our high-level concept to will behave the same way we expect our high-level concept to behave)).
How does this work? There is only one objective ontology—true physics. That we don’t know it complicates things somewhat, but on our current understanding (ethically significant) consciousness is not an ontological primitive. Everything is just quantum amplitude. Nothing really changes whether you call some part of universe “consciousness” or “chair” or whatever. Your reduction can’t say things that contradict real ontology, of course—you can’t say “chair is literally these atoms and also it teleports faster than light”. But there is nothing that contradict real ontology in “I am my body”.
There is no objective justification for a concept of a chair. AIXI doesn’t need to think about chairs. Like, really, try to specify correctness of a reduction of chair without appeal to usefulness.
Wait, but we already assumed that we are using “I am my body” definition that is correct about reality. Well, I assumed. Being incorrect means there must be some atoms in your model that are not in there real place. But “I am my body” doesn’t mean you forget that there will be another body with the same pattern at the destination of a teleporter. Or any other physical consequence. You still haven’t specified what incorrect things about atoms “I am my body” entails.
Is it about you thinking that consciousness specifically is ontologically primitive or that “blacking out” can’t be reduced to whatever you want or something and you would agree if we only talked about chairs? Otherwise I really want to see your specification of what does “correct reduction” means.
It doesn’t have to be. Aspects of the pattern are the correct ontology of consciousness. They’re not ontologically primitive, but that’s the correct ontology of consciousness. Someone else can use the word consciousness to denote something else, like a chair, but then they are no longer talking about consciousness. They didn’t start talking about chairs (instead of consciousness) because they care about chairs. They started talking about chairs because they mistakenly believe that what we call consciousness reduces to chair-shaped collections of atoms that people can sit in, and if some random good-doer found for them the mistake they made in reducing the concept of consciousness, they would agree that they made a mistake, stop caring about chair-shaped collections of atoms that people can sit in and start caring about consciousness.
I don’t have an explicit specification. Maybe it could be something like a process that maps a high-level concept to a lowest-level one while preserving the implicit and explicit properties of the concept.
“Different bodies have different consciousness” is an implicit property of the concept. The whole problem of changing ontology is that you can’t keep all the properties. And there is nothing except your current preferences that can decide which ones you keep.
In what parts (the reduction of) your concept of being mistaken is not isomorphic to caring? Or, if someone just didn’t talk about high-level concepts at all, what are your explicit and implicit properties of correctness that are not satisfied by knowing where all the atoms are and still valuing your body?
No, it’s not, you just think it is. If you could reflect on all your beliefs, you’d come to the conclusion you were wrong. (For example, transplanting a brain to another body is something you’d (hopefully) agree preserves you. Etc.)
Why not? All properties are reducible, so you can reduce them along with the concept.
All of them. Those are two different concepts. Being mistaken in the reduction means making a logical error at some step that a computer could point at. Caring means having a high-level (or a low-level) concept in your utility function.
If you mean not talking about them in the sense of not referring to them, I’d want to know how they reduced consciousness (or their personal survival) if they couldn’t refer to those concepts in the first place. If they were a computer who already started out as only referring to low-level concepts, they might not be making any mistakes, but they don’t care about the survival of anyone’s consciousness. No human is like that.
Wait, “logical” error? Like, you believe that “transplanting a brain to another body preserves you” is a theorem of QFT + ZFC or something? That… doesn’t make sense—there is no symbol for “brain” in QFT + ZFC.
How does it make that conclusion correct?
I mean after they stopped believing in (and valuing) soul they switched to valuing physically correct description of their body without thinking whether it was correct reduction of a soul. And not being a computer they are not perfectly correct in their description, but the point is why not help them correct their description and make them more like a computer valuing the body? Where is mistake in that?
Or even, can you give an example of just one correct step in the reasoning about what are the real properties of a concept (of a chair or consciousness or whatever)?
Why would ZFC have to play a role there? By a logical error, I had in mind committing a contradiction, an invalid implication, etc.
In other words, if you consider the explicit properties you believe the concept of yourself to have, and then you compare them against other beliefs you already hold, you’ll discover a contradiction which can be only remedied by accepting the reduction of “you” into a substanceless pattern. There is no other way.
That’s not possible. There must’ve been some step in between, even if it wasn’t made explicit, in their decision chain. (For example, they were seeking what low-level concept would fit their high-level concept of themselves (since a soul can no longer fit the bill) and didn’t do the reduction correctly.)
For example, that step could be imagining slowly replacing your neurons by mechanical ones performing the same function. (Then there would be subsequent steps, which would end with concluding the only possible reduction is to a substanceless pattern.)
Implication from what? There is no chain of implications that starts with “I think I value me” and “everything is atoms” and ends with “transplanting a brain to another body preserves me”. Unless you already have laws for how you reduce things.
Your beliefs and explicit properties are in different ontologies—there is no law for comparing them. If your current reduction of yourself contradicts your beliefs, you can change your reduction. Yes, a substanceless pattern is a valid change of reduction. But a vast space of other reductions is also contradiction-free (physically possible, in other words) - “I am my body” doesn’t require atoms to be in wrong places. You didn’t present an example of where it does, so you agree, right?
If by “beliefs” you mean high-level approximations, like in addition to “I am my body”, you have “I remain myself after sleep” and then you figure out atoms and start to use “the body after sleep is not really the same”, then obviously there are many other ways to resolve this instead of “I am substanceless pattern”. There is nothing preventing you from saying that “body” should mean different things in “I am my body” and “the body after sleep”, you can conclude that you are not you after sleep—why is one explicit property is better than another if excluding either solves the contradiction? Like I said, is it about consciousness specifically, where you think people can’t be wrong about what way point to when they think about blacking out? Because it’s totally possible to be wrong about your consciousness.
“To be sure, Fading Qualia may be logically possible. Arguably, there is no contradiction in the notion of a system that is so wrong about its experiences.” So, by “correct” you mean “doesn’t feel implausible”? Or what else makes imagining slowly replacing your neurons “correct”?
I mean, where did you even got the idea that it is possible to derive anything ethical using only correctness? That’s is/ought distinction, isn’t it?
Right, you need more than those two statements. (Also, the first one doesn’t actually help—it doesn’t matter to the conclusion if you value yourself or not.)
Contradiction-free doesn’t mean physically possible.
Right. The contradiction is in your brain in the form of the data encoded there. It’s not an incorrect belief about where atoms are.
There are. The problem is that there is more than one (“I remain myself after sleep”) statement and if you consider all of them together, there is no longer another way.
You can’t. Nobody can actually believe that.
People can be wrong when doing this sort of reasoning, but the solution isn’t to postulate the answer by an axiom. The solution is to be really careful about the reasoning.
That would require some extremely convoluted theory of consciousness that nobody could believe. (For example, it would contradict one of the things you said previously, where a consciousness belongs to a macroscopic, spatially extended object (like a human body), and that’s what makes the object experience that consciousness. (That wouldn’t be possible on the theory of Fading Qualia (because Joe from the thought experiment doesn’t have fully functioning consciousness even though both the consciousness and his body function correctly, etc.).))
Oh, I don’t have ethics in mind there.
Well, yes, there are other statements—“I am my body” and “I remain myself after sleep” are among them. If your way allows contradicting “I am my body” then it’s not the only contradiction-free way, and other ways (that contradict other initial statements) are on the same footing. At least as far as logic goes.
Then patternist identity encodes contradiction to “I am my body” in the same way. And if your choice of statements to contradict is not determined by either logic or beliefs about atoms, then it is determined by your preferences. There is just not much other kinds of stuff in the universe.
So like I said, requirement for a theory of consciousness to not be convoluted is just your preference. Just like any definition of what it means for someone to actually believe something—it’s not logic that forces you, because as long as you have a contradiction anyway, you can say that someone was wrong about themselves not believing in convoluted theory of consciousness—and not knowledge about reality. That’s why it’s about ethics. Or why do you thing someone should prefer non-convoluted theory of consciousness?
Nah, you can just always make it more convoluted^^. For example I could say that usually microscopic changes are safe, but changing neurons into silicon is too much and destroys consciousness.
The first one can’t be there. If you put it there and then we add everything else, there will be some statements we’re psychologically incapable of disbelieving, and us being our body isn’t among them, so it is that statement that will have to go.
There is a fourth kind of data—namely, what our psychological makeup determines we’re capable of believing. (Those aren’t our preferences.)
Right, but the key part there isn’t that it’s convoluted, but that we’re incapable of believing it.
Caring about what our psychological makeup determines we’re capable of believing, instead of partially operating only on surface reasoning until you change your psychological makeup, is a preference. It’s not a law that you must believe things in whatever sense you mean it for these things to matter. It may be useful for acquiring knowledge, but it’s not “correct” to always do everything that maximally helps your brain know true things. It’s not avoiding mistakes—it’s just selling your soul for knowledge.
You can’t change your psychological makeup to allow you to hold a self-consistent system of beliefs that would include the belief that you are your body. Even if you could (which you can’t), you haven’t done it yet, so you can’t currently hold such a system of beliefs.
If you don’t believe any system of statements that includes that you are your body, then you have no reason to avoid a mind upload or a teleporter.
If you want to declare that you have null beliefs about what you are and say that you only care about your physical body (instead of believing that that is you), that’s not possible. Humans don’t psychologically work like that.
You can’t avoid that. By the time you are avoiding doing something that would maximally help you know the truth, you already know your current belief is false.
I don’t see why this follows? I would find it sad if the substrate forming my self ceased to exist, but nothing about a simulation implies that—my “body” wouldn’t exist, but there would be something somewhere that hosted the signals forming me.
We don’t know in the present, yeah. But in my model it’s like if Omega decided to torture all of us in half the Everett branches leading forward—to the the ones in the remaining branches, there wouldn’t be any experiential difference. But we’d still value negatively that happening to copies of ourselves.
It’s less intuitive in a case where Omega instead decides to kill us in those branches because death by definition isn’t experiential—and I think that might lead to some ontological confusion? Like, I don’t really see there being much of a difference between the two apart from that in one we suffer, and in the other we die. In the case where us being destroyed and recreated constantly is how reality works, I would think that’s sad for the same reason it’s sad if it were Omega making a copy of us constantly and torturing it instead.
I think it’s possible we might agree on some of the practical points if “I anticipate dying as one of the experiences if I use a teletransporter” is valid in your eyes? My disagreement past that would only be that for a given body / substrate my thoughts are currently hosted on, I anticipate just the experiences those thoughts undergo, which I’ll address in the last paragraph.
I can see how parts of this resemble soul-theory behaviourally (but not completely I think? Unless this is just my being bad at modelling soul-theorists, I don’t think they’d consider a copy of themselves to be a perfectly normal sentient being in all ways), but I don’t think they’re the same internally (while this might explain why I believed it in the first place if the object-level arguments were wrong, I’m not convinced of that, so it doesn’t feel that way to me on reflection).
I’m somewhat sceptical about this. If a copy of yourself appeared before your eyes right now, and Omega riddled him with bullets until he died, would you at that moment assign that the same amount of negative utility as you having gotten shot yourself? Insofar as we assign the pain we experience a unique utility value relative to knowing someone else is experiencing it, it’s (at least partly) because we’re feeling it, and the you in that body wouldn’t have. In that view, preferences before the symmetry is broken should still care about the specific body one copy is in. That’s the difference I guess I’m talking about?
(I spent way too long writing this before deciding to just post it, so apologies if it seems janky).