I actually read all the way through and found the broad argument quite understandable (although many of the smaller details were confusing). I also found it obviously wrong on many levels. The one I would consider most essential is that you say:
The existence of personal identities is purely an illusion that cannot be justified by argument, and clearly disintegrates upon deeper analysis...
Different instances in time of a physical organism relate to it in the same way that any other physical organism in the universe does. There is no logical basis for privileging a physical organism’s own viewpoint, nor the satisfaction of their own values over that of other physical organisms...
Assuming I understood correctly, you’re saying that because continuous personal identity isn’t a real thing, there’s no reason to favour one conscious being over another. But that doesn’t follow at all. Just because the “you” a year from now is only somewhat similar to the “you” now doesn’t mean you shouldn’t favour him over everyone else (and indeed there are good reasons for doing so). I wrote a longer comment along these lines in response to some doubts in a recent a discussion of a post about dissolving personal identity.
And it wouldn’t defeat the OT because you’d still have to prove you couldn’t have a utility function over e.g. causal continuity (note: you can have a utility function over causal continuity).
A certain machine could perhaps be programmed with an utility function over causal continuity, but a privileged stance for one’s own values wouldn’t be rational lacking a personal identity, in an objective “God’s eye view”, as David Pearce says. That would call at least for something like coherent extrapolated volition, at least including agents with contextually equivalent reasoning capacity. Note that I use “at least” twice, to accommodate your ethical views. More sensible would be to include not only humans, but all known sentient perspectives, because the ethical value(s) of subjects arguably depend more on sentience than on reasoning capacity.
I argue (in this article) that the you (consciousness) in one second bears little resemblance to the you in the next second.
In the subatomic world, the smallest passage of time changes our composition and arrangement to a great degree, instantly. In physical terms, the frequency of discrete change at this level, even in just one second, is a number with 44 digits, so vast as to be unimaginable… In comparison, the amount of seconds that have passed since the start of the universe, estimated at 13.5 billion years ago, is a number with just 18 digits. At the most fundamental level, our structure, which seems outwardly stable, moves at a staggering pace, like furiously boiling water. Many of our particles are continually lost, and new ones are acquired, as blood frantically keeps matter flowing in and out of each cell.
I also explain why you can’t have partial identity in that paper, and that argues against the position you took (which is similar to that explained by philosopher David Lewis in his paper Survival and Identity).
If we were to be defined as a precise set of particles or arrangement thereof, its permanence in time would be implausibly short-lived; we would be born dead. If this were one’s personal identity, it would have been set for the first time to one’s baby state, having one’s first sparkle of consciousness. In a subatomic level, each second is
like many trillions of years in the macroscopic world, and our primordial state as a babies would be incredibly short-lived. In the blink of an eye, our similarity to what that personal identity was would be reduced to a tiny fraction, if any, by the sheer magnitude of change. That we could survive, in a sense, as a tiny fraction of what we once were, would be an hypothesis that goes against our experience, because we feel consciousness always as an integrated whole, not as a vanishing separated fraction. We either exist completely or not at all.
I recommend reading, whether you agree with this essay or not. The advanced and tenable philosophical positions on this subject are two. Empty individualism, characterized by Derek Parfit in his book “Reasons and Persons”, and open individualism, for which there are better arguments, explained in 4 pages in my essay and more at length in Daniel Kolak’s book “I Am You: The Metaphysical Foundations for Global Ethics”.
I read Kaj Sotala’s post, as you may surmise from the fact that I was the one who first linked (to a comment on it) in the grandparent. I also skimmed your article, and it seems equivalent to the idea of considering algorithmic identity or humans as optimizations processes or what-have-you (not sure if there’s a specific term or post on it) that’s pretty mainstream on LW, and with which I at least partially sympathise.
However, this has nothing to do with my objection. Let me rephrase in more general and philosophical terms, I guess. As far as I can tell, somewhere in your post you purport to solve the is-out problem. However, I do not find that any such solution follows from anything you say.
We seem to be moving from personal identity to ethics. In ethics it is defined that good is what ought to be, and bad is what ought not to be. Ethics is about defining values (what is good and ought to be), and how to cause them.
Good and bad feelings are good and bad as direct data, being direct perceptions, and this quality they have is not an inference. Their good and bad quality is directly accessible by consciousness, as data with the highest epistemic certainty. Being data they are “is”, and being good and bad, under the above definition of ethics, they are “ought” too. This is a special status that only good and bad feelings have, and no other values do.
I’m not convinced by that (specifically that feelings can be sorted into bad and good in a neat way and that we can agree on which ones are more bad/good), however that is still not my point. Sorry, I thought I was being clear, but apparently not.
You claim that a general superintelligence ought to care about all sorts of consciousnesses because it is very very intelligent (and understands what good/bad feelings are and the illusion of personal identities and whatnot). Why? Why wouldn’t it only care about something like the stereotypical example of creating more paperclips?
What is defined as ethically good is by definition what ought to be done, at least rationally. Some agents, such as humans, often don’t act rationally, due to a conflict of reason with evolutionarily selected motivations, which have really their own evolutionary values in mind (e.g. have as many children as possible), not ours. This shouldn’t happen for much more intelligent agents, with stronger rationality (and possibly a capability to self-modify).
Then your argument is circular/tautological. You define a “rational” action as one that “does that which ethically good”, and then you suppose that a superintelligence must be very “rational”. However, this is not the conventional usage of “rational” in economics or decision theory (and not on Less Wrong). Also, by this definition, I would not necessarily wish to be “rational”, and the problem of making a superintelligence “rational” is exactly as hard, and basically equivalent to, making it “friendly”.
I’m not sure I’m using rational in that sense, I could substitute “being rational” with “using reason”, “thinking intelligently”, “making sense”, “being logical”, what seems to follow from being generally superintelligent. Ethics is the study of defining what ought to be done and how to achieve it, so it seems to follow from general superintelligence as well. The trickier part seems to be defining ethics. Humans often act with motivations which are not based on formal ethics, but ethics is like a formal elaboration of what one’s (or everyone’s) motivations and actions ought to be.
Hm, sorry, it’s looking increasingly difficult to reach a consensus on this, so I’m going to bow out after this post.
With that in mind, I’d like to say that what I have in mind when I say “an action is rational” is approximately “this action is the best one for achieving one’s goals” (approximately because that ignores practical considerations like the cost of figuring out which action this is exactly). I also personally believe that insofar as ethics is worth talking about at all, it is simply the study of what we socially consider to be convenient to term good, not the search for an absolute, universal good, since such a good (almost certainly) does not exist. As such, the claim that you should always act ethically is not very convincing in my worldview (it is basically equivalent to the claim that you should try to benefit society and is similarly differently persuasive for different people). Instead, each individual should satisfy her own goals, which may be completely umm… orthogonal… to whatever we decide to use for “ethics”. The class of agents that will indeed decide to care about the ethics we like seems like a tiny subset of all potential agents, as well as of all potential superintelligent agents (which is of course just a restatement of the thesis).
Consequently, to me, the idea that we should expect a superintelligence to figure out some absolute ethics (that probably don’t exist) and decide that it should adhere to them looks fanciful.
I see. I think that ethics could be taken as, even individually, the formal definition of one’s goals and how to reach them, although in the orthogonality thesis ethics is taken in a collective level. Since personal identities cannot be sustained by logic, the distinction between individual goals and societal goals becomes trivial, and both are mutually inclusive.
I actually read all the way through and found the broad argument quite understandable (although many of the smaller details were confusing). I also found it obviously wrong on many levels. The one I would consider most essential is that you say:
Assuming I understood correctly, you’re saying that because continuous personal identity isn’t a real thing, there’s no reason to favour one conscious being over another. But that doesn’t follow at all. Just because the “you” a year from now is only somewhat similar to the “you” now doesn’t mean you shouldn’t favour him over everyone else (and indeed there are good reasons for doing so). I wrote a longer comment along these lines in response to some doubts in a recent a discussion of a post about dissolving personal identity.
And it wouldn’t defeat the OT because you’d still have to prove you couldn’t have a utility function over e.g. causal continuity (note: you can have a utility function over causal continuity).
A certain machine could perhaps be programmed with an utility function over causal continuity, but a privileged stance for one’s own values wouldn’t be rational lacking a personal identity, in an objective “God’s eye view”, as David Pearce says. That would call at least for something like coherent extrapolated volition, at least including agents with contextually equivalent reasoning capacity. Note that I use “at least” twice, to accommodate your ethical views. More sensible would be to include not only humans, but all known sentient perspectives, because the ethical value(s) of subjects arguably depend more on sentience than on reasoning capacity.
I argue (in this article) that the you (consciousness) in one second bears little resemblance to the you in the next second.
I also explain why you can’t have partial identity in that paper, and that argues against the position you took (which is similar to that explained by philosopher David Lewis in his paper Survival and Identity).
I recommend reading, whether you agree with this essay or not. The advanced and tenable philosophical positions on this subject are two. Empty individualism, characterized by Derek Parfit in his book “Reasons and Persons”, and open individualism, for which there are better arguments, explained in 4 pages in my essay and more at length in Daniel Kolak’s book “I Am You: The Metaphysical Foundations for Global Ethics”.
For another interesting take on the subject here on Less Wrong, check Kaj Sotala’s An attempt to dissolve subjective expectation and personal identity.
I read Kaj Sotala’s post, as you may surmise from the fact that I was the one who first linked (to a comment on it) in the grandparent. I also skimmed your article, and it seems equivalent to the idea of considering algorithmic identity or humans as optimizations processes or what-have-you (not sure if there’s a specific term or post on it) that’s pretty mainstream on LW, and with which I at least partially sympathise.
However, this has nothing to do with my objection. Let me rephrase in more general and philosophical terms, I guess. As far as I can tell, somewhere in your post you purport to solve the is-out problem. However, I do not find that any such solution follows from anything you say.
We seem to be moving from personal identity to ethics. In ethics it is defined that good is what ought to be, and bad is what ought not to be. Ethics is about defining values (what is good and ought to be), and how to cause them.
Good and bad feelings are good and bad as direct data, being direct perceptions, and this quality they have is not an inference. Their good and bad quality is directly accessible by consciousness, as data with the highest epistemic certainty. Being data they are “is”, and being good and bad, under the above definition of ethics, they are “ought” too. This is a special status that only good and bad feelings have, and no other values do.
I’m not convinced by that (specifically that feelings can be sorted into bad and good in a neat way and that we can agree on which ones are more bad/good), however that is still not my point. Sorry, I thought I was being clear, but apparently not.
You claim that a general superintelligence ought to care about all sorts of consciousnesses because it is very very intelligent (and understands what good/bad feelings are and the illusion of personal identities and whatnot). Why? Why wouldn’t it only care about something like the stereotypical example of creating more paperclips?
What is defined as ethically good is by definition what ought to be done, at least rationally. Some agents, such as humans, often don’t act rationally, due to a conflict of reason with evolutionarily selected motivations, which have really their own evolutionary values in mind (e.g. have as many children as possible), not ours. This shouldn’t happen for much more intelligent agents, with stronger rationality (and possibly a capability to self-modify).
Then your argument is circular/tautological. You define a “rational” action as one that “does that which ethically good”, and then you suppose that a superintelligence must be very “rational”. However, this is not the conventional usage of “rational” in economics or decision theory (and not on Less Wrong). Also, by this definition, I would not necessarily wish to be “rational”, and the problem of making a superintelligence “rational” is exactly as hard, and basically equivalent to, making it “friendly”.
I’m not sure I’m using rational in that sense, I could substitute “being rational” with “using reason”, “thinking intelligently”, “making sense”, “being logical”, what seems to follow from being generally superintelligent. Ethics is the study of defining what ought to be done and how to achieve it, so it seems to follow from general superintelligence as well. The trickier part seems to be defining ethics. Humans often act with motivations which are not based on formal ethics, but ethics is like a formal elaboration of what one’s (or everyone’s) motivations and actions ought to be.
Hm, sorry, it’s looking increasingly difficult to reach a consensus on this, so I’m going to bow out after this post.
With that in mind, I’d like to say that what I have in mind when I say “an action is rational” is approximately “this action is the best one for achieving one’s goals” (approximately because that ignores practical considerations like the cost of figuring out which action this is exactly). I also personally believe that insofar as ethics is worth talking about at all, it is simply the study of what we socially consider to be convenient to term good, not the search for an absolute, universal good, since such a good (almost certainly) does not exist. As such, the claim that you should always act ethically is not very convincing in my worldview (it is basically equivalent to the claim that you should try to benefit society and is similarly differently persuasive for different people). Instead, each individual should satisfy her own goals, which may be completely umm… orthogonal… to whatever we decide to use for “ethics”. The class of agents that will indeed decide to care about the ethics we like seems like a tiny subset of all potential agents, as well as of all potential superintelligent agents (which is of course just a restatement of the thesis).
Consequently, to me, the idea that we should expect a superintelligence to figure out some absolute ethics (that probably don’t exist) and decide that it should adhere to them looks fanciful.
I see. I think that ethics could be taken as, even individually, the formal definition of one’s goals and how to reach them, although in the orthogonality thesis ethics is taken in a collective level. Since personal identities cannot be sustained by logic, the distinction between individual goals and societal goals becomes trivial, and both are mutually inclusive.