I read Kaj Sotala’s post, as you may surmise from the fact that I was the one who first linked (to a comment on it) in the grandparent. I also skimmed your article, and it seems equivalent to the idea of considering algorithmic identity or humans as optimizations processes or what-have-you (not sure if there’s a specific term or post on it) that’s pretty mainstream on LW, and with which I at least partially sympathise.
However, this has nothing to do with my objection. Let me rephrase in more general and philosophical terms, I guess. As far as I can tell, somewhere in your post you purport to solve the is-out problem. However, I do not find that any such solution follows from anything you say.
We seem to be moving from personal identity to ethics. In ethics it is defined that good is what ought to be, and bad is what ought not to be. Ethics is about defining values (what is good and ought to be), and how to cause them.
Good and bad feelings are good and bad as direct data, being direct perceptions, and this quality they have is not an inference. Their good and bad quality is directly accessible by consciousness, as data with the highest epistemic certainty. Being data they are “is”, and being good and bad, under the above definition of ethics, they are “ought” too. This is a special status that only good and bad feelings have, and no other values do.
I’m not convinced by that (specifically that feelings can be sorted into bad and good in a neat way and that we can agree on which ones are more bad/good), however that is still not my point. Sorry, I thought I was being clear, but apparently not.
You claim that a general superintelligence ought to care about all sorts of consciousnesses because it is very very intelligent (and understands what good/bad feelings are and the illusion of personal identities and whatnot). Why? Why wouldn’t it only care about something like the stereotypical example of creating more paperclips?
What is defined as ethically good is by definition what ought to be done, at least rationally. Some agents, such as humans, often don’t act rationally, due to a conflict of reason with evolutionarily selected motivations, which have really their own evolutionary values in mind (e.g. have as many children as possible), not ours. This shouldn’t happen for much more intelligent agents, with stronger rationality (and possibly a capability to self-modify).
Then your argument is circular/tautological. You define a “rational” action as one that “does that which ethically good”, and then you suppose that a superintelligence must be very “rational”. However, this is not the conventional usage of “rational” in economics or decision theory (and not on Less Wrong). Also, by this definition, I would not necessarily wish to be “rational”, and the problem of making a superintelligence “rational” is exactly as hard, and basically equivalent to, making it “friendly”.
I’m not sure I’m using rational in that sense, I could substitute “being rational” with “using reason”, “thinking intelligently”, “making sense”, “being logical”, what seems to follow from being generally superintelligent. Ethics is the study of defining what ought to be done and how to achieve it, so it seems to follow from general superintelligence as well. The trickier part seems to be defining ethics. Humans often act with motivations which are not based on formal ethics, but ethics is like a formal elaboration of what one’s (or everyone’s) motivations and actions ought to be.
Hm, sorry, it’s looking increasingly difficult to reach a consensus on this, so I’m going to bow out after this post.
With that in mind, I’d like to say that what I have in mind when I say “an action is rational” is approximately “this action is the best one for achieving one’s goals” (approximately because that ignores practical considerations like the cost of figuring out which action this is exactly). I also personally believe that insofar as ethics is worth talking about at all, it is simply the study of what we socially consider to be convenient to term good, not the search for an absolute, universal good, since such a good (almost certainly) does not exist. As such, the claim that you should always act ethically is not very convincing in my worldview (it is basically equivalent to the claim that you should try to benefit society and is similarly differently persuasive for different people). Instead, each individual should satisfy her own goals, which may be completely umm… orthogonal… to whatever we decide to use for “ethics”. The class of agents that will indeed decide to care about the ethics we like seems like a tiny subset of all potential agents, as well as of all potential superintelligent agents (which is of course just a restatement of the thesis).
Consequently, to me, the idea that we should expect a superintelligence to figure out some absolute ethics (that probably don’t exist) and decide that it should adhere to them looks fanciful.
I see. I think that ethics could be taken as, even individually, the formal definition of one’s goals and how to reach them, although in the orthogonality thesis ethics is taken in a collective level. Since personal identities cannot be sustained by logic, the distinction between individual goals and societal goals becomes trivial, and both are mutually inclusive.
I read Kaj Sotala’s post, as you may surmise from the fact that I was the one who first linked (to a comment on it) in the grandparent. I also skimmed your article, and it seems equivalent to the idea of considering algorithmic identity or humans as optimizations processes or what-have-you (not sure if there’s a specific term or post on it) that’s pretty mainstream on LW, and with which I at least partially sympathise.
However, this has nothing to do with my objection. Let me rephrase in more general and philosophical terms, I guess. As far as I can tell, somewhere in your post you purport to solve the is-out problem. However, I do not find that any such solution follows from anything you say.
We seem to be moving from personal identity to ethics. In ethics it is defined that good is what ought to be, and bad is what ought not to be. Ethics is about defining values (what is good and ought to be), and how to cause them.
Good and bad feelings are good and bad as direct data, being direct perceptions, and this quality they have is not an inference. Their good and bad quality is directly accessible by consciousness, as data with the highest epistemic certainty. Being data they are “is”, and being good and bad, under the above definition of ethics, they are “ought” too. This is a special status that only good and bad feelings have, and no other values do.
I’m not convinced by that (specifically that feelings can be sorted into bad and good in a neat way and that we can agree on which ones are more bad/good), however that is still not my point. Sorry, I thought I was being clear, but apparently not.
You claim that a general superintelligence ought to care about all sorts of consciousnesses because it is very very intelligent (and understands what good/bad feelings are and the illusion of personal identities and whatnot). Why? Why wouldn’t it only care about something like the stereotypical example of creating more paperclips?
What is defined as ethically good is by definition what ought to be done, at least rationally. Some agents, such as humans, often don’t act rationally, due to a conflict of reason with evolutionarily selected motivations, which have really their own evolutionary values in mind (e.g. have as many children as possible), not ours. This shouldn’t happen for much more intelligent agents, with stronger rationality (and possibly a capability to self-modify).
Then your argument is circular/tautological. You define a “rational” action as one that “does that which ethically good”, and then you suppose that a superintelligence must be very “rational”. However, this is not the conventional usage of “rational” in economics or decision theory (and not on Less Wrong). Also, by this definition, I would not necessarily wish to be “rational”, and the problem of making a superintelligence “rational” is exactly as hard, and basically equivalent to, making it “friendly”.
I’m not sure I’m using rational in that sense, I could substitute “being rational” with “using reason”, “thinking intelligently”, “making sense”, “being logical”, what seems to follow from being generally superintelligent. Ethics is the study of defining what ought to be done and how to achieve it, so it seems to follow from general superintelligence as well. The trickier part seems to be defining ethics. Humans often act with motivations which are not based on formal ethics, but ethics is like a formal elaboration of what one’s (or everyone’s) motivations and actions ought to be.
Hm, sorry, it’s looking increasingly difficult to reach a consensus on this, so I’m going to bow out after this post.
With that in mind, I’d like to say that what I have in mind when I say “an action is rational” is approximately “this action is the best one for achieving one’s goals” (approximately because that ignores practical considerations like the cost of figuring out which action this is exactly). I also personally believe that insofar as ethics is worth talking about at all, it is simply the study of what we socially consider to be convenient to term good, not the search for an absolute, universal good, since such a good (almost certainly) does not exist. As such, the claim that you should always act ethically is not very convincing in my worldview (it is basically equivalent to the claim that you should try to benefit society and is similarly differently persuasive for different people). Instead, each individual should satisfy her own goals, which may be completely umm… orthogonal… to whatever we decide to use for “ethics”. The class of agents that will indeed decide to care about the ethics we like seems like a tiny subset of all potential agents, as well as of all potential superintelligent agents (which is of course just a restatement of the thesis).
Consequently, to me, the idea that we should expect a superintelligence to figure out some absolute ethics (that probably don’t exist) and decide that it should adhere to them looks fanciful.
I see. I think that ethics could be taken as, even individually, the formal definition of one’s goals and how to reach them, although in the orthogonality thesis ethics is taken in a collective level. Since personal identities cannot be sustained by logic, the distinction between individual goals and societal goals becomes trivial, and both are mutually inclusive.