Then your argument is circular/tautological. You define a “rational” action as one that “does that which ethically good”, and then you suppose that a superintelligence must be very “rational”. However, this is not the conventional usage of “rational” in economics or decision theory (and not on Less Wrong). Also, by this definition, I would not necessarily wish to be “rational”, and the problem of making a superintelligence “rational” is exactly as hard, and basically equivalent to, making it “friendly”.
I’m not sure I’m using rational in that sense, I could substitute “being rational” with “using reason”, “thinking intelligently”, “making sense”, “being logical”, what seems to follow from being generally superintelligent. Ethics is the study of defining what ought to be done and how to achieve it, so it seems to follow from general superintelligence as well. The trickier part seems to be defining ethics. Humans often act with motivations which are not based on formal ethics, but ethics is like a formal elaboration of what one’s (or everyone’s) motivations and actions ought to be.
Hm, sorry, it’s looking increasingly difficult to reach a consensus on this, so I’m going to bow out after this post.
With that in mind, I’d like to say that what I have in mind when I say “an action is rational” is approximately “this action is the best one for achieving one’s goals” (approximately because that ignores practical considerations like the cost of figuring out which action this is exactly). I also personally believe that insofar as ethics is worth talking about at all, it is simply the study of what we socially consider to be convenient to term good, not the search for an absolute, universal good, since such a good (almost certainly) does not exist. As such, the claim that you should always act ethically is not very convincing in my worldview (it is basically equivalent to the claim that you should try to benefit society and is similarly differently persuasive for different people). Instead, each individual should satisfy her own goals, which may be completely umm… orthogonal… to whatever we decide to use for “ethics”. The class of agents that will indeed decide to care about the ethics we like seems like a tiny subset of all potential agents, as well as of all potential superintelligent agents (which is of course just a restatement of the thesis).
Consequently, to me, the idea that we should expect a superintelligence to figure out some absolute ethics (that probably don’t exist) and decide that it should adhere to them looks fanciful.
I see. I think that ethics could be taken as, even individually, the formal definition of one’s goals and how to reach them, although in the orthogonality thesis ethics is taken in a collective level. Since personal identities cannot be sustained by logic, the distinction between individual goals and societal goals becomes trivial, and both are mutually inclusive.
Then your argument is circular/tautological. You define a “rational” action as one that “does that which ethically good”, and then you suppose that a superintelligence must be very “rational”. However, this is not the conventional usage of “rational” in economics or decision theory (and not on Less Wrong). Also, by this definition, I would not necessarily wish to be “rational”, and the problem of making a superintelligence “rational” is exactly as hard, and basically equivalent to, making it “friendly”.
I’m not sure I’m using rational in that sense, I could substitute “being rational” with “using reason”, “thinking intelligently”, “making sense”, “being logical”, what seems to follow from being generally superintelligent. Ethics is the study of defining what ought to be done and how to achieve it, so it seems to follow from general superintelligence as well. The trickier part seems to be defining ethics. Humans often act with motivations which are not based on formal ethics, but ethics is like a formal elaboration of what one’s (or everyone’s) motivations and actions ought to be.
Hm, sorry, it’s looking increasingly difficult to reach a consensus on this, so I’m going to bow out after this post.
With that in mind, I’d like to say that what I have in mind when I say “an action is rational” is approximately “this action is the best one for achieving one’s goals” (approximately because that ignores practical considerations like the cost of figuring out which action this is exactly). I also personally believe that insofar as ethics is worth talking about at all, it is simply the study of what we socially consider to be convenient to term good, not the search for an absolute, universal good, since such a good (almost certainly) does not exist. As such, the claim that you should always act ethically is not very convincing in my worldview (it is basically equivalent to the claim that you should try to benefit society and is similarly differently persuasive for different people). Instead, each individual should satisfy her own goals, which may be completely umm… orthogonal… to whatever we decide to use for “ethics”. The class of agents that will indeed decide to care about the ethics we like seems like a tiny subset of all potential agents, as well as of all potential superintelligent agents (which is of course just a restatement of the thesis).
Consequently, to me, the idea that we should expect a superintelligence to figure out some absolute ethics (that probably don’t exist) and decide that it should adhere to them looks fanciful.
I see. I think that ethics could be taken as, even individually, the formal definition of one’s goals and how to reach them, although in the orthogonality thesis ethics is taken in a collective level. Since personal identities cannot be sustained by logic, the distinction between individual goals and societal goals becomes trivial, and both are mutually inclusive.