I am very confident that an FAI could, if necessary create a person to order, who would be perfectly tuned to becoming someone’s friend in a few hours. How often does this kind of thing happen by accident in kindergarten?
Impossibility should be reserved for things like FTL and reversal of entropy, not straightforward problems of human interaction.
That’s a worst case scenario. Even if necessary, are you willing to die so as to avoid a little creeeeeeeeeeepiness? Honestly, don’t you value your life? Why are you so willing to assume that super intelligence can’t think of any better solutions than you can?
In principle, I’m willing to die to prevent the unethical creation of a person. (I might not act in accordance with this principle if I were presented with a very immediate threat to my survival, which I could avert by unethically creating a person; but the threats here are not immediate enough to cause me to so compromise my ethics.)
Why would the creation of such a person be unethical? Eir life would be worth living, and ey would make you happy as well. Human instincts around creepiness are not good metrics when discussing morality.
I think that people should be created by other persons who are motivated, at least in part, by an expectation to intrinsically value the person so created. If a FAI created a person for the express purpose of being my friend, it would presumably expect to value the person intrinsically, but that wouldn’t be its motivation in creating the person; its motivation in creating the person would have to do with valuing me. And if it modified its motivations to avoid annoying me in this way before it created the person, that would probably have other consequences on its actions that I wouldn’t care for, like motivating it to go around creating lots of persons left and right because people are just so darned intrinsically valuable and more are needed.
I’m sorry, but I’m going to have to call bollocks on this. Jesus Christ, don’t you want to live? Why aren’t you currently opting for euthanasia on the risk you end up friendless tomorrow?
Why aren’t you currently opting for euthanasia on the risk you end up friendless tomorrow?
Well, I probably won’t end up friendless tomorrow; and most of the mechanisms by which that could happen would not prohibit me from “opting for euthanasia”.
You probably won’t end up friendless in the event of a recovery from cryo storage. There is no reason you couldn’t chose to opt for euthanasia then either.
If we modify the case so the FAI isn’t autonomously creating the person, but rather waking me up and quizzing me on what I want em to be like, a) I really doubt I could do that in a timely fashion, and b) I think the creepiness might prevent me from wanting to do it at all.
Would it be less creepy if the FAI found an existing person, out of the billions available, with whom you would be very likely to make friends in a few hours?
That would be fine, and the possibility has already been covered (it was described, I think, as “super-Facebook”) but I wouldn’t bet on it. Frankly, I’m not even sure I’m comfortable with the level of mind-reading the AI would have to do to implement any of these finer-tuned solutions. I like my mental privacy.
You prefer that the hardware inside your head, with its known (and unknown) limitations compute your utility function, as opposed to internal to the aforementioned omniscient being? Why?
Utility functions are actually an extreme of consequentialism; they state that your actions should not just be based on consequences, but a weighted probability distribution over outcomes.
Hmm… I think Eliezer might have overstated his case a little (for the lay audience). If you take a utility function to be normative with respect to your actions, it’s not merely descriptive of your preferences, for some meanings of “preference”—not including, I would think, the definition Eliezer would use.
Using more ordinary language, a Kantian might have preferences about the outcomes of his actions, but doesn’t think such preferences are the primary concern in what one ought to do.
Using more ordinary language, a Kantian might have preferences about the outcomes of his actions, but doesn’t think such preferences are the primary concern in what one ought to do.
Oh. Well, that’s not a distinction that seems terribly important to me. I’m happy to talk about “preferences” as being (necessarily) causally related to one’s actions.
Utility functions describe your preferences. Their existence doesn’t presuppose consequentialism, I don’t think.
There are a few things meant by “consequentialism”. It can be as general as “outcomes/consequences are what’s important when making decisions” to as specific as “Mill’s Utilitarianism”. The term was only coined mid-20th century and it’s not-very-technical jargon, so it hasn’t quite settled yet. I’m pretty sure the use here is more on the general side.
Other theories about what’s important when making decisions (deontology, virtue ethics) could possibly be expressed as utility functions, but are not amenable to it.
Other theories about what’s important when making decisions (deontology, virtue ethics) could possibly be expressed as utility functions, but are not amenable to it.
Why not, if they’re about preferences?
My understanding is that a utility function is nothing but a scaled preference ordering, and I interpret ethical debates as being disputes about what one’s preferences—i.e. one’s utility function—ought to be.
For example (to oversimplify and caricature): the “consequentialist” might argue that one should be willing to torture one person to save 1000 from certain death, while the “deontologist” argues that one should not because Torture is Wrong. Both sides of this argument are asserting preferences about the state of the world: the “consequentialist” assigns higher utility to the situation in which 1000 people are alive and you’re guilty of torture, and the “deontologist” assigns higher utility to the situation in which the 1000 have perished but your hands are clean.
This is called the “consequentialist doppelganger” phenomenon, when I’ve heard it described, and it’s very, very annoying to non-consequentialists. Yes, you can turn any ethical system into a consequentialism by applying the following transformation:
What would the world be like if everyone followed Non-Consequentialism X?
You should act to achieve the outcome yielded by Step 1.
But this ignores what we might call the point of Non-Consequentialism X, which holds that you should follow it for reasons unrelated to how it will make the world be.
But this ignores what we might call the point of Non-Consequentialism X, which holds that you should follow it for reasons unrelated to how it will make the world be.
I’m tempted to ask what kind of reasons could possibly fall into such a category—but we don’t have to have that discussion now unless you particularly want to.
Mainly, I just wanted to point out that when whoever-it-was above mentioned “your utility function”, you probably should have interpreted that as “your preferences”.
I’m tempted to ask what kind of reasons could possibly fall into such a category—but we don’t have to have that discussion now unless you particularly want to.
There should be a “Deontology for Consequentialists” post, if there isn’t already.
Actually, it was exactly the problems with this formulation that I was talking about in the pub with LessWrongers on Saturday. Consequentialism isn’t about maximizing anything; that’s a deontologist’s way of looking at it. Consequentialism says that if action A has a Y better outcome than action B, then action A is better than action B by Y. It follows that the best action is the one with the best outcome, but there isn’t some bright crown on the best action compared to which all other actions are dull and tarnished; other actions are worse to exactly the extent to which they bring about worse consequences, that’s all.
I don’t think this is right. This would seem to indicate that one could do the ethical thing by being a paragon of viciousness if people learned from your example.
Strictly, no. Virtue ethics is self-regarding that way. But it isn’t like virtue ethics says you shouldn’t care about other people’s virtue. It just isn’t calculated at that level of the theory. Helping other people be virtuous is the compassionate and generous thing to do.
I don’t think this is right. This would seem to indicate that one could do the ethical thing by being a paragon of viciousness if people learned from your example.
Such a person is sometimes called a “Mad Bodhisattva”.
Certainly a way I’ve framed it in the past (and it sounds perfectly in line with the Confucian conception of virtue ethics) but I don’t think it’s quite right. At the very least, it’s worth mentioning that a lot of virtue ethicists don’t believe a theory of right action is appropriately part of virtue ethics.
I’m tempted to ask what kind of reasons could possibly fall into such a category—but we don’t have to have that discussion now unless you particularly want to.
Not to butt in but “x is morally obligatory” is a perfectly good reason to do any x. That is the case where x is exhibiting some virtue, following some rule or maximizing some end.
You may run into problems trying to create a utility function for some forms of deontology, at least if you’re mapping into the real numbers. For instance, some deontologists would say that killing a person has infinite negative utility which can’t be cancelled out by any number of positive utility outcomes.
That wouldn’t be mapping into the real numbers, of course, since infinity isn’t a real number.
As I understand it, utility functions are supposed to be equivalence classes of mappings into the real numbers, where two such mappings are said to be equivalent if they are related by a (positive) affine transformation (x → ax + b where a>0).
A strictly monotonic transformation will preserve your preference ordering of states but not your preference ordering for actions to achieve those states. That is, only affine transformations preserve the ordering of expected values of different actions.
Right, which is why I was saying that some ethical theories can’t be expressed by a utility function. And there could be many such incomparable qualities: even adding in infinity and negative infinity may not be enough (though the transfinite ordinals, or the surreal numbers, might be).
I’m surprised at that +b, because that doesn’t preserve utility ratios.
Right, which is why I was saying that some ethical theories can’t be expressed by a utility function.
Ah, I see. But I’m still not actually sure that’s true, though...see below.
I’m surprised at that +b, because that doesn’t preserve utility ratios.
Indeed not; utilities are measured on an interval scale, not a ratio scale. There’s no “absolute zero”. (I believe Eliezer made a youthful mistake along these lines, IIRC.) This expresses the fact that utility functions are just (scaled) preference orderings.
You say you are software, which could be implemented on other computational substrates. You deny the preferability of having a more knowledgeable, less error prone substrate be used to compute your preferences. This is a contradiction. Why are you currently endorsing stupid “terminal” values?
You say you are software, which could be implemented on other computational substrates. You deny the preferability of having a more knowledgeable, less error prone substrate be used to compute your preferences.
Wait, are you suggesting that I be uploaded into something with really excellent computational power so I myself would become a superintelligence? As opposed to an external agent that happened to be superintelligent? That might actually work. I will have to think about that. You could have been less rude in proposing it, though.
No. I am suggesting that the situation I described is what you would find in an FAI. You really should be deferring to Eliezer’s expertise in this case.
What about my statements was rude? How can I present these arguments without making you feel uncomfortable?
No. I am suggesting that the situation I described is what you would find in an FAI.
Then I don’t understand what you said.
You really should be deferring to Eliezer’s expertise in this case.
I will not do that as long as he seems confused about the psychology he’s trying to predict things for.
What about my statements was rude? How can I present these arguments without making you feel uncomfortable?
I think calling my terminal values “stupid” was probably the most egregious bit. It is wise to avoid that word as applied to people and things they care about. I would appreciate it if people who want to help me would react with curiosity, not screeching incredulity and metaphorical tearing out of hair, when they find my statements about myself or other things puzzling or apparently inconsistent.
If he and I are confused, you are seriously failing to describe your situation. You are a human brain. Brains work by physical laws. Bayesian super-intelligences can figure out how to fix the issues you have, even with the handicap of making sure their intervention is acceptable to you.
I understand your antipathy for the word stupid. I shall try to avoid it in the future.
If he and I are confused, you are seriously failing to describe your situation.
Yes, this is very likely. I don’t think I ever claimed that the problem wasn’t in how I was explaining myself; but a fact about my explanation isn’t a fact about the (poorly) explained phenomenon.
Bayesian super-intelligences can figure out how to fix the issues you have, even with the handicap of making sure their intervention is acceptable to you.
I can figure out how to fix the issues I have too: I’m in the process of befriending some more cryonics-friendly people. Why do people think this isn’t going to work? Or does it just seem like a bad way to approach the problem for some reason? Or do people think I won’t follow through on signing up should I acquire a suitable friend, even though I’ve offered to bet money on my being signed up within two years barring immense financial disaster?
Your second paragraph clears up my lingering misunderstandings; that was the missing piece of information for me. We were (or at least I was) arguing about a hypothetical situation instead of the actual situation. What you’re doing sounds perfectly reasonable to me.
A “user-friendly” way to do this would be for the FAI to send an avatar/proxy to act as a guide when you wake up. Explain how things work, introduce you to others who you might enjoy the company off, answer any question you might have, help you get set up in a way that works for you, help you locate people who you know that might be alive, etc.
A FAI would know better than we do what we find creepy/uncomfortable/etc, and would probably avoid it as much as possible.
I am very confident that an FAI could, if necessary create a person to order, who would be perfectly tuned to becoming someone’s friend in a few hours. How often does this kind of thing happen by accident in kindergarten?
Impossibility should be reserved for things like FTL and reversal of entropy, not straightforward problems of human interaction.
Dude, creeeeeeeeeeepy.
That’s a worst case scenario. Even if necessary, are you willing to die so as to avoid a little creeeeeeeeeeepiness? Honestly, don’t you value your life? Why are you so willing to assume that super intelligence can’t think of any better solutions than you can?
In principle, I’m willing to die to prevent the unethical creation of a person. (I might not act in accordance with this principle if I were presented with a very immediate threat to my survival, which I could avert by unethically creating a person; but the threats here are not immediate enough to cause me to so compromise my ethics.)
Why would the creation of such a person be unethical? Eir life would be worth living, and ey would make you happy as well. Human instincts around creepiness are not good metrics when discussing morality.
I think that people should be created by other persons who are motivated, at least in part, by an expectation to intrinsically value the person so created. If a FAI created a person for the express purpose of being my friend, it would presumably expect to value the person intrinsically, but that wouldn’t be its motivation in creating the person; its motivation in creating the person would have to do with valuing me. And if it modified its motivations to avoid annoying me in this way before it created the person, that would probably have other consequences on its actions that I wouldn’t care for, like motivating it to go around creating lots of persons left and right because people are just so darned intrinsically valuable and more are needed.
I’m sorry, but I’m going to have to call bollocks on this. Jesus Christ, don’t you want to live? Why aren’t you currently opting for euthanasia on the risk you end up friendless tomorrow?
Well, I probably won’t end up friendless tomorrow; and most of the mechanisms by which that could happen would not prohibit me from “opting for euthanasia”.
You probably won’t end up friendless in the event of a recovery from cryo storage. There is no reason you couldn’t chose to opt for euthanasia then either.
But in this case, it would be you that creates the person, with purpose of intrinsically valuing em, and the FAI is just a tool you use to do it.
If we modify the case so the FAI isn’t autonomously creating the person, but rather waking me up and quizzing me on what I want em to be like, a) I really doubt I could do that in a timely fashion, and b) I think the creepiness might prevent me from wanting to do it at all.
Would it be less creepy if the FAI found an existing person, out of the billions available, with whom you would be very likely to make friends in a few hours?
That would be fine, and the possibility has already been covered (it was described, I think, as “super-Facebook”) but I wouldn’t bet on it. Frankly, I’m not even sure I’m comfortable with the level of mind-reading the AI would have to do to implement any of these finer-tuned solutions. I like my mental privacy.
I’m not sure mind reading would be necessary. I hear Netflix does a pretty good job of guessing which movies people would like.
You like your mental privacy vis-a-vis an (effectively) omnipotent, perfectly moral being, more than you value your life?
*thinks*
I value the ability to consciously control which of my preferences are acted on that much. Mental privacy qua mental privacy, perhaps not.
You prefer that the hardware inside your head, with its known (and unknown) limitations compute your utility function, as opposed to internal to the aforementioned omniscient being? Why?
No. I’m software. My preferences stand even if you hypothetically implement me in silico.
No. Geez, can we drop the “utility functions” and all the other consequentialism debris for like a week sometime? It would be a welcome respite.
It’s a terminal value. We have a convention of not having to answer “why” about those.
Utility functions describe your preferences. Their existence doesn’t presuppose consequentialism, I don’t think.
Utility functions are actually an extreme of consequentialism; they state that your actions should not just be based on consequences, but a weighted probability distribution over outcomes.
In that case, how could you be said to have preferences about outcomes without being a consequentialist?
Can we not have preferences without a utility function?
Hmm… I think Eliezer might have overstated his case a little (for the lay audience). If you take a utility function to be normative with respect to your actions, it’s not merely descriptive of your preferences, for some meanings of “preference”—not including, I would think, the definition Eliezer would use.
Using more ordinary language, a Kantian might have preferences about the outcomes of his actions, but doesn’t think such preferences are the primary concern in what one ought to do.
Oh. Well, that’s not a distinction that seems terribly important to me. I’m happy to talk about “preferences” as being (necessarily) causally related to one’s actions.
There are a few things meant by “consequentialism”. It can be as general as “outcomes/consequences are what’s important when making decisions” to as specific as “Mill’s Utilitarianism”. The term was only coined mid-20th century and it’s not-very-technical jargon, so it hasn’t quite settled yet. I’m pretty sure the use here is more on the general side.
Other theories about what’s important when making decisions (deontology, virtue ethics) could possibly be expressed as utility functions, but are not amenable to it.
Why not, if they’re about preferences?
My understanding is that a utility function is nothing but a scaled preference ordering, and I interpret ethical debates as being disputes about what one’s preferences—i.e. one’s utility function—ought to be.
For example (to oversimplify and caricature): the “consequentialist” might argue that one should be willing to torture one person to save 1000 from certain death, while the “deontologist” argues that one should not because Torture is Wrong. Both sides of this argument are asserting preferences about the state of the world: the “consequentialist” assigns higher utility to the situation in which 1000 people are alive and you’re guilty of torture, and the “deontologist” assigns higher utility to the situation in which the 1000 have perished but your hands are clean.
This is called the “consequentialist doppelganger” phenomenon, when I’ve heard it described, and it’s very, very annoying to non-consequentialists. Yes, you can turn any ethical system into a consequentialism by applying the following transformation:
What would the world be like if everyone followed Non-Consequentialism X?
You should act to achieve the outcome yielded by Step 1.
But this ignores what we might call the point of Non-Consequentialism X, which holds that you should follow it for reasons unrelated to how it will make the world be.
I’m tempted to ask what kind of reasons could possibly fall into such a category—but we don’t have to have that discussion now unless you particularly want to.
Mainly, I just wanted to point out that when whoever-it-was above mentioned “your utility function”, you probably should have interpreted that as “your preferences”.
There should be a “Deontology for Consequentialists” post, if there isn’t already.
I might write that.
Perhaps I should write “Utilitarianism for Deontologists”. Here goes:
“Follow the maxim: ‘Maximize utility’”.
Actually, it was exactly the problems with this formulation that I was talking about in the pub with LessWrongers on Saturday. Consequentialism isn’t about maximizing anything; that’s a deontologist’s way of looking at it. Consequentialism says that if action A has a Y better outcome than action B, then action A is better than action B by Y. It follows that the best action is the one with the best outcome, but there isn’t some bright crown on the best action compared to which all other actions are dull and tarnished; other actions are worse to exactly the extent to which they bring about worse consequences, that’s all.
I’d like to see you write Virtue Ethics for Consequentialists, or for Deontologists.
“Being virtuous is obligatory, being vicious is forbidden.”
This feels like cheating.
“Do that which leads to people being virtuous.”
I don’t think this is right. This would seem to indicate that one could do the ethical thing by being a paragon of viciousness if people learned from your example.
How about, “Maximize your virtue.”
So other people’s virtue is worth nothing?
Strictly, no. Virtue ethics is self-regarding that way. But it isn’t like virtue ethics says you shouldn’t care about other people’s virtue. It just isn’t calculated at that level of the theory. Helping other people be virtuous is the compassionate and generous thing to do.
Agreed, at least on the common (recent American) ethical egoist reading of virtue ethics.
Such a person is sometimes called a “Mad Bodhisattva”.
Certainly a way I’ve framed it in the past (and it sounds perfectly in line with the Confucian conception of virtue ethics) but I don’t think it’s quite right. At the very least, it’s worth mentioning that a lot of virtue ethicists don’t believe a theory of right action is appropriately part of virtue ethics.
Please do. I’d love to read it.
Ha! I was about to say, “I wonder if Alicorn might be interested in writing such a post”.
Not to butt in but “x is morally obligatory” is a perfectly good reason to do any x. That is the case where x is exhibiting some virtue, following some rule or maximizing some end.
You may run into problems trying to create a utility function for some forms of deontology, at least if you’re mapping into the real numbers. For instance, some deontologists would say that killing a person has infinite negative utility which can’t be cancelled out by any number of positive utility outcomes.
That wouldn’t be mapping into the real numbers, of course, since infinity isn’t a real number.
As I understand it, utility functions are supposed to be equivalence classes of mappings into the real numbers, where two such mappings are said to be equivalent if they are related by a (positive) affine transformation (x → ax + b where a>0).
Why do you think this restricts to positive affine transformations, rather than any strictly monotonic transformation?
Other monotonic transformations don’t preserve preferences over gambles.
Ah, right, that’s what I was missing. Thanks.
A strictly monotonic transformation will preserve your preference ordering of states but not your preference ordering for actions to achieve those states. That is, only affine transformations preserve the ordering of expected values of different actions.
Right, which is why I was saying that some ethical theories can’t be expressed by a utility function. And there could be many such incomparable qualities: even adding in infinity and negative infinity may not be enough (though the transfinite ordinals, or the surreal numbers, might be).
I’m surprised at that +b, because that doesn’t preserve utility ratios.
Ah, I see. But I’m still not actually sure that’s true, though...see below.
Indeed not; utilities are measured on an interval scale, not a ratio scale. There’s no “absolute zero”. (I believe Eliezer made a youthful mistake along these lines, IIRC.) This expresses the fact that utility functions are just (scaled) preference orderings.
You say you are software, which could be implemented on other computational substrates. You deny the preferability of having a more knowledgeable, less error prone substrate be used to compute your preferences. This is a contradiction. Why are you currently endorsing stupid “terminal” values?
Wait, are you suggesting that I be uploaded into something with really excellent computational power so I myself would become a superintelligence? As opposed to an external agent that happened to be superintelligent? That might actually work. I will have to think about that. You could have been less rude in proposing it, though.
No. I am suggesting that the situation I described is what you would find in an FAI. You really should be deferring to Eliezer’s expertise in this case.
What about my statements was rude? How can I present these arguments without making you feel uncomfortable?
Then I don’t understand what you said.
I will not do that as long as he seems confused about the psychology he’s trying to predict things for.
I think calling my terminal values “stupid” was probably the most egregious bit. It is wise to avoid that word as applied to people and things they care about. I would appreciate it if people who want to help me would react with curiosity, not screeching incredulity and metaphorical tearing out of hair, when they find my statements about myself or other things puzzling or apparently inconsistent.
If he and I are confused, you are seriously failing to describe your situation. You are a human brain. Brains work by physical laws. Bayesian super-intelligences can figure out how to fix the issues you have, even with the handicap of making sure their intervention is acceptable to you.
I understand your antipathy for the word stupid. I shall try to avoid it in the future.
Yes, this is very likely. I don’t think I ever claimed that the problem wasn’t in how I was explaining myself; but a fact about my explanation isn’t a fact about the (poorly) explained phenomenon.
I can figure out how to fix the issues I have too: I’m in the process of befriending some more cryonics-friendly people. Why do people think this isn’t going to work? Or does it just seem like a bad way to approach the problem for some reason? Or do people think I won’t follow through on signing up should I acquire a suitable friend, even though I’ve offered to bet money on my being signed up within two years barring immense financial disaster?
Your second paragraph clears up my lingering misunderstandings; that was the missing piece of information for me. We were (or at least I was) arguing about a hypothetical situation instead of the actual situation. What you’re doing sounds perfectly reasonable to me.
If you are willing to take the 1 in 500 chance, my best wishes.
Where did that number come from and what does it refer to?
Actuarial tables, odds of death for a two year period for someone in their twenties (unless I misread the table, which is not at all impossible).
It’s really that likely? Can I see the tables? The number sounds too pessimistic to me.
http://www.socialsecurity.gov/OACT/STATS/table4c6.html
Looks like it should be 1/1000 for two years to me.
It should be around 1 in 400 for males in their 20s and 1 in 1000 for females in their 20s.
I like my mental privacy too, but I am OK with the idea of a non-sentient FAI reading my mind to better predict what it can do for me.
I don’t have much expectation of non-sentience in a sufficiently smart AI.
A “user-friendly” way to do this would be for the FAI to send an avatar/proxy to act as a guide when you wake up. Explain how things work, introduce you to others who you might enjoy the company off, answer any question you might have, help you get set up in a way that works for you, help you locate people who you know that might be alive, etc.
A FAI would know better than we do what we find creepy/uncomfortable/etc, and would probably avoid it as much as possible.