Perhaps the notion that we’re obligated not just to care about the future, but to want it to have our values. There’s a step skipped in going from “I value X” to “I value other agents valuing X.”
This step is glossed over by saying that I have a utility function that values X. But my utility function is full of quasi-indexicals—a fancy way of saying that it has terms like “me” in it, as in, “I value my happiness,” or, “I value the happiness of my people”. If you copy that utility function into another agent’s mind, the my will now refer to that agent.
If we look to the real human world we see immediately that people don’t always want other agents to share their utility function. Kings like being king, and they want everyone else not to want to be king. Social parasites want other people not to be social parasites. Etc.
We also see that, while people profess to care about people on the other side of the world, they don’t. There is a decay of our concern for people with geographic distance, and with distance in time going forward. We care a lot more about people existing today than about people to exist 100 years from now. I would argue that it is impossible for our evolved utility functions to say anything about what goes on outside our “influence cone” (parts of spacetime that can influence us or our descendants, or that we or our descendants can influence), and any care we feel about them is us following some abstract model we’ve built about ourselves, which will be the first thing to go if we ever do find real “human values”.
I’d like the future to have nice things in it: life, consciousness, love. But to have my values? That’s… kind of boring. I want the future to tell an interesting story. That probably requires having a lot of people who don’t share my values.
I know somebody’s going to say, “Well, then that’s your utility function!” Yeah, sorta… but it’s not the sort of thing that “human values” suggests. It’s one or two levels of abstraction above “love your neighbors” or “music based on a 12-tone scale”. It’s not the sort of thing that needs to be controlled tightly to satisfy. It’s not the sort of thing I’d try to optimize rather than satisfice.
The way CEV is phrased, it sounds more like a father trying to raise his kids to be just like him than a father trying to help them grow up.
There isn’t a clear distinction, but CEV is exactly what the Amish have done. They took the values they had in the 18th century, tried to figure out what the minimal, essential values behind them were, and then developed a system for using those core values to extrapolate the Amish position on new developments, like electricity, the telephone, gasoline engines, the Internet, etc. It isn’t a simple rejection of new things; they have an eclectic selection of new things that may be used in certain ways or for certain purposes.
This is an interesting clarification of your early point, but I don’t see how this is a response to what I said.
For one thing, you’re ignoring the ‘if we were smarter, thought clearer’ part since of course the Amish can’t do that since they’re human.
But really, you just gave one negative example. Okay, being Amish is not growing up. What is growing up, and why would we predictably not value it while also finding it proper to object to its being not valued?
When you let your kids grow up, you accept that they won’t do things the way you want them to. They will have other values. You don’t try to optimize them for your value system.
Retaining values is one thing. FAI / CEV is designed to maximize a utility function based on your values. It corresponds to brainwashing your kids to have all of your values and stay as close to your value system as possible. Increasing smartness is beside the point.
I know that is the standard answer. I tried to discourage people from making it by saying, in the parent comment,
I know somebody’s going to say, “Well, then that’s your utility function!”
I’m talking about a real and important distinction, which is the degree of freedom in values to give the next generation. Under standard CEV, it’s zero.
I don’t think that parameter, the degree of freedom, should be thought of as a value, which we can plug any number we like into. It should be thought of as a parameter of the system, which has a predictable impact on the efficacy of the CEV system regardless of what values it is implementing.
I don’t think people allow their children freedom to make up their own minds because they value them doing so. They do it because we have centuries of experience showing that zero-freedom CEV doesn’t work. The oft-attempted process of getting kids to hold the same values as their parents, just modified for the new environment, always turns out badly.
I’m talking about a real and important distinction, which is the degree of freedom in values to give the next generation. Under standard CEV, it’s zero.
No, it’s not.
Zero is the number of degrees of freedom in the AI’s utility function. not the next generation’s utility functions.
You’ve completely lost me. Do you mean, this AI is our child? Do you mean that the way we will have children in a more conventional sense will be an instance of CEV?
If the former, I don’t see a moral problem. A singleton doesn’t get to be a person, even if it contains multitudes (much as the USA does not to get to be a person, though I would hope a singleton would function better).
If the latter… words fail me, at least for the moment, and I will wait for your confirmation before trying again.
Interesting—my interpretation was that ‘I’ would refer to Katja, not the AI, and that the future might not care about the details of music etc if we don’t want the future to care about music per se. But perhaps just because indeed the alternative doesn’t sound very good. I think conversations actually flip flop between the two interpretations, without explicitly flagging it.
I guess I disagree with the premise that we will have superintelligent successors who will think circles around us, and yet we get to specify in detail what ethical values they will have, and it will stick. Forever. So let’s debate what values to specify.
A parent would be crazy to think this way about a daughter, optimizing in detail the order of priorities that he intends to implant into her, and expecting them to stick. But if your daughter is a superintelligence, it’s even crazier.
A parent would be crazy to think this way about a daughter, optimizing in detail the order of priorities that he intends to implant into her, and expecting them to stick. But if your daughter is a superintelligence, it’s even crazier.
Suppose it’s twenty years from now, and know exactly what genes go into the heritable portion of intelligence and personality, which includes both stuff like the Big Five and the weird preferences twins sometimes share.
Suppose further that genetic modification of children is possible and acceptable, and you and your partner have decided that you’ll have a daughter, and naturally you want her IQ to be as high as possible (suppose that’s 170 on today’s scale). So she’s going to be able to think circles around you, but be comparable to her augmented classmates. But personality isn’t as obvious. Do you really want her to be maximally agreeable? Extraverted? Open? The other two might be easy to agree on; you might decide to zero out her neuroticism without much debate, and maximize her conscientiousness without much more.
But, importantly, her ability to outthink you doesn’t mean she will outthink the personality you chose for her. Why would she want to? It’s her personality.
That’s what a non-crazy version looks like: we know that personality traits are at least partly heritable for humans, and so we can imagine manipulating what personality traits future humans have by manipulating their genes. We also have some idea of how raising children impacts their personality / methods of relating with other people, and we can similarly imagine manipulating their early environment to get the personalities and relationships that we want.
We can further strengthen the analogy by considering the next generation. Your daughter has found a partner and is considering having a granddaughter; the IQ manipulation technology has improved to the point where the granddaughter is expected to score the equivalent of 220 on today’s scale, but there’s still a comparable personality question. If you were highly open and decided that your daughter should be highly open too, it seems likely that your daughter will use similar logic to decide that your granddaughter should also be highly open.
That is, a value-optimizing AI will only create a descendant AI (including later versions of itself) that it thinks will fulfill its values, and so it will likely encode its values (at least implicitly) into that AI. So it is important to get right the process of thinking about value preservation, so we can both pass our values and our sense of how to look at values and decide what to preserve down.
The older I get and the more I think of the AI issues the more I realize how perfectly our universe is designed! I think about the process of growing up: I cherish the time I spent in each stage of life, unaware of what’s to come later, because there are things to be learned that can only derive from that particular segment’s challenges. Each stage has its own level of “foolishness”, but that is absolutely necessary for those lessons to be learned! So too I think of the catastrophes I have endured that I would not have chosen, but that I would not trade for anything now due to the wisdom they provided me. I cannot see any way around the difficult life as the most supreme and loving teacher. This I think most parents would recognize as they wish for their kids: a life not too painful but not too easy, either.
CEV assumes that there is an arrival point that is more valuable than the dynamic process we undergo daily. Much as we delight in imagining a utopia, a truly good future is one that we STRUGGLE to achieve, balancing one hard-won value against another, is it not? I have not yet heard a single concept that arrives at wisdom without a difficult journey. Even the idea of a SI that dictates our behavior so that all act within its accordance has destroyed free will, much like a God who has revoked human volition. This leads me to a seemingly inevitable conclusion that no universe is preferable to the one we inhabit (though I have yet to see the value of horrible events in my future that I still try like the devil to avoid!) But despite this ‘perfection’ we’re seemingly unable to stop ourselves from destroying it.
What do you most disagree with this week?
Perhaps the notion that we’re obligated not just to care about the future, but to want it to have our values. There’s a step skipped in going from “I value X” to “I value other agents valuing X.”
This step is glossed over by saying that I have a utility function that values X. But my utility function is full of quasi-indexicals—a fancy way of saying that it has terms like “me” in it, as in, “I value my happiness,” or, “I value the happiness of my people”. If you copy that utility function into another agent’s mind, the my will now refer to that agent.
If we look to the real human world we see immediately that people don’t always want other agents to share their utility function. Kings like being king, and they want everyone else not to want to be king. Social parasites want other people not to be social parasites. Etc.
We also see that, while people profess to care about people on the other side of the world, they don’t. There is a decay of our concern for people with geographic distance, and with distance in time going forward. We care a lot more about people existing today than about people to exist 100 years from now. I would argue that it is impossible for our evolved utility functions to say anything about what goes on outside our “influence cone” (parts of spacetime that can influence us or our descendants, or that we or our descendants can influence), and any care we feel about them is us following some abstract model we’ve built about ourselves, which will be the first thing to go if we ever do find real “human values”.
I’d like the future to have nice things in it: life, consciousness, love. But to have my values? That’s… kind of boring. I want the future to tell an interesting story. That probably requires having a lot of people who don’t share my values.
I know somebody’s going to say, “Well, then that’s your utility function!” Yeah, sorta… but it’s not the sort of thing that “human values” suggests. It’s one or two levels of abstraction above “love your neighbors” or “music based on a 12-tone scale”. It’s not the sort of thing that needs to be controlled tightly to satisfy. It’s not the sort of thing I’d try to optimize rather than satisfice.
The way CEV is phrased, it sounds more like a father trying to raise his kids to be just like him than a father trying to help them grow up.
The ‘if we were smarter, thought clearer, etc. etc.’ seems to be asking it to go beyond us.
What else do you mean by ‘growing up’, and why should we value it if it isn’t something we’d approve of?
There isn’t a clear distinction, but CEV is exactly what the Amish have done. They took the values they had in the 18th century, tried to figure out what the minimal, essential values behind them were, and then developed a system for using those core values to extrapolate the Amish position on new developments, like electricity, the telephone, gasoline engines, the Internet, etc. It isn’t a simple rejection of new things; they have an eclectic selection of new things that may be used in certain ways or for certain purposes.
This is an interesting clarification of your early point, but I don’t see how this is a response to what I said.
For one thing, you’re ignoring the ‘if we were smarter, thought clearer’ part since of course the Amish can’t do that since they’re human.
But really, you just gave one negative example. Okay, being Amish is not growing up. What is growing up, and why would we predictably not value it while also finding it proper to object to its being not valued?
When you let your kids grow up, you accept that they won’t do things the way you want them to. They will have other values. You don’t try to optimize them for your value system.
Retaining values is one thing. FAI / CEV is designed to maximize a utility function based on your values. It corresponds to brainwashing your kids to have all of your values and stay as close to your value system as possible. Increasing smartness is beside the point.
If we value them getting to go and make their own choices, then that will be included in CEV.
If we do not value them being brainwashed, it will not be included in CEV.
I strongly suspect that both of these are the case.
I know that is the standard answer. I tried to discourage people from making it by saying, in the parent comment,
I’m talking about a real and important distinction, which is the degree of freedom in values to give the next generation. Under standard CEV, it’s zero.
I don’t think that parameter, the degree of freedom, should be thought of as a value, which we can plug any number we like into. It should be thought of as a parameter of the system, which has a predictable impact on the efficacy of the CEV system regardless of what values it is implementing.
I don’t think people allow their children freedom to make up their own minds because they value them doing so. They do it because we have centuries of experience showing that zero-freedom CEV doesn’t work. The oft-attempted process of getting kids to hold the same values as their parents, just modified for the new environment, always turns out badly.
No, it’s not.
Zero is the number of degrees of freedom in the AI’s utility function. not the next generation’s utility functions.
When using the parent-child relationship as an instance of CEV, it is. The child takes the position of the AI.
You’ve completely lost me. Do you mean, this AI is our child? Do you mean that the way we will have children in a more conventional sense will be an instance of CEV?
If the former, I don’t see a moral problem. A singleton doesn’t get to be a person, even if it contains multitudes (much as the USA does not to get to be a person, though I would hope a singleton would function better).
If the latter… words fail me, at least for the moment, and I will wait for your confirmation before trying again.
Interesting—my interpretation was that ‘I’ would refer to Katja, not the AI, and that the future might not care about the details of music etc if we don’t want the future to care about music per se. But perhaps just because indeed the alternative doesn’t sound very good. I think conversations actually flip flop between the two interpretations, without explicitly flagging it.
I guess I disagree with the premise that we will have superintelligent successors who will think circles around us, and yet we get to specify in detail what ethical values they will have, and it will stick. Forever. So let’s debate what values to specify.
A parent would be crazy to think this way about a daughter, optimizing in detail the order of priorities that he intends to implant into her, and expecting them to stick. But if your daughter is a superintelligence, it’s even crazier.
Suppose it’s twenty years from now, and know exactly what genes go into the heritable portion of intelligence and personality, which includes both stuff like the Big Five and the weird preferences twins sometimes share.
Suppose further that genetic modification of children is possible and acceptable, and you and your partner have decided that you’ll have a daughter, and naturally you want her IQ to be as high as possible (suppose that’s 170 on today’s scale). So she’s going to be able to think circles around you, but be comparable to her augmented classmates. But personality isn’t as obvious. Do you really want her to be maximally agreeable? Extraverted? Open? The other two might be easy to agree on; you might decide to zero out her neuroticism without much debate, and maximize her conscientiousness without much more.
But, importantly, her ability to outthink you doesn’t mean she will outthink the personality you chose for her. Why would she want to? It’s her personality.
That’s what a non-crazy version looks like: we know that personality traits are at least partly heritable for humans, and so we can imagine manipulating what personality traits future humans have by manipulating their genes. We also have some idea of how raising children impacts their personality / methods of relating with other people, and we can similarly imagine manipulating their early environment to get the personalities and relationships that we want.
We can further strengthen the analogy by considering the next generation. Your daughter has found a partner and is considering having a granddaughter; the IQ manipulation technology has improved to the point where the granddaughter is expected to score the equivalent of 220 on today’s scale, but there’s still a comparable personality question. If you were highly open and decided that your daughter should be highly open too, it seems likely that your daughter will use similar logic to decide that your granddaughter should also be highly open.
That is, a value-optimizing AI will only create a descendant AI (including later versions of itself) that it thinks will fulfill its values, and so it will likely encode its values (at least implicitly) into that AI. So it is important to get right the process of thinking about value preservation, so we can both pass our values and our sense of how to look at values and decide what to preserve down.
The older I get and the more I think of the AI issues the more I realize how perfectly our universe is designed! I think about the process of growing up: I cherish the time I spent in each stage of life, unaware of what’s to come later, because there are things to be learned that can only derive from that particular segment’s challenges. Each stage has its own level of “foolishness”, but that is absolutely necessary for those lessons to be learned! So too I think of the catastrophes I have endured that I would not have chosen, but that I would not trade for anything now due to the wisdom they provided me. I cannot see any way around the difficult life as the most supreme and loving teacher. This I think most parents would recognize as they wish for their kids: a life not too painful but not too easy, either.
CEV assumes that there is an arrival point that is more valuable than the dynamic process we undergo daily. Much as we delight in imagining a utopia, a truly good future is one that we STRUGGLE to achieve, balancing one hard-won value against another, is it not? I have not yet heard a single concept that arrives at wisdom without a difficult journey. Even the idea of a SI that dictates our behavior so that all act within its accordance has destroyed free will, much like a God who has revoked human volition. This leads me to a seemingly inevitable conclusion that no universe is preferable to the one we inhabit (though I have yet to see the value of horrible events in my future that I still try like the devil to avoid!) But despite this ‘perfection’ we’re seemingly unable to stop ourselves from destroying it.