Perhaps the notion that we’re obligated not just to care about the future, but to want it to have our values. There’s a step skipped in going from “I value X” to “I value other agents valuing X.”
This step is glossed over by saying that I have a utility function that values X. But my utility function is full of quasi-indexicals—a fancy way of saying that it has terms like “me” in it, as in, “I value my happiness,” or, “I value the happiness of my people”. If you copy that utility function into another agent’s mind, the my will now refer to that agent.
If we look to the real human world we see immediately that people don’t always want other agents to share their utility function. Kings like being king, and they want everyone else not to want to be king. Social parasites want other people not to be social parasites. Etc.
We also see that, while people profess to care about people on the other side of the world, they don’t. There is a decay of our concern for people with geographic distance, and with distance in time going forward. We care a lot more about people existing today than about people to exist 100 years from now. I would argue that it is impossible for our evolved utility functions to say anything about what goes on outside our “influence cone” (parts of spacetime that can influence us or our descendants, or that we or our descendants can influence), and any care we feel about them is us following some abstract model we’ve built about ourselves, which will be the first thing to go if we ever do find real “human values”.
I’d like the future to have nice things in it: life, consciousness, love. But to have my values? That’s… kind of boring. I want the future to tell an interesting story. That probably requires having a lot of people who don’t share my values.
I know somebody’s going to say, “Well, then that’s your utility function!” Yeah, sorta… but it’s not the sort of thing that “human values” suggests. It’s one or two levels of abstraction above “love your neighbors” or “music based on a 12-tone scale”. It’s not the sort of thing that needs to be controlled tightly to satisfy. It’s not the sort of thing I’d try to optimize rather than satisfice.
The way CEV is phrased, it sounds more like a father trying to raise his kids to be just like him than a father trying to help them grow up.
There isn’t a clear distinction, but CEV is exactly what the Amish have done. They took the values they had in the 18th century, tried to figure out what the minimal, essential values behind them were, and then developed a system for using those core values to extrapolate the Amish position on new developments, like electricity, the telephone, gasoline engines, the Internet, etc. It isn’t a simple rejection of new things; they have an eclectic selection of new things that may be used in certain ways or for certain purposes.
This is an interesting clarification of your early point, but I don’t see how this is a response to what I said.
For one thing, you’re ignoring the ‘if we were smarter, thought clearer’ part since of course the Amish can’t do that since they’re human.
But really, you just gave one negative example. Okay, being Amish is not growing up. What is growing up, and why would we predictably not value it while also finding it proper to object to its being not valued?
When you let your kids grow up, you accept that they won’t do things the way you want them to. They will have other values. You don’t try to optimize them for your value system.
Retaining values is one thing. FAI / CEV is designed to maximize a utility function based on your values. It corresponds to brainwashing your kids to have all of your values and stay as close to your value system as possible. Increasing smartness is beside the point.
I know that is the standard answer. I tried to discourage people from making it by saying, in the parent comment,
I know somebody’s going to say, “Well, then that’s your utility function!”
I’m talking about a real and important distinction, which is the degree of freedom in values to give the next generation. Under standard CEV, it’s zero.
I don’t think that parameter, the degree of freedom, should be thought of as a value, which we can plug any number we like into. It should be thought of as a parameter of the system, which has a predictable impact on the efficacy of the CEV system regardless of what values it is implementing.
I don’t think people allow their children freedom to make up their own minds because they value them doing so. They do it because we have centuries of experience showing that zero-freedom CEV doesn’t work. The oft-attempted process of getting kids to hold the same values as their parents, just modified for the new environment, always turns out badly.
I’m talking about a real and important distinction, which is the degree of freedom in values to give the next generation. Under standard CEV, it’s zero.
No, it’s not.
Zero is the number of degrees of freedom in the AI’s utility function. not the next generation’s utility functions.
You’ve completely lost me. Do you mean, this AI is our child? Do you mean that the way we will have children in a more conventional sense will be an instance of CEV?
If the former, I don’t see a moral problem. A singleton doesn’t get to be a person, even if it contains multitudes (much as the USA does not to get to be a person, though I would hope a singleton would function better).
If the latter… words fail me, at least for the moment, and I will wait for your confirmation before trying again.
Interesting—my interpretation was that ‘I’ would refer to Katja, not the AI, and that the future might not care about the details of music etc if we don’t want the future to care about music per se. But perhaps just because indeed the alternative doesn’t sound very good. I think conversations actually flip flop between the two interpretations, without explicitly flagging it.
Perhaps the notion that we’re obligated not just to care about the future, but to want it to have our values. There’s a step skipped in going from “I value X” to “I value other agents valuing X.”
This step is glossed over by saying that I have a utility function that values X. But my utility function is full of quasi-indexicals—a fancy way of saying that it has terms like “me” in it, as in, “I value my happiness,” or, “I value the happiness of my people”. If you copy that utility function into another agent’s mind, the my will now refer to that agent.
If we look to the real human world we see immediately that people don’t always want other agents to share their utility function. Kings like being king, and they want everyone else not to want to be king. Social parasites want other people not to be social parasites. Etc.
We also see that, while people profess to care about people on the other side of the world, they don’t. There is a decay of our concern for people with geographic distance, and with distance in time going forward. We care a lot more about people existing today than about people to exist 100 years from now. I would argue that it is impossible for our evolved utility functions to say anything about what goes on outside our “influence cone” (parts of spacetime that can influence us or our descendants, or that we or our descendants can influence), and any care we feel about them is us following some abstract model we’ve built about ourselves, which will be the first thing to go if we ever do find real “human values”.
I’d like the future to have nice things in it: life, consciousness, love. But to have my values? That’s… kind of boring. I want the future to tell an interesting story. That probably requires having a lot of people who don’t share my values.
I know somebody’s going to say, “Well, then that’s your utility function!” Yeah, sorta… but it’s not the sort of thing that “human values” suggests. It’s one or two levels of abstraction above “love your neighbors” or “music based on a 12-tone scale”. It’s not the sort of thing that needs to be controlled tightly to satisfy. It’s not the sort of thing I’d try to optimize rather than satisfice.
The way CEV is phrased, it sounds more like a father trying to raise his kids to be just like him than a father trying to help them grow up.
The ‘if we were smarter, thought clearer, etc. etc.’ seems to be asking it to go beyond us.
What else do you mean by ‘growing up’, and why should we value it if it isn’t something we’d approve of?
There isn’t a clear distinction, but CEV is exactly what the Amish have done. They took the values they had in the 18th century, tried to figure out what the minimal, essential values behind them were, and then developed a system for using those core values to extrapolate the Amish position on new developments, like electricity, the telephone, gasoline engines, the Internet, etc. It isn’t a simple rejection of new things; they have an eclectic selection of new things that may be used in certain ways or for certain purposes.
This is an interesting clarification of your early point, but I don’t see how this is a response to what I said.
For one thing, you’re ignoring the ‘if we were smarter, thought clearer’ part since of course the Amish can’t do that since they’re human.
But really, you just gave one negative example. Okay, being Amish is not growing up. What is growing up, and why would we predictably not value it while also finding it proper to object to its being not valued?
When you let your kids grow up, you accept that they won’t do things the way you want them to. They will have other values. You don’t try to optimize them for your value system.
Retaining values is one thing. FAI / CEV is designed to maximize a utility function based on your values. It corresponds to brainwashing your kids to have all of your values and stay as close to your value system as possible. Increasing smartness is beside the point.
If we value them getting to go and make their own choices, then that will be included in CEV.
If we do not value them being brainwashed, it will not be included in CEV.
I strongly suspect that both of these are the case.
I know that is the standard answer. I tried to discourage people from making it by saying, in the parent comment,
I’m talking about a real and important distinction, which is the degree of freedom in values to give the next generation. Under standard CEV, it’s zero.
I don’t think that parameter, the degree of freedom, should be thought of as a value, which we can plug any number we like into. It should be thought of as a parameter of the system, which has a predictable impact on the efficacy of the CEV system regardless of what values it is implementing.
I don’t think people allow their children freedom to make up their own minds because they value them doing so. They do it because we have centuries of experience showing that zero-freedom CEV doesn’t work. The oft-attempted process of getting kids to hold the same values as their parents, just modified for the new environment, always turns out badly.
No, it’s not.
Zero is the number of degrees of freedom in the AI’s utility function. not the next generation’s utility functions.
When using the parent-child relationship as an instance of CEV, it is. The child takes the position of the AI.
You’ve completely lost me. Do you mean, this AI is our child? Do you mean that the way we will have children in a more conventional sense will be an instance of CEV?
If the former, I don’t see a moral problem. A singleton doesn’t get to be a person, even if it contains multitudes (much as the USA does not to get to be a person, though I would hope a singleton would function better).
If the latter… words fail me, at least for the moment, and I will wait for your confirmation before trying again.
Interesting—my interpretation was that ‘I’ would refer to Katja, not the AI, and that the future might not care about the details of music etc if we don’t want the future to care about music per se. But perhaps just because indeed the alternative doesn’t sound very good. I think conversations actually flip flop between the two interpretations, without explicitly flagging it.