In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together;
If Archimedes (or a random person) could live for a thousand years (together with the rest of humanity), could think a billion times faster, could learn all the FAI knows, etc, etc, then they’d very likely arrive at the same answer as a modern person. So it shouldn’t matter which person/people you pick. This is how the extrapolation is supposed to work.
If Archimedes (or a random person) could live for a thousand years (together with the rest of humanity), could think a billion times faster, could learn all the FAI knows, etc, etc, then they’d very likely arrive at the same answer as a modern person.
First, this seems implausible (people really do have different desires and personalities), and second, if plausible, the starting location doesn’t seem to matter. If you take Archimedes and a brain-damaged murderer and a chimpanzee and end up with similar outputs, what did you need the inputs for? They don’t appear to have made much difference. If the answer is that you’ll modify the chimp and the murderer to become like Archimedes, then why don’t you just copy Archimedes directly instead of doing it by scrubbing out those individuals and then planting Archimedes inside?
Until CEV has a plan for dealing with envy, it strikes me as underplanned. Real humans interfere with each other- that’s part of what makes life human.
So, what all humans have in common is that their most dangerous enemy are other humans. I don’t see how that’s getting us anywhere good.
[edit]I forgot to add- I’m looking for specificity here. What do humans care about that chimps don’t? If chimps thought ten thousand times faster, would they care about those things?
I will clarify: My second link, “Thou Art Godshatter”, says that our values are causally rooted in evolution, which gave us our brains. CEV is supposed to extrapolate that chain of causality, in a certain way. My first link, “The Psychological Unity Of Humankind”, says that human brains are similar to each other. Chimpanzee brains are unlike human brains. There is no reason to expect human universal values to be identical to chimpanzee universal values. There is no reason to expect these species’ extrapolated values to be identical. That is what i am saying. Are you skeptical of this argument?
What do humans care about that chimps don’t?
Art, for example.
If chimps thought ten thousand times faster, would they care about those things?
The question you want to ask is, Would a CEV-extrapolated chimpanzee care about art? I don’t see why it should.
There is no reason to expect human universal values to be identical to chimpanzee universal values.
Sure there is- “maximize inclusive genetic fitness.”
Art, for example.
Are you familiar with the theory that the human brain is essentially a peacock’s tail, and that language, creative endeavors, and so on were evolved primarily to attract mating partners?
It’s possible that a chimp that gets a lot smarter won’t start having more intellectual tastes, but the basic problem of “who do I have sex with, and how do I get them to agree?” is still around and it seems like they would try very similar methods to what we have.
Ah, it seems we have different ideas about what human values actually are.
“maximize inclusive genetic fitness.”
Organisms are adaptation-executers, not fitness-maximizers. Art is a human terminal value. Maximizing inclusive genetic fitness is not. Even if the causal reason for us having this value were that artsy people get more sex, we would value art for its own sake.
I highly recommend the wiki article on the complexity of value and the articles linked from it.
Ah, it seems we have different ideas about what human values actually are.
Wait wait wait. Are we talking about human values, or human universal values? Those seem to me to be different concepts. It seems to me that what we have in common is that we’re competing with one another and what differs between humans is how we compete and what we seek to maximize.
I think the difference between my approach to music and a musician’s approach to music is more than a difference of degree, and so am reluctant to say we share the same value. Given the many differing attitudes towards justice, it seems more apt to call it a political ploy about precedents than one objective standard all humans strive towards. I could keep going, but hopefully two examples is sufficient. Human values seem to be individual values, first and foremost.
And when we remove some human values- predation, cruelty, rape, domination- from the list because their time has passed, it is not clear to me why the others remain. If we alter ourselves so we no longer understand events as narratives (since that’s a motherlode of bias right there), will literature be a casualty of that improvement?
Agreed that individual humans have differing values. However I believe there’s more to say about human universal values than “we’re competing with one another”. Here are two pieces of evidence:
(1) Almost all human cultures have what we recognize as art. So far as I know, chimpanzee cultures do not. (Although a quick google search tells me that chimps who live in human cultures do create art.) So art, broadly defined, is universal; but individual humans have more specific tastes. I expect humans and chimpanzees have senses of justice (differing from individual to individual).
(2) Individuals can change their values through moral arguments and education. If I try, through moral argument, to convince someone of my idea of justice, it’s because I feel that my idea of justice has at least a somewhat universal appeal. And, with sufficient resources, you could become a musician and gain a richer appreciation of music. One could imagine an extrapolation process under which individuals’ values converge, at least some of the time.
However:
And when we remove some human values- predation, cruelty, rape, domination- from the list because their time has passed, it is not clear to me why the others remain.
I share the concern that I think you’re raising here: For all we know, moral progress might not be pointing towards any particular destination. More research needs to be done on the mechanics of morality.
And, with sufficient resources, you could become a musician and gain a richer appreciation of music.
This I’m not as sure about. I can become competent at playing an instrument and better at interpreting music, but I’m not sure I can rewrite my talents. If those are determined by a relatively inelastic part of my brain’s configuration, then it seems likely that the pathway to becoming a Beethoven simply does not exist for me.
A better example might be heroic compassion- people who put themselves at risk without thinking to save others. The consensus opinion is it’s probably at least somewhat genetic, and you either are a hero (in that narrow sense) or you aren’t. I could be roused to tears by tales of heroic compassion but not have the unthinking impulse to do it, and it’s not clear that if I don’t have it I could acquire it. There are other things that people do acquire- bodyguards learn how to operate after being shot, and soldiers learn how to stay active in combat situations- so it might be possible. But I think bounded potentials are more realistic than unbounded potentials.
Aumann’s agreement theorem seems to imply that individual extrapolated volition assignments must agree on statements of fact: the setup implies that they’re simulated as perfect reasoners and share a knowledge pool, and the extrapolation process provides for an unbounded number of Bayesian updates. So we can expect extrapolated volition to cohere exactly to the extent that it’s based on common fundamental goals: not immediate desires and not possibly-fallible philosophical results, but the low-level affective assignments that lead us to think of those higher-level results as desirable or undesirable.
To what extent do common fundamental goals drive our moral reasoning? I don’t know, but individual differences do exist (the existence of masochistic people should prove that), and if they’re large enough then CEV may end up looking incomplete or unpleasantly compromise-driven.
I don’t know, but individual differences do exist (the existence of masochistic people should prove that)
But is that relevant to the question that CEV tries to answer? As far as I know, most masochistic people don’t also hold a belief that everybody should be masochistic.
Even if individual differences in fundamental goals are not extended to other people as imperatives, they imply that the ability of a coherent extrapolated volition scheme to satisfy individual preferences must be limited.
Depending on the size of those differences, this may or may not be a big deal. And we’re very likely to have fundamental social goals that do include external imperatives, although masochism isn’t one.
Mea culpa; I seem to have overgeneralized the extrapolation process.
But unless all its flaws are context-independent and uniformly distributed across humanity, I suspect they’d make human volition less likely to cohere, not more.
Here’s a quote from the CEV paper:
If Archimedes (or a random person) could live for a thousand years (together with the rest of humanity), could think a billion times faster, could learn all the FAI knows, etc, etc, then they’d very likely arrive at the same answer as a modern person. So it shouldn’t matter which person/people you pick. This is how the extrapolation is supposed to work.
First, this seems implausible (people really do have different desires and personalities), and second, if plausible, the starting location doesn’t seem to matter. If you take Archimedes and a brain-damaged murderer and a chimpanzee and end up with similar outputs, what did you need the inputs for? They don’t appear to have made much difference. If the answer is that you’ll modify the chimp and the murderer to become like Archimedes, then why don’t you just copy Archimedes directly instead of doing it by scrubbing out those individuals and then planting Archimedes inside?
Until CEV has a plan for dealing with envy, it strikes me as underplanned. Real humans interfere with each other- that’s part of what makes life human.
The CEV of chimpanzees would not be the same as the CEV of humans.
What sort of differences are we looking at, here?
The CEV of humans, if it exists, would depend in part on those things all human minds have in common. Chimpanzees have been endowed with a different sort of mind — and, I expect, different values.
So, what all humans have in common is that their most dangerous enemy are other humans. I don’t see how that’s getting us anywhere good.
[edit]I forgot to add- I’m looking for specificity here. What do humans care about that chimps don’t? If chimps thought ten thousand times faster, would they care about those things?
I will clarify: My second link, “Thou Art Godshatter”, says that our values are causally rooted in evolution, which gave us our brains. CEV is supposed to extrapolate that chain of causality, in a certain way. My first link, “The Psychological Unity Of Humankind”, says that human brains are similar to each other. Chimpanzee brains are unlike human brains. There is no reason to expect human universal values to be identical to chimpanzee universal values. There is no reason to expect these species’ extrapolated values to be identical. That is what i am saying. Are you skeptical of this argument?
Art, for example.
The question you want to ask is, Would a CEV-extrapolated chimpanzee care about art? I don’t see why it should.
Sure there is- “maximize inclusive genetic fitness.”
Are you familiar with the theory that the human brain is essentially a peacock’s tail, and that language, creative endeavors, and so on were evolved primarily to attract mating partners?
It’s possible that a chimp that gets a lot smarter won’t start having more intellectual tastes, but the basic problem of “who do I have sex with, and how do I get them to agree?” is still around and it seems like they would try very similar methods to what we have.
That is an anthropomorphic representation of the ‘values’ of a gene allele. It is not the value of actual humans or chimpanzees.
Ah, it seems we have different ideas about what human values actually are.
Organisms are adaptation-executers, not fitness-maximizers. Art is a human terminal value. Maximizing inclusive genetic fitness is not. Even if the causal reason for us having this value were that artsy people get more sex, we would value art for its own sake.
I highly recommend the wiki article on the complexity of value and the articles linked from it.
Wait wait wait. Are we talking about human values, or human universal values? Those seem to me to be different concepts. It seems to me that what we have in common is that we’re competing with one another and what differs between humans is how we compete and what we seek to maximize.
I think the difference between my approach to music and a musician’s approach to music is more than a difference of degree, and so am reluctant to say we share the same value. Given the many differing attitudes towards justice, it seems more apt to call it a political ploy about precedents than one objective standard all humans strive towards. I could keep going, but hopefully two examples is sufficient. Human values seem to be individual values, first and foremost.
And when we remove some human values- predation, cruelty, rape, domination- from the list because their time has passed, it is not clear to me why the others remain. If we alter ourselves so we no longer understand events as narratives (since that’s a motherlode of bias right there), will literature be a casualty of that improvement?
Agreed that individual humans have differing values. However I believe there’s more to say about human universal values than “we’re competing with one another”. Here are two pieces of evidence:
(1) Almost all human cultures have what we recognize as art. So far as I know, chimpanzee cultures do not. (Although a quick google search tells me that chimps who live in human cultures do create art.) So art, broadly defined, is universal; but individual humans have more specific tastes. I expect humans and chimpanzees have senses of justice (differing from individual to individual).
(2) Individuals can change their values through moral arguments and education. If I try, through moral argument, to convince someone of my idea of justice, it’s because I feel that my idea of justice has at least a somewhat universal appeal. And, with sufficient resources, you could become a musician and gain a richer appreciation of music. One could imagine an extrapolation process under which individuals’ values converge, at least some of the time.
However:
I share the concern that I think you’re raising here: For all we know, moral progress might not be pointing towards any particular destination. More research needs to be done on the mechanics of morality.
Mostly agreed.
This I’m not as sure about. I can become competent at playing an instrument and better at interpreting music, but I’m not sure I can rewrite my talents. If those are determined by a relatively inelastic part of my brain’s configuration, then it seems likely that the pathway to becoming a Beethoven simply does not exist for me.
A better example might be heroic compassion- people who put themselves at risk without thinking to save others. The consensus opinion is it’s probably at least somewhat genetic, and you either are a hero (in that narrow sense) or you aren’t. I could be roused to tears by tales of heroic compassion but not have the unthinking impulse to do it, and it’s not clear that if I don’t have it I could acquire it. There are other things that people do acquire- bodyguards learn how to operate after being shot, and soldiers learn how to stay active in combat situations- so it might be possible. But I think bounded potentials are more realistic than unbounded potentials.
Aumann’s agreement theorem seems to imply that individual extrapolated volition assignments must agree on statements of fact: the setup implies that they’re simulated as perfect reasoners and share a knowledge pool, and the extrapolation process provides for an unbounded number of Bayesian updates. So we can expect extrapolated volition to cohere exactly to the extent that it’s based on common fundamental goals: not immediate desires and not possibly-fallible philosophical results, but the low-level affective assignments that lead us to think of those higher-level results as desirable or undesirable.
To what extent do common fundamental goals drive our moral reasoning? I don’t know, but individual differences do exist (the existence of masochistic people should prove that), and if they’re large enough then CEV may end up looking incomplete or unpleasantly compromise-driven.
But is that relevant to the question that CEV tries to answer? As far as I know, most masochistic people don’t also hold a belief that everybody should be masochistic.
Even if individual differences in fundamental goals are not extended to other people as imperatives, they imply that the ability of a coherent extrapolated volition scheme to satisfy individual preferences must be limited.
Depending on the size of those differences, this may or may not be a big deal. And we’re very likely to have fundamental social goals that do include external imperatives, although masochism isn’t one.
That would make them throughly non-human in psychology. It’s a possibly useful take on CEV but I’m not sure it’s a standard one.
Mea culpa; I seem to have overgeneralized the extrapolation process.
But unless all its flaws are context-independent and uniformly distributed across humanity, I suspect they’d make human volition less likely to cohere, not more.