Change in values of the future agents, however sudden of gradual, means that the Future (the whole freackin’ Future!) won’t be optimized according to our values, won’t be anywhere as good as it could’ve been otherwise.
That really depends of what you mean by “our values”:
1) The values of modern, western, educated humans? (as opposed to those of the ancient Greek, or of Confucius, or of medieval Islam), or
2) The “core” human values common to all human civilizations so far? (“stabbing someone who just saved your life is a bit of a dick move”, “It would be a shame if humanity was exterminated in order to pave the universe with paperclips”, etc.)
Both of those are quite fuzzy and I would find it hard to describe either of them precisely enough that even a computer could understand them.
When Eliezer talks of Friendly AI having human value, I think he’s mostly talking about the second set (in The Psychological Unity of Mankind. But when Ben or Robin talk about how it isn’t such a big deal if values change, because they’ve already changed in the past, they seem to be referring to the first kind of value.
I would agree with Ben and Robin that it isn’t a big deal if our descendents (or Ems or AIs) have values that are at odds with our current, western, values (because they might be “wrong”, some might be instrumental values we confuse for terminal values, etc.); but I wouldn’t extend that to changes in “fundamental human values”.
So I don’t think “Ben and Robin are OK with a future without our values” is a good way of phrasing it. The question is more whether there is such a thing as fundamental human values (or is everything cultural?), whether it’s easy to hit those in mind-space, etc.
I find this distinction useful. According to the OP, I’d be considered a proponent of values-deathism proper, but only in terms of the values you place in the first set; I consider the exploration of values-space to be one of the values in the second set, and a significant part of my objection to the idea of tiling the universe with paperclips is that it would stop that process.
That really depends of what you mean by “our values”
Your values is at least something that on reflection you’d be glad happened, which doesn’t apply to acting on human explicit beliefs that are often imprecise or wrong. More generally, any heuristic for good decisions you know doesn’t qualify. “Don’t kill people” doesn’t qualify. Values are a single criterion that doesn’t tolerate exceptions and status quo assumptions. See magical categories for further discussion.
I read it as “our ancestors” meaning “the ancient Greeks”, not, “early primates” but I may be wrong
In a certain sense “primordial single celled replicator” may be an even more relevant comparison than either. Left free to deviate Nash would weed out those pesky ‘general primate values’.
I don’t think 2 accurately reflects Eliezer’s Preservation target. CEV doesn’t ensure beforehand that any of those core values aren’t thrown out. What’s important is the process by which we decide to adopt or reject values, how that process changes when we learn more, and things like that.
That is also one thing that could now change as the direct result of choices we make, through brain modification, or genetic engineering, or AI’s with whole new value-adoption systems. Our intuition tends to treat this as stable even when we know we’re dealing with ‘different’ cultures.
That really depends of what you mean by “our values”:
1) The values of modern, western, educated humans? (as opposed to those of the ancient Greek, or of Confucius, or of medieval Islam), or
2) The “core” human values common to all human civilizations so far? (“stabbing someone who just saved your life is a bit of a dick move”, “It would be a shame if humanity was exterminated in order to pave the universe with paperclips”, etc.)
Both of those are quite fuzzy and I would find it hard to describe either of them precisely enough that even a computer could understand them.
When Eliezer talks of Friendly AI having human value, I think he’s mostly talking about the second set (in The Psychological Unity of Mankind. But when Ben or Robin talk about how it isn’t such a big deal if values change, because they’ve already changed in the past, they seem to be referring to the first kind of value.
I would agree with Ben and Robin that it isn’t a big deal if our descendents (or Ems or AIs) have values that are at odds with our current, western, values (because they might be “wrong”, some might be instrumental values we confuse for terminal values, etc.); but I wouldn’t extend that to changes in “fundamental human values”.
So I don’t think “Ben and Robin are OK with a future without our values” is a good way of phrasing it. The question is more whether there is such a thing as fundamental human values (or is everything cultural?), whether it’s easy to hit those in mind-space, etc.
Counterpoints: The Psychological Diversity of Mankind, Human values differ as much as values can differ.
I find this distinction useful. According to the OP, I’d be considered a proponent of values-deathism proper, but only in terms of the values you place in the first set; I consider the exploration of values-space to be one of the values in the second set, and a significant part of my objection to the idea of tiling the universe with paperclips is that it would stop that process.
Your values is at least something that on reflection you’d be glad happened, which doesn’t apply to acting on human explicit beliefs that are often imprecise or wrong. More generally, any heuristic for good decisions you know doesn’t qualify. “Don’t kill people” doesn’t qualify. Values are a single criterion that doesn’t tolerate exceptions and status quo assumptions. See magical categories for further discussion.
But that may not be what Ben implied when saying
(I read it as “our ancestors” meaning “the ancient Greeks”, not, “early primates” but I may be wrong)
In a certain sense “primordial single celled replicator” may be an even more relevant comparison than either. Left free to deviate Nash would weed out those pesky ‘general primate values’.
Spelling notice (bold added):
Fixed, thanks.
I don’t think 2 accurately reflects Eliezer’s Preservation target. CEV doesn’t ensure beforehand that any of those core values aren’t thrown out. What’s important is the process by which we decide to adopt or reject values, how that process changes when we learn more, and things like that.
That is also one thing that could now change as the direct result of choices we make, through brain modification, or genetic engineering, or AI’s with whole new value-adoption systems. Our intuition tends to treat this as stable even when we know we’re dealing with ‘different’ cultures.