Phil, you make a good point here. However, note that when the people you are talking about are extrapolated in a CEV-like FAI, they will also understand this point. Elizabeth Taylor will understand that not everyone can have high status, and therefore people will have to settle on some solution, which could involve “fake people” as low status entities, editing of everyone’s top-level goals (a sort of decision-theoretic compromise), etc.
One corollary of this is that for existing high status people, CEV would be a terrible thing. [EDIT: I was thinking “Compared to nonegalitarian AGI”; realistically that is the other option, not business as usual]
Since existing high-status people’s cooperation may be required to build a CEV-like AI, a weighted-influence CEV might be built in stead, so that every non-positional goal can be optimized to the full extent for everyone, but for every positional goal, the existing ranking would be preserved.
One corollary of this is that for existing high status people, CEV would be a terrible thing.
(As compared to what?) Only assuming that the benefits of living in a post-Singularity world are less valuable than making other people miserable, which strikes me as implausible. A middle-class person today likely has way better life than a king of 4000BC.
What do you mean “for someone who highly values status”? Are you that someone who prefers to have been born a king of 4000BC to comforts of modern world? Do you think people who would profess this verbal preference do so because it’s their actual preference?
As evidence for someone like this, consider dictators like Kim Jong Il. Opening up North Korea would result in much greater wealth for both him and his people, but it comes with a loss of power and status for Kim Jong. No one thinks he’s opening those borders anytime soon. The comparison isn’t as drastic, however—Kim Jong’s comforts are probably only a decade or two behind modern (I’m speculating).
Kim Jong’s comforts are probably only a decade or two behind modern (I’m speculating).
His likes are idiosyncratic, but as far as they go, he’s cutting-edge.
Cognac is routinely cited as one of the top illicit imports, and I don’t think he’s getting bad cognac; one of his principal interests is/was movies, of which he has a 20,000-strong collection—world-class, I think—and he was infamous for kidnapping ‘the famous South Korean movie director Shin Sang Ok and his ex-wife, actress Che Eun Hui, and kept them for eight years while making them produce propaganda films’, which is something which is inaccessible to just about everyone, modern or no.
He may be an evil dictator, but in my opinion he gets extreme bonus points for style:
and he was infamous for kidnapping ‘the famous South Korean movie director Shin Sang Ok and his ex-wife, actress Che Eun Hui, and kept them for eight years while making them produce propaganda films’
And from wikipedia:
From 1983 Shin directed seven films with Kim Jong-il acting as an executive producer. The best known of these films is Pulgasari, a giant-monster film similar to the Japanese Godzilla.
Not even the Bond badguys did anything that amusing.
Hmm, Kim Jong Il is apparently a bad example since he’s so wealthy. Surely there are dictators who don’t have the resources that Kim Jong has (such that they’re living in sub-modern conditions), but they still want to hold on to the power and status they hold despite the potential for wealth. right? (again, speculating, no hard evidence in mind)
Just about any nation-size dictator will be able to scratch up enough cash to live like a millionaire. Extort a few dollars from a million destitute inhabitants and you’re talking real money. So you have to look at city or tribal scale units, and even then, I think most chieftains are happier to be in power than out of power in a wealthier nation. How many Afghanistani elders are cooperating with the US, and out of enlightened self-interest, knowing that in any modernized industrial society their clans will be hopelessly obsolete? How many out of naked fear of the Taliban or US, and bribes?
In How the Mind Works, Steven Pinker has an excellent discussion of Schelling’s work on game theory, and argues that, per Schelling’s work, the appearance of being a rational individual can actually be a liability for a rogue dictator, so they have an incentive to look kooky.
Good point. However, why would the dictator put on the charade and try to keep his status/power unless he valued it more than the wealth he could obtain by opening the country up? If the gains are small, this is probably a good margin to look irrational on, but if the gains are large enough, opening up outweighs the irrational act (on this margin). There are plenty of other things to appear irrational about with lower stakes. You don’t have to appear kooky about every single decision you make in order to convince others that you are kooky—just enough of them.
So in a nutshell, if the difference in standards of living for the dictator under the two scenarios are large enough, the irrationality ploy shouldn’t matter (much).
Opening up North Korea would result in much greater wealth for both him and his people, but it comes with a loss of power and status for Kim Jong.
I find the premise of Kim Jong-il sharing the poor standard of living with his people (or, not making the most of what the modern world has to offer because of living in his country), completely implausible.
I think that there are probably people for whom the ability to boss people around, kill others with impunity, have a harem of women, etc, is worth more than a shower and flushing loo.
Hmm, I moved some towards agreement on this one. Though the particular argument you use doesn’t apply to post-Singularity lower bound benefits. For a start, add immortality and much deeper insight into all things.
If there’s a universal, it’s that people enjoy gaining deeper insight—they value the first derivative of insight. Actually having insight can be a drag.
(Whatever, this is a technicality not relevant to the argument.) I doubt having insight is a downside in itself, only perhaps in as much as it makes it no longer possible to gain that insight without also losing it first; and beside the gaining of insight, there are lots of other things people value.
Having an insight can be a downside if the insight disrupts your worldview, or makes you face an unpleasant truth. There is no law saying that truth and happiness are always allies.
If I had to make a wild guess, I might guess that 75% of people in the modern world would say they would rather have been a king in 4000BC. (More, if you exclude the people who say they would rather have been a farmer in 4000BC than a king in 4000BC.) My 50% confidence interval is 25%-95%. Anybody want to do a survey?
I would also guess the number who say they would rather be a king is smaller than the number of people who would actually prefer being a king, because people overestimate how much they would miss modern conveniences, and because saying you’d like to be king is frowned on nowadays.
It’s interesting to look at what traits people assume they’d carry into the past. I suspect that gender is one of them. I don’t have a strong feeling for what proportion would like to be a queen in the ancient world.
In discussions I’ve seen about going back, a fair number say they’d be dead because of the lack of modern medicine.
Correct me if I’m wrong—I’m pretty new to this game. Does this entail that you’d assign about a 50% probability that either 0%-25% or 95%+ of the people would say that?
So, based on those numbers, you think it’s more likely that either 0%-25% or 95%+ of the people would say that, than that 35%-85% would say that? (assuming nonzero probability to 25%-35% or 85%-95%)
Based on those number, yes. But I didn’t consider both sides like that. I may have erred in overcompensating for the tendency of people to make too-small confidence intervals.
If I had to make a wild guess, I might guess that 75% of people in the modern world would say they would rather have been a king in 4000BC.
I considered well-off middle-class people in the modern world, which isn’t such a big portion of population of the modern world. Of course, for a person in poverty, becoming a king of the savages is probably an improvement. (Very likely, not what you meant.)
I would also guess the number who say they would rather be a king is smaller than the number of people who would actually prefer being a king, because people overestimate how much they would miss modern conveniences, and because saying you’d like to be king is frowned on nowadays.
I agree that these factors are present, but am not sure that they outweigh the factors prompting people to bias their beliefs in the opposite direction (or even that these are the main factors in the direction you indicate).
You’re saying that you don’t believe there are people for whom status is a dominant value.
Yes. This would fall into the category of “people” who don’t share a human universal balance of basic aspects of value, “people” who are magical mutants with complex motivations shaped primarily by something other than evolution (which won’t allow large complex differences from the rest of the population). (See also this comment.)
I’m not as much interested in the difficulty of satisfying positional values, as in developing a typology of values, noting that positional / non-positional is a really big difference between values, and concluding that reconciling all human values is not much easier than reconciling all possible values.
Since existing high-status people’s cooperation may be required to build a CEV-like AI
Yes, this is a concern I’ve had: a CEV-valuing FAI would also have to be a very good politician to have the social support necessary to keep the project from being shut down. But good politicians are a danger in and of themselves.
The other possibility is that someone builds CEV before the existing elites get to a high enough future-shock level to even pay attention. Dictators will wake up one morning to find that the FAI has just robbed them of the only thing that makes anyone care about them: their power over other people. And furthermore, there will be a very large number of people whose volitions say something like “I want to punish [Dictator X] because he killed my family”.
Phil, you make a good point here. However, note that when the people you are talking about are extrapolated in a CEV-like FAI, they will also understand this point. Elizabeth Taylor will understand that not everyone can have high status, and therefore people will have to settle on some solution, which could involve “fake people” as low status entities, editing of everyone’s top-level goals (a sort of decision-theoretic compromise), etc.
One corollary of this is that for existing high status people, CEV would be a terrible thing. [EDIT: I was thinking “Compared to nonegalitarian AGI”; realistically that is the other option, not business as usual]
Since existing high-status people’s cooperation may be required to build a CEV-like AI, a weighted-influence CEV might be built in stead, so that every non-positional goal can be optimized to the full extent for everyone, but for every positional goal, the existing ranking would be preserved.
(As compared to what?) Only assuming that the benefits of living in a post-Singularity world are less valuable than making other people miserable, which strikes me as implausible. A middle-class person today likely has way better life than a king of 4000BC.
For someone who highly values status, I disagree with that statement.
What do you mean “for someone who highly values status”? Are you that someone who prefers to have been born a king of 4000BC to comforts of modern world? Do you think people who would profess this verbal preference do so because it’s their actual preference?
As evidence for someone like this, consider dictators like Kim Jong Il. Opening up North Korea would result in much greater wealth for both him and his people, but it comes with a loss of power and status for Kim Jong. No one thinks he’s opening those borders anytime soon. The comparison isn’t as drastic, however—Kim Jong’s comforts are probably only a decade or two behind modern (I’m speculating).
His likes are idiosyncratic, but as far as they go, he’s cutting-edge.
Cognac is routinely cited as one of the top illicit imports, and I don’t think he’s getting bad cognac; one of his principal interests is/was movies, of which he has a 20,000-strong collection—world-class, I think—and he was infamous for kidnapping ‘the famous South Korean movie director Shin Sang Ok and his ex-wife, actress Che Eun Hui, and kept them for eight years while making them produce propaganda films’, which is something which is inaccessible to just about everyone, modern or no.
He may be an evil dictator, but in my opinion he gets extreme bonus points for style:
And from wikipedia:
Not even the Bond badguys did anything that amusing.
Hmm, Kim Jong Il is apparently a bad example since he’s so wealthy. Surely there are dictators who don’t have the resources that Kim Jong has (such that they’re living in sub-modern conditions), but they still want to hold on to the power and status they hold despite the potential for wealth. right? (again, speculating, no hard evidence in mind)
Just about any nation-size dictator will be able to scratch up enough cash to live like a millionaire. Extort a few dollars from a million destitute inhabitants and you’re talking real money. So you have to look at city or tribal scale units, and even then, I think most chieftains are happier to be in power than out of power in a wealthier nation. How many Afghanistani elders are cooperating with the US, and out of enlightened self-interest, knowing that in any modernized industrial society their clans will be hopelessly obsolete? How many out of naked fear of the Taliban or US, and bribes?
In How the Mind Works, Steven Pinker has an excellent discussion of Schelling’s work on game theory, and argues that, per Schelling’s work, the appearance of being a rational individual can actually be a liability for a rogue dictator, so they have an incentive to look kooky.
Kim Jong Il is playing it by the book.
Good point. However, why would the dictator put on the charade and try to keep his status/power unless he valued it more than the wealth he could obtain by opening the country up? If the gains are small, this is probably a good margin to look irrational on, but if the gains are large enough, opening up outweighs the irrational act (on this margin). There are plenty of other things to appear irrational about with lower stakes. You don’t have to appear kooky about every single decision you make in order to convince others that you are kooky—just enough of them.
So in a nutshell, if the difference in standards of living for the dictator under the two scenarios are large enough, the irrationality ploy shouldn’t matter (much).
I find the premise of Kim Jong-il sharing the poor standard of living with his people (or, not making the most of what the modern world has to offer because of living in his country), completely implausible.
see my reply here
I think that there are probably people for whom the ability to boss people around, kill others with impunity, have a harem of women, etc, is worth more than a shower and flushing loo.
And modern medicine?
All such questions are tests of the imagination, really.
The harem bit looses some of its appeal when you think about the standards of dental care and how rarely people used to bathe.
Hmm, I moved some towards agreement on this one. Though the particular argument you use doesn’t apply to post-Singularity lower bound benefits. For a start, add immortality and much deeper insight into all things.
If there’s a universal, it’s that people enjoy gaining deeper insight—they value the first derivative of insight. Actually having insight can be a drag.
(Whatever, this is a technicality not relevant to the argument.) I doubt having insight is a downside in itself, only perhaps in as much as it makes it no longer possible to gain that insight without also losing it first; and beside the gaining of insight, there are lots of other things people value.
Having an insight can be a downside if the insight disrupts your worldview, or makes you face an unpleasant truth. There is no law saying that truth and happiness are always allies.
If I had to make a wild guess, I might guess that 75% of people in the modern world would say they would rather have been a king in 4000BC. (More, if you exclude the people who say they would rather have been a farmer in 4000BC than a king in 4000BC.) My 50% confidence interval is 25%-95%. Anybody want to do a survey?
I would also guess the number who say they would rather be a king is smaller than the number of people who would actually prefer being a king, because people overestimate how much they would miss modern conveniences, and because saying you’d like to be king is frowned on nowadays.
It’s interesting to look at what traits people assume they’d carry into the past. I suspect that gender is one of them. I don’t have a strong feeling for what proportion would like to be a queen in the ancient world.
In discussions I’ve seen about going back, a fair number say they’d be dead because of the lack of modern medicine.
About half on the most recent such discussion I recall reading.
Correct me if I’m wrong—I’m pretty new to this game. Does this entail that you’d assign about a 50% probability that either 0%-25% or 95%+ of the people would say that?
That’s an implication, yes.
So, based on those numbers, you think it’s more likely that either 0%-25% or 95%+ of the people would say that, than that 35%-85% would say that? (assuming nonzero probability to 25%-35% or 85%-95%)
Based on those number, yes. But I didn’t consider both sides like that. I may have erred in overcompensating for the tendency of people to make too-small confidence intervals.
Ah, good. I was afraid I’d misunderstood.
I considered well-off middle-class people in the modern world, which isn’t such a big portion of population of the modern world. Of course, for a person in poverty, becoming a king of the savages is probably an improvement. (Very likely, not what you meant.)
I agree that these factors are present, but am not sure that they outweigh the factors prompting people to bias their beliefs in the opposite direction (or even that these are the main factors in the direction you indicate).
You’re saying that you don’t believe there are people for whom status is a value.
Not at all. I’m merely saying that status is not in total dominance of what people prefer.
Okay. You’re saying that you don’t believe there are people for whom status is a dominant value.
Yes. This would fall into the category of “people” who don’t share a human universal balance of basic aspects of value, “people” who are magical mutants with complex motivations shaped primarily by something other than evolution (which won’t allow large complex differences from the rest of the population). (See also this comment.)
Perhaps they would prefer to be weighted equally with low status people in a CEV to dying at the end of their natural lifespans.
I’m not as much interested in the difficulty of satisfying positional values, as in developing a typology of values, noting that positional / non-positional is a really big difference between values, and concluding that reconciling all human values is not much easier than reconciling all possible values.
I disagree with the connotation of this conclusion, and the word “reconcile” is too vague for it to have a precise denotation.
Clearly, human verbalized preferences contradict each other. In terms of “formal preference” I suspect that the same is true.
Yes, this is a concern I’ve had: a CEV-valuing FAI would also have to be a very good politician to have the social support necessary to keep the project from being shut down. But good politicians are a danger in and of themselves.
The other possibility is that someone builds CEV before the existing elites get to a high enough future-shock level to even pay attention. Dictators will wake up one morning to find that the FAI has just robbed them of the only thing that makes anyone care about them: their power over other people. And furthermore, there will be a very large number of people whose volitions say something like “I want to punish [Dictator X] because he killed my family”.
Once an AGI is out, you won’t be able to shut it down. The Internet makes it pretty much impossible, even if the AGI is not superintelligent.