There is no reason to think that this is because they are less intelligent or less well-informed than me, as opposed to actually having different preferences.
There are lots of reasons to think so. For example, they might want the death of your friends because they mistakenly believe that a deity exists.
Or for any number of other, non-religious reasons. And it could well be that extrapolating those people’s preferences would lead, not to them rejecting their beliefs, but to them wishing to bring their god into existence.
Either people have fundamentally different, irreconcilable, values or they don’t. If they do, then the argument I made is valid. If they don’t, then CEV(any random person) will give exactly the same result as CEV(humanity).
That means that either calculating CEV(humanity) is an unnecessary inefficiency, or CEV(humanity) will do nothing at all, or CEV(humanity) would lead to a world that is intolerable for at least some minority of people. I actually doubt that any of the people from the SI would disagree with that (remember the torture vs flyspecks argument).
That may be considered a reasonable tradeoff by the developers of an “F”AI, but it gives those minority groups to whom the post-AI world would be inimical equally rational reasons to oppose such a development.
As someone who does not believe in moral realism, I agree that CEV over all humans who ever lived (excluding sociopaths and such) will not output anything.
But I think that a moral realist should believe that CEV will output some value system, and that the produced value system will be right.
Edit2: It appears that copying and pasting from some places includes “http” even when my browser address doesn’t. But I did something wrong when copying from the philosophy dictionary.
I agree—assuming that CEV didn’t impose a majority view on a minority. My understanding of the SI’s arguments (and it’s only my understanding) is that they believe it will impose a majority view on a minority, but that they think that would be the right thing to do—that if the choice were beween 3^^^3 people getting a dustspeck in the eye or one person getting tortured for fifty years, the FAI would always make a choice, and that choice would be for the torture rather than the dustspecks.
Now, this may well be, overall, the rational choice to make as far as humanity as a whole goes, but it would most definitely not be the rational choice for the person who was getting tortured to support it.
And since, as far as I can see, most people only value a very small subset of humanity who identify as belonging to the same groups as them, I strongly suspect that in the utilitarian calculations of a “friendly” AI programmed with CEV, they would end up in the getting-tortured group, rather than the avoiding-dustspecks one.
that if the choice were beween 3^^^3 people getting a dustspeck in the eye or one person getting tortured for fifty years, the FAI would always make a choice, and that choice would be for the torture rather than the dustspecks
That is an entirely separate issue. If CEV(everyone) outputted a moral theory that held utility was additive, then the AI implementing it would choose torture over specks. In other words, utilitarians are committed to believing that specks is the wrong choice.
But there is no guarantee that CEV will output a utilitarian theory, even if you believe it will output something. SI (Eliezer, at least) believes CEV will output a utilitarian theory because SI believes utilitarian theories are right. But everyone agrees that “whether CEV will output something” is a different issue than “what CEV will output.”
Personally, I suspect CEV(everyone in the United States) would output something deotological—and might even output something that would pick specks. Again, assuming it outputs anything.
Either people have fundamentally different, irreconcilable, values or they don’t. If they do, then the argument I made is valid. If they don’t, then CEV(any random person) will give exactly the same result as CEV(humanity).
This is a false dilemma. If people have some values that are the same or reconcilable, then you will get different output from CEV(any random person) and CEV(humanity).
And note that an actual move by virtue ethicists is to exclude sociopaths from “humanity”.
There are lots of reasons to think so. For example, they might want the death of your friends because they mistakenly believe that a deity exists.
Or for any number of other, non-religious reasons. And it could well be that extrapolating those people’s preferences would lead, not to them rejecting their beliefs, but to them wishing to bring their god into existence.
Either people have fundamentally different, irreconcilable, values or they don’t. If they do, then the argument I made is valid. If they don’t, then CEV(any random person) will give exactly the same result as CEV(humanity).
That means that either calculating CEV(humanity) is an unnecessary inefficiency, or CEV(humanity) will do nothing at all, or CEV(humanity) would lead to a world that is intolerable for at least some minority of people. I actually doubt that any of the people from the SI would disagree with that (remember the torture vs flyspecks argument).
That may be considered a reasonable tradeoff by the developers of an “F”AI, but it gives those minority groups to whom the post-AI world would be inimical equally rational reasons to oppose such a development.
As someone who does not believe in moral realism, I agree that CEV over all humans who ever lived (excluding sociopaths and such) will not output anything.
But I think that a moral realist should believe that CEV will output some value system, and that the produced value system will be right.
In short, I think one’s belief about whether CEV will output something is isomorphic on whether one believes in [moral realism] (plato.stanford.edu/entries/moral-realism/).
Edit: link didn’t work, so separated it out.
Have you tried putting
http://in front of the URL?(Edit: the backtick thing to show verbatim code isn’t working properly for some reason, but you know what I mean.)
moral realism.
Edit: Apparently that was the problem. Thanks.
Edit2: It appears that copying and pasting from some places includes “http” even when my browser address doesn’t. But I did something wrong when copying from the philosophy dictionary.
I agree—assuming that CEV didn’t impose a majority view on a minority. My understanding of the SI’s arguments (and it’s only my understanding) is that they believe it will impose a majority view on a minority, but that they think that would be the right thing to do—that if the choice were beween 3^^^3 people getting a dustspeck in the eye or one person getting tortured for fifty years, the FAI would always make a choice, and that choice would be for the torture rather than the dustspecks.
Now, this may well be, overall, the rational choice to make as far as humanity as a whole goes, but it would most definitely not be the rational choice for the person who was getting tortured to support it.
And since, as far as I can see, most people only value a very small subset of humanity who identify as belonging to the same groups as them, I strongly suspect that in the utilitarian calculations of a “friendly” AI programmed with CEV, they would end up in the getting-tortured group, rather than the avoiding-dustspecks one.
This is not clear.
That is an entirely separate issue. If CEV(everyone) outputted a moral theory that held utility was additive, then the AI implementing it would choose torture over specks. In other words, utilitarians are committed to believing that specks is the wrong choice.
But there is no guarantee that CEV will output a utilitarian theory, even if you believe it will output something. SI (Eliezer, at least) believes CEV will output a utilitarian theory because SI believes utilitarian theories are right. But everyone agrees that “whether CEV will output something” is a different issue than “what CEV will output.”
Personally, I suspect CEV(everyone in the United States) would output something deotological—and might even output something that would pick specks. Again, assuming it outputs anything.
This is a false dilemma. If people have some values that are the same or reconcilable, then you will get different output from CEV(any random person) and CEV(humanity).
And note that an actual move by virtue ethicists is to exclude sociopaths from “humanity”.