I’m not saying that you didn’t express yourself precisely enough. I am saying that there is no such thing as “best (full stop)” There is “best for me”, there is “best for you”, but there is not “best for both of us”. No more than there is an objective (or intersubjective) probability that I am wearing a red shirt as I type.
Your argument above only works if “best” is interpreted as “best for every mind”. If that is what you meant, then your implicit definition of FAI proves that FAI is impossible.
Perhaps you should explain, by providing a link, what is meant by CEV. The only text I know of describing it is dated 2004, and, … how shall I put this …, it doesn’t seem to cohere.
But, I have to say, based on what I can infer, that I see no reason to expect coherence, and the concept of “extrapolation” scares the sh.t out of me.
“Coherence” seems a bit like the human genome project. Yes there are many individual differences—but if you throw them all away, you are still left with something.
So we are going to build a giant AI to help us discover and distill that residue of humanity which is there after you discard the differences?
And here I thought that was the easy part, the part we had already figured out pretty well by ourselves.
And I’m not sure I care for the metaphor of “throwing away” the differences. Shouldn’t we instead be looking for practices and mechanisms that make use of those differences, that weave them into a fabric of resilience and mutual support rather than a hodgepodge of weakness and conflict?
“We”? You mean: you and me, baby? Or are you asking after a prediction about whether something like CEV will beat the other philosophies about what to do with an intelligent machine?
CEV is an alien document from my perspective. It isn’t like anything I would ever write.
It reminds me a bit of the ideal of democracy—where the masses have a say in running things.
I tend to see the world as more run by the government and its corporations—with democracy acting like a smokescreen for the voters—to give them an illusion of control, and to prevent them from revolting.
Also, technology has a long history of increasing wealth inequality—by giving the powerful controllers and developers of the technology ever more means of tracking and controlling those who would take away their stuff.
That sort of vision is not so useful as an election promise to help rally the masses around a cause—but then, I am not really a politician.
with democracy acting like a smokescreen for the voters—to give them an illusion of control, and to prevent them from revolting.
Voting prevents revolts in the same sense that a hydroelectric dam prevents floods. It’s not a matter of stopping up the revolutionary urge; in fact, any attempt to do so would be disastrous sooner or later. Instead it provides a safe, easy channel, and in the process, captures all the power of the movement before that flow can build up enough to cause damage.
The voters can have whatever they want, and the rest of the system does it’s best to stop them from wanting anything dangerous.
It wouldn’t form a utility function at all. It has no answer for any of the interesting or important questions: the questions on which there is disagreement. Or am I missing something here?
Ok, you are changing the analogy. Initially you said, throw away the differences. Now you are saying throw away all but one of them.
So our revised approximation of the CEV is the expressed volition of … Craig Venter?!
Would that horrify the vast majority of humanity? I think it might. Mostly because people just would not know how it would play out. People generally prefer the devil they know to the one they don’t.
Well, it was I who wrote that. The differences were thrown away in the genome project—but that isn’t exactly the corresponding thing according to the CEV proposal.
A certain lack of coherence doesn’t mean all the conflicting desires cancel out leaving nothing behind—thus the emphasis on still being “left with something”.
The phrase “the best of all possible worlds” ought to be the canonical example of the Mind Projection Fallacy.
It would be unreasonably burdensome to append “with respect to a given mind” to every statement that involves subjectivity in any way.
ETA: For comparison, imagine if you had to say “with respect to a given reference frame” every time you talked about velocity.
I’m not saying that you didn’t express yourself precisely enough. I am saying that there is no such thing as “best (full stop)” There is “best for me”, there is “best for you”, but there is not “best for both of us”. No more than there is an objective (or intersubjective) probability that I am wearing a red shirt as I type.
Your argument above only works if “best” is interpreted as “best for every mind”. If that is what you meant, then your implicit definition of FAI proves that FAI is impossible.
ETA: What given frame do you have in mind??????
The usual assumption in this context would be CEV. Are you saying you strongly expect humanity’s extrapolated volition not to cohere?
Perhaps you should explain, by providing a link, what is meant by CEV. The only text I know of describing it is dated 2004, and, … how shall I put this …, it doesn’t seem to cohere.
But, I have to say, based on what I can infer, that I see no reason to expect coherence, and the concept of “extrapolation” scares the sh.t out of me.
“Coherence” seems a bit like the human genome project. Yes there are many individual differences—but if you throw them all away, you are still left with something.
So we are going to build a giant AI to help us discover and distill that residue of humanity which is there after you discard the differences?
And here I thought that was the easy part, the part we had already figured out pretty well by ourselves.
And I’m not sure I care for the metaphor of “throwing away” the differences. Shouldn’t we instead be looking for practices and mechanisms that make use of those differences, that weave them into a fabric of resilience and mutual support rather than a hodgepodge of weakness and conflict?
“We”? You mean: you and me, baby? Or are you asking after a prediction about whether something like CEV will beat the other philosophies about what to do with an intelligent machine?
CEV is an alien document from my perspective. It isn’t like anything I would ever write.
It reminds me a bit of the ideal of democracy—where the masses have a say in running things.
I tend to see the world as more run by the government and its corporations—with democracy acting like a smokescreen for the voters—to give them an illusion of control, and to prevent them from revolting.
Also, technology has a long history of increasing wealth inequality—by giving the powerful controllers and developers of the technology ever more means of tracking and controlling those who would take away their stuff.
That sort of vision is not so useful as an election promise to help rally the masses around a cause—but then, I am not really a politician.
Voting prevents revolts in the same sense that a hydroelectric dam prevents floods. It’s not a matter of stopping up the revolutionary urge; in fact, any attempt to do so would be disastrous sooner or later. Instead it provides a safe, easy channel, and in the process, captures all the power of the movement before that flow can build up enough to cause damage.
The voters can have whatever they want, and the rest of the system does it’s best to stop them from wanting anything dangerous.
But would that something form a utility function that wouldn’t be deeply horrifying to the vast majority of humanity?
It wouldn’t form a utility function at all. It has no answer for any of the interesting or important questions: the questions on which there is disagreement. Or am I missing something here?
In the human genome project analogy, they wound up with one person’s DNA.
Humans have various eye colours—and the sequence they wound up with seems likely to have some eye colour or another.
Ok, you are changing the analogy. Initially you said, throw away the differences. Now you are saying throw away all but one of them.
So our revised approximation of the CEV is the expressed volition of … Craig Venter?!
Would that horrify the vast majority of humanity? I think it might. Mostly because people just would not know how it would play out. People generally prefer the devil they know to the one they don’t.
FWIW, it wasn’t really Craig Venter, but a combination of multiple people—see:
http://en.wikipedia.org/wiki/Human_Genome_Project#Genome_donors
No, I agree. I just don’t understand where you were going when you emphasized that
The guy who wrote and emphasized that was timtyler—It wasn’t me
The anti-kibitzer is more confusing than I realized.
Well, it was I who wrote that. The differences were thrown away in the genome project—but that isn’t exactly the corresponding thing according to the CEV proposal.
A certain lack of coherence doesn’t mean all the conflicting desires cancel out leaving nothing behind—thus the emphasis on still being “left with something”.
I’m looking at the same document you are, and I actually agree that EV almost certainly ~C. I just wanted to make sure the assumption was explicit.