To have this attitude, you need a strong presumption of your own superiority. Instead of engaging them in a conversation where you can both better discover the truth, with you also remaining open to info they may offer, you decide this is war and “all is fair in love and war.” You know what is true and what is good for the world and you’ve decided it is important enough to the world for them to believe your truth that you will force it upon them in any way you can. No doubt this is a possible situation, and there is possible evidence that could reasonably convince you that this is your situation. But do pause and consider whether your confidence might be overconfidence, biased by arrogance, and also consider the consequences of their hearing that this is in fact your attitude toward them.
First of all, let me say that rationalist-to-rationalist conversations don’t really have this problem. This is all about what happens when we talk to people who think less in “far-mode” if you will. What I’ve found is when talking with non-rationalists, I have to consciously switch into a different mindset to get “flow”. Let me give a personal example.
I was once at a bar where I met some random people, and one of them told me something that a rationalist would consider “woo”. He explained that he’d read that an atomic bomb was a particularly terrible thing, because unlike when you die normally, the radiation destroys the souls. I paused for a moment, swallowed all my rationalist impulses, and thought: “Is there anyway what he said could be meaningful?” I responded: “Well the terrible thing about an atomic explosion, is that it kills not just a person in isolation, but whole families, whole communities… if just one person dies, their friends and families can respect that person’s death and celebrate their memories, but when that many people die all at once, their entire history of who they are, their souls, are just erased in an instant”. He told me that was deep, and bought me a drink.
Did I feel dishonest? Not really. I decided what was relevant and what was not. Obviously the bit he said about radiation didn’t make scientific sense, but I didn’t feel the reason he’d brought up the idea was that he cared for a science lesson. Similarly, I could have asked the question: “well exactly what do you mean by a ‘soul’?” Instead I chose an interpretation that seemed agreeable to both of us. Now, had he specifically asked me for an analytical opinion, I would have absolutely given him that. But for now, what I’d done had earned myself some credibility, so that later in the conversation, if I wanted to be persuasive of an important “rationalist” opinion, I’d actually be someone worth listening too.
Yes of course you should not habitually divert conversations into you lecturing others on how some specific thing they said might be in error. For example, when I listen to an academic talk I do not speak up about most of the questionable claims made—I wait until I can see the main point of the talk and see what questionable claims might actually matter for their main claim. Those are the points I consider raising. Always keep in mind the purpose of your conversation.
Depending on the purpose of the conversation, do you think dark arts are sometimes legitimate? Or perhaps a more interesting question for an economist: can you speculate as to the utility (let’s say some measure of persuasive effectiveness) of dark arts, depending on the conversation type (e.g. a State of the Union Address and a Conversation on Less Wrong would presumably be polar opposites)
I’d rather ask the question without the word “sometimes”. Because what people do is use that word “sometimes” as a rationalization. “We’ll only use the Dark Arts in the short term, in the run-up to the Singularity.” The notion is that once everybody becomes rational, we can stop using them.
I’m skeptical that will happen. As we become more complex reasoners, we will develop new bugs and weaknesses in our reasoning for more-sophisticated dark artists to exploit. And we will have more complicated disagreements with each other, with higher stakes; so we will keep justifying the use of the Dark Arts.
As we become more complex reasoners, we will develop new bugs and weaknesses in our reasoning for more-sophisticated dark artists to exploit.
Are we expecting to become more complex reasoners? It seems to be the opposite to me. We are certainly moving in the direction of reasoning about increasingly complex things, but by all indications, the mechanisms of normal human reasoning are much more complex than they should be, which is why it has so many bugs and weaknesses in the first place. Becoming better at reasoning, in the LW tradition, appears to consist entirely of removing components (biases, obsolete heuristics, bad epistemologies and cached thoughts, etc.), not adding them.
If the goal is to become perfect Bayesians, then the goal is simplicity itself. I realize that is probably an impossible goal — even if the Singularity happens and we all upload ourselves into supercomputer robot brains, we’d need P=NP in order to compute all of our probabilities to exactly where they should be — but every practical step we take, away from our evolutionary patchwork of belief-acquisition mechanisms and toward this ideal of rationality, is one less opportunity for things to go wrong.
As we become more complex reasoners, we will develop new bugs and weaknesses in our reasoning for more-sophisticated dark artists to exploit. And we will have more complicated disagreements with each other, with higher stakes; so we will keep justifying the use of the Dark Arts.
This is exactly the chain of reasoning I had in mind in my original post when I referred to the “big if”.
I’d like to hear some justification—some extensive justification, at least a sequence’s worth—explaining how building a Friendly AI, with the already-expressed intent of beating all other AIs to the punch and then using your position of power to suppress or destroy construction of any other AIs at any cost, and to make yours a singleton designed in such a way that the values you programmed it with can never be altered -
-- can amount to anything other than what Robin just described.
(Elaborating after a day with no responses)
I realize that the first answer is going to be something along the lines of, “But we don’t program in the values. We just design an algorithm that can extrapolate values from everyone else.”
First, I’ve spoken with many of the people involved, and haven’t heard any of them express views consistent with this—they want their values to be preserved, in fact two said explicitly that they did not care what happened to the universe if their personal values were not preserved—and yet they also believe that their values are extreme minority views among humanity. What’s more, they have views of transhumanism and humanity that make excluding lower animals from this extrapolated volition unjustifiable on any grounds that would not also exclude humans.
Second, the problem of trying to program an AI in such a way that your values do not determine the values it acquires, is isomorphic to the problem of trying to write a program to do Bayesian analysis in a way that will not be influenced by your priors; or trying to evaluate a new scientific idea that isn’t influenced by your current scientific paradigm. It can’t be done, except by defining your terms in a way that hides the problem.
Third, the greatest concern here is how much respect will be given to our free will when setting up the AI governor over us. Given that the people doing the setting up unanimously don’t believe in free will, the only logical answer is, Zero.
I’ve spoken with many of the people involved, and haven’t heard any of them express views consistent with this—they want their values to be preserved, in fact two said explicitly that they did not care what happened to the universe if their personal values were not preserved—and yet they also believe that their values are extreme minority views among humanity.
Could you add more details?
I’m really interested in this issue since I have similar concerns regarding the whole CEV-idea: It presumes that the values of almost every human somehow converge if they were only smarter, had more time etc. . I don’t know, it would definitely be nice if this were true, but if I look at most people around me, read a random history book, or just watch 5 minutes TV, I see values absurdly different from mine.
To be frank, I think I would trust a CEV more if the FAI would only extrapolate the volition of highly intelligent people.
Damn, thinking about it, I have to say: If I had to choose between a FAI only scanning the brain of Eliezer and a FAI scanning every human on earth, then I would choose Eliezer!
Well, you could argue that this only shows that I’m a fanatic lunatic or a cynical misanthrope...
Anyway, I would like to hear your current thoughts on this subject!
To be frank, I think I would trust a CEV more if the FAI would only extrapolate the volition of highly intelligent people.
By ‘extrapolated’ we mean that the FAI is calculating what the wishes of those people would be IF they were as intelligent and well-informed as the FAI.
Given that, what difference do you think it would make for the FAI to only scan intelligent people? I can imagine only negatives: a potential neglect of physical/non intellectuals pursuits as a potential source of Fun, greater political opposition if not everyone is scanned, harder time justifying the morality of letting something take control that doesn’t take EVERYONE’S desires into consideration...
By ‘extrapolated’ we mean that the FAI is calculating what the wishes of those people would be IF they were as intelligent and well-informed as the FAI.
I don’t think I understand this.
If the FAI would make Stalin as intelligent and well-informed as the FAI, then this newly created entity wouldn’t be Stalin anymore. In fact, it would be something totally different.
But maybe I’m just too stupid and there is some continuity of identity going on.
Then I have to ask: Why not extrapolate the volition of every animal on earth? If you can make Stalin intelligent and moral and you somehow don’t annihilate the personality of Stalin then I propose the same thing is possible for every (other) pig.
Now you could say something like ” Well, Stalin is an exception, he obviously was hopelessly evil”, but even today Stalin is the hero of many people.
To but it bluntly: Many folks seem to me rather stupid or malicious or both. If the FAI makes them “as intelligent and well-informed as the FAI itself” then they would be different people.
Anyway, these were only my misanthropic ramblings, and I don’t believe they are accurate, since they are probably at least partly caused by depression and frustration. But somehow I felt the urge to utter them;)
Look, I really hope this whole CEV-stuff makes sense, but I think I’m not the only one who doubts it. ( And not only ordinary folks like me, but e.g. Wei Dai, Phil Goetz, etc. )
Do you know of any further explanations of CEV besides this one?
If the FAI would make Stalin as intelligent and well-informed as the FAI, then this newly created entity wouldn’t be Stalin anymore. In fact, it would be something totally different.
Yeah, so? Nobody said the intelligently extrapolated volition of Stalin would be Stalin. It would be a parameter in the calculation parameters of the FAI.
But maybe I’m just too stupid and there is some continuity of identity going on.
We’re not talking about simulating people. We’re talking about extrapolating their volition. Continuity of identity has nothing to do with anything.
To put it bluntly: Many folks seem to me either utterly stupid or evil or both.If the FAI makes them “as intelligent and well-informed as the FAI itself” then they would be different people.
Which is the exact point, that we don’t want the current volition of people (since people are currently stupid), we want their extrapolated volition, what they would choose if they were much more intelligent than they currently are.
To have this attitude, you need a strong presumption of your own superiority. Instead of engaging them in a conversation where you can both better discover the truth, with you also remaining open to info they may offer, you decide this is war and “all is fair in love and war.” You know what is true and what is good for the world and you’ve decided it is important enough to the world for them to believe your truth that you will force it upon them in any way you can. No doubt this is a possible situation, and there is possible evidence that could reasonably convince you that this is your situation. But do pause and consider whether your confidence might be overconfidence, biased by arrogance, and also consider the consequences of their hearing that this is in fact your attitude toward them.
First of all, let me say that rationalist-to-rationalist conversations don’t really have this problem. This is all about what happens when we talk to people who think less in “far-mode” if you will. What I’ve found is when talking with non-rationalists, I have to consciously switch into a different mindset to get “flow”. Let me give a personal example.
I was once at a bar where I met some random people, and one of them told me something that a rationalist would consider “woo”. He explained that he’d read that an atomic bomb was a particularly terrible thing, because unlike when you die normally, the radiation destroys the souls. I paused for a moment, swallowed all my rationalist impulses, and thought: “Is there anyway what he said could be meaningful?” I responded: “Well the terrible thing about an atomic explosion, is that it kills not just a person in isolation, but whole families, whole communities… if just one person dies, their friends and families can respect that person’s death and celebrate their memories, but when that many people die all at once, their entire history of who they are, their souls, are just erased in an instant”. He told me that was deep, and bought me a drink.
Did I feel dishonest? Not really. I decided what was relevant and what was not. Obviously the bit he said about radiation didn’t make scientific sense, but I didn’t feel the reason he’d brought up the idea was that he cared for a science lesson. Similarly, I could have asked the question: “well exactly what do you mean by a ‘soul’?” Instead I chose an interpretation that seemed agreeable to both of us. Now, had he specifically asked me for an analytical opinion, I would have absolutely given him that. But for now, what I’d done had earned myself some credibility, so that later in the conversation, if I wanted to be persuasive of an important “rationalist” opinion, I’d actually be someone worth listening too.
Yes of course you should not habitually divert conversations into you lecturing others on how some specific thing they said might be in error. For example, when I listen to an academic talk I do not speak up about most of the questionable claims made—I wait until I can see the main point of the talk and see what questionable claims might actually matter for their main claim. Those are the points I consider raising. Always keep in mind the purpose of your conversation.
Depending on the purpose of the conversation, do you think dark arts are sometimes legitimate? Or perhaps a more interesting question for an economist: can you speculate as to the utility (let’s say some measure of persuasive effectiveness) of dark arts, depending on the conversation type (e.g. a State of the Union Address and a Conversation on Less Wrong would presumably be polar opposites)
I’d rather ask the question without the word “sometimes”. Because what people do is use that word “sometimes” as a rationalization. “We’ll only use the Dark Arts in the short term, in the run-up to the Singularity.” The notion is that once everybody becomes rational, we can stop using them.
I’m skeptical that will happen. As we become more complex reasoners, we will develop new bugs and weaknesses in our reasoning for more-sophisticated dark artists to exploit. And we will have more complicated disagreements with each other, with higher stakes; so we will keep justifying the use of the Dark Arts.
Are we expecting to become more complex reasoners? It seems to be the opposite to me. We are certainly moving in the direction of reasoning about increasingly complex things, but by all indications, the mechanisms of normal human reasoning are much more complex than they should be, which is why it has so many bugs and weaknesses in the first place. Becoming better at reasoning, in the LW tradition, appears to consist entirely of removing components (biases, obsolete heuristics, bad epistemologies and cached thoughts, etc.), not adding them.
If the goal is to become perfect Bayesians, then the goal is simplicity itself. I realize that is probably an impossible goal — even if the Singularity happens and we all upload ourselves into supercomputer robot brains, we’d need P=NP in order to compute all of our probabilities to exactly where they should be — but every practical step we take, away from our evolutionary patchwork of belief-acquisition mechanisms and toward this ideal of rationality, is one less opportunity for things to go wrong.
This is exactly the chain of reasoning I had in mind in my original post when I referred to the “big if”.
I’d like to hear some justification—some extensive justification, at least a sequence’s worth—explaining how building a Friendly AI, with the already-expressed intent of beating all other AIs to the punch and then using your position of power to suppress or destroy construction of any other AIs at any cost, and to make yours a singleton designed in such a way that the values you programmed it with can never be altered -
-- can amount to anything other than what Robin just described.
(Elaborating after a day with no responses)
I realize that the first answer is going to be something along the lines of, “But we don’t program in the values. We just design an algorithm that can extrapolate values from everyone else.”
First, I’ve spoken with many of the people involved, and haven’t heard any of them express views consistent with this—they want their values to be preserved, in fact two said explicitly that they did not care what happened to the universe if their personal values were not preserved—and yet they also believe that their values are extreme minority views among humanity. What’s more, they have views of transhumanism and humanity that make excluding lower animals from this extrapolated volition unjustifiable on any grounds that would not also exclude humans.
Second, the problem of trying to program an AI in such a way that your values do not determine the values it acquires, is isomorphic to the problem of trying to write a program to do Bayesian analysis in a way that will not be influenced by your priors; or trying to evaluate a new scientific idea that isn’t influenced by your current scientific paradigm. It can’t be done, except by defining your terms in a way that hides the problem.
Third, the greatest concern here is how much respect will be given to our free will when setting up the AI governor over us. Given that the people doing the setting up unanimously don’t believe in free will, the only logical answer is, Zero.
Could you add more details? I’m really interested in this issue since I have similar concerns regarding the whole CEV-idea: It presumes that the values of almost every human somehow converge if they were only smarter, had more time etc. . I don’t know, it would definitely be nice if this were true, but if I look at most people around me, read a random history book, or just watch 5 minutes TV, I see values absurdly different from mine.
To be frank, I think I would trust a CEV more if the FAI would only extrapolate the volition of highly intelligent people. Damn, thinking about it, I have to say: If I had to choose between a FAI only scanning the brain of Eliezer and a FAI scanning every human on earth, then I would choose Eliezer!
Well, you could argue that this only shows that I’m a fanatic lunatic or a cynical misanthrope...
Anyway, I would like to hear your current thoughts on this subject!
By ‘extrapolated’ we mean that the FAI is calculating what the wishes of those people would be IF they were as intelligent and well-informed as the FAI.
Given that, what difference do you think it would make for the FAI to only scan intelligent people? I can imagine only negatives: a potential neglect of physical/non intellectuals pursuits as a potential source of Fun, greater political opposition if not everyone is scanned, harder time justifying the morality of letting something take control that doesn’t take EVERYONE’S desires into consideration...
I don’t think I understand this. If the FAI would make Stalin as intelligent and well-informed as the FAI, then this newly created entity wouldn’t be Stalin anymore. In fact, it would be something totally different. But maybe I’m just too stupid and there is some continuity of identity going on. Then I have to ask: Why not extrapolate the volition of every animal on earth? If you can make Stalin intelligent and moral and you somehow don’t annihilate the personality of Stalin then I propose the same thing is possible for every (other) pig. Now you could say something like ” Well, Stalin is an exception, he obviously was hopelessly evil”, but even today Stalin is the hero of many people.
To but it bluntly: Many folks seem to me rather stupid or malicious or both. If the FAI makes them “as intelligent and well-informed as the FAI itself” then they would be different people.
Anyway, these were only my misanthropic ramblings, and I don’t believe they are accurate, since they are probably at least partly caused by depression and frustration. But somehow I felt the urge to utter them;) Look, I really hope this whole CEV-stuff makes sense, but I think I’m not the only one who doubts it. ( And not only ordinary folks like me, but e.g. Wei Dai, Phil Goetz, etc. ) Do you know of any further explanations of CEV besides this one?
Yeah, so? Nobody said the intelligently extrapolated volition of Stalin would be Stalin. It would be a parameter in the calculation parameters of the FAI.
We’re not talking about simulating people. We’re talking about extrapolating their volition. Continuity of identity has nothing to do with anything.
Which is the exact point, that we don’t want the current volition of people (since people are currently stupid), we want their extrapolated volition, what they would choose if they were much more intelligent than they currently are.