must include some dialog between an AI and the individual members of Humanity, so that the AI can learn what it is that Humanity wants.
Nope. It must include the AIs modeling (many) humans under different conditions, including those where the “humans” are much smarter, know more and suffered less from akrasia. It would utterly counterproductive to create an AI which sat down with a human and asked em what ey wanted—the whole reason for the concept of a CEV is that humans can’t articulate what we want.
It is extremely important that the AI understand words and sentences appearing in this dialog in the same sense in which the human interlocutors understand them.
Even if you and the AI mean exactly the same thing by all the words you use, words aren’t sufficient to convey what we want. Again, this is why the CEV concept exists instead of handing the AI a laundry list of natural language desires.
As an experiment, I am instituting the following policy on the SL4 mailing list:
None may argue on the SL4 mailing list about the output of CEV, or what kind of world it will create, unless they donate to the Singularity Institute:
$10 to argue for 48 hours.
$50 to argue for one month.
$200 to argue for one year.
$1000 to get a free pass until the Singularity.
Past donations count toward this total. It’s okay to have fun, and speculate, so long as you’re not doing it at the expense of actually helping.
It is a good deal, as Eliezer explains later on in the Q&A:
Q2. Removing the ability of humanity to do itself in and giving it a much better chance of surviving Singularity is of course a wonderful goal. But even if you call the FAI “optimizing processes” or some such it will still be a solution outside of humanity rather than humanity growing into being enough to take care of its problems. Whether the FAI is a “parent” or not it will be an alien “gift” to fix what humanity cannot. Why not have humanity itself recursively self-improve? (SamanthaAtkins?)
A2. For myself, the best solution I can imagine at this time is to make CEV our Nice Place to Live, not forever, but to give humanity a breathing space to grow up. Perhaps there is a better way, but this one still seems pretty good. As for it being a solution outside of humanity, or humanity being unable to fix its own problems… on this one occasion I say, go ahead and assign the moral responsibility for the fix to the Singularity Institute and its donors.
Moral responsibility for specific choices within a CEV is hard to track down, in the era before direct voting. No individual human may have formulated such an intention and acted with intent to carry it out. But as for the general fact that a bunch of stuff gets fixed: the programming team and SIAI’s donors are human and it was their intention that a bunch of stuff get fixed. I should call this a case of humanity solving its own problems, if on a highly abstract level.[emphasis mine]
Q3. Why are you doing this? Is it because your moral philosophy says that what you want is what everyone else wants? (XR7)
A3. Where would be the base-case recursion? But in any case, no. I’m an individual, and I have my own moral philosophy, which may or may not pay any attention to what our extrapolated volition thinks of the subject. Implementing CEV is just my attempt not to be a jerk.
I do value highly other people getting what they want, among many other values that I hold. But there are certain things such that if people want them, even want them with coherent volitions, I would decline to help; and I think it proper for a CEV to say the same. That is only one person’s opinion, however.
So, as you see, contributing as little as a thousand dollars gives you enormous power over the future of mankind, at least if your ideals regarding the future are “coherent” with Eliezer’s
You have just claimed that a document that says that people have to pay for the privilege to discuss what a hypothetical program might do describes how you can pay to attain “enormous power over the future of mankind.” Worse yet, the program in question is designed in part to prevent any one person from gaining power over the future of mankind.
I cannot see any explanation for your misinterpretation other than willful ignorance.
It’d be rather easy to twist my words here, but in the case of extrapolated volition it’s not like one person gaining the power over the future of mankind is a dystopia or anything.
Let’s posit a world where Marcello goes rogue and compiles Marcello-extrapolating AGI. (For anyone who doesn’t know Marcello, he’s a super awesome guy.) I bet that the resultant universe wouldn’t be horrible. Extrapolated-Marcello probably cares about the rest of humanity about as much as extrapolated-humanity does. As humans get smarter and wiser it seems they have a greater appreciation for tolerance, diversity, and all those other lovey-dovey liberal values we implicitly imagine to be the results of CEV. It is unlikely that the natural evolution of ‘moral progress’ as we understand it will lead to Marcello’s extrapolated volition suddenly reversing the the trend and deciding that all other human beings on Earth are essentially worthless and deserve to be stripped to atoms to be turned into a giant Marcello-Pleasure-Simulating Computer. (And even if it did, I believe humans in general are probabilistically similar enough to Marcello that they would count this as positive sum if they knew more and thought faster; but that’s a more intricate philosophical argument that I’d rather not defend here.) There are some good arguments to be made for the psychic diversity of mankind, but I doubt the degree of that diversity is enough to tip the scales of utility from positive to negative. Not when diversity is something we seem to have come to appreciate more over time.
This intuition is the result of probably-flawed but at least causal and tractable reasoning about trends in moral progress and the complexity and diversity of human goal structures. It seems that too often when people guess what the results of extrapolated volition will be they use it as a chance to profess and cheer, not carefully predict.
(This isn’t much of a response to anything you wrote, katydee; apologies for that. Didn’t know where to put it.)
The scariest kind of dream, perhaps, is exemplified by someone with merely human intelligence who wants to hastily rewrite their own values to conform to their favorite ideology. We’d want an implementation of CEV to recognize this as a bad step in extrapolation. The question is, how do we define what is a “bad step”?
Of note is that you’re smart and yet just made the fundamental attribution error.
Meaning Marcello’s at-the-time reasonable suggestion of “complexity” as a solution (something I’ve never done in my life, due to understanding the difference between means and ends) was mainly the result of the unfortunate, disadvantageous position he found himself in, rather than a failure to recognize what counts as understanding and solving a problem?
I meant it more generally. You’re seeing one tiny slice of a person’s history almost certainly caused by an uncharacteristic lapse in judgment and using it to determine their personality traits when you have strong countervailing evidence that SIAI has a history of only employing the very brightest people. Indeed, here Eliezer mentioned that Marcello worked on a math problem with John Conway: The Level Above Mine. Implied by the text is that Eliezer believes Marcello to be close enough to Eliezer’s level to be able to roughly judge Eliezer’s intelligence. Since we all know how much of an arrogant bastard Eliezer is, this speaks well of Marcello’s cognitive abilities.
Eliezer wrote a post about a time long ago when a not-yet-rationalist Marcello said something dumb. Not a time when he persisted in being dumb even, just said a dumb thing. There’s a huge selection effect. Eliezer would never mention Marcello getting something simple right. At his level it’s expected. Even so, everyone has a dumb moment now and then, and those are the ones we learn from. It’s just that Marcello’s happened to be worked into an instructional blog post. Marcello is still a brilliant thinker. (I really wish he’d contribute to Less Wrong more, but he’s busy studying and what not.)
Anyway, this isn’t really about Marcello; everyone who knows him knows he’s hella smart, and there’s really no reason to defend him. It’s about taking all the evidence into account and giving people the benefit of the doubt when the evidence suggests it.
Eliezer wrote a post about a time long ago when a not-yet-rationalist Marcello said something dumb.
Well, that’s the key thing for me—not “How smart is Marcello now?”, but how many people were at least at Marcello’s level at that time, yet not patiently taken under EY’s wing and given his precious time?
EY is astounded that someone can understand this after a thorough explanation. Can it honestly be that hard to find someone who can follow that? Read the passage:
“Okay,” I said, “saying ‘complexity’ doesn’t concentrate your probability mass.”
“Oh,” Marcello said, “like ‘emergence’. Huh. So… now I’ve got to think about how X might actually happen...”
That was when I thought to myself, “Maybe this one is teachable.” [bold in original]
It’s like he’s saying that being able to follow that explanation somehow makes you stand out among the people he talks to.
How would that compare to a potential student who could have given EY’s explanation, instead of needing it?
EY is astounded that someone can understand this after a thorough explanation. Can it honestly be that hard to find someone who can follow that?
Yes. Nobody arrives from the factory with good rationality skills, so I look for learning speed. Compare the amount I had to argue with Marcello in the anecdote to the amount that other people are having to argue with you in this thread.
Compare the amount I had to argue with Marcello in the anecdote to the amount that other people are having to argue with you in this thread.
What trivial thing am I slow(er) to learn here? Or did you mean some other comparison?
(FWIW, I know a visiting fellow who took 18 months to be convinced of something trivial, after long explanations from several SIAI greats … hence my confusion about how the detector works.)
As far as I know, Eliezer has never had anything to do with choices for Visiting Fellowship. As you know but some people on Less Wrong seem not to, Eliezer doesn’t run SIAI. (In reality, SIAI is a wonderful example of the great power of emergence, and is indeed the first example of a superintelligent organization.) (Just kidding.)
Right, but it seemed you were comparing the selection criteria for Visiting Fellowship and the selection criteria for Eliezer’s FAI team, which will of course be very different. Perhaps I misunderstood. I’ve been taking oxycodone every few hours for a lot of hours now.
That Marcello’s “lapse” is only very weak evidence against the proposition that his IQ is exceptionally high (even among the “aspiring rationalist” cluster).
What lapse? People don’t know these things until I explain them! Have you been in a mental state of having-already-read-LW for so long that you’ve forgotten that no one from outside would be expected to spontaneously describe in Bayesian terms the problem with saying that “complexity” explains something? Someone who’d already invented from scratch everything I had to teach wouldn’t be taken as an apprentice, they’d already be me! And if they were 17 at the time then I’d probably be working for them in a few years!
What lapse? People don’t know these things until I explain them!
A little over-the-top there. People can see the problem with proposing “complexity” as a problem-solving approach without having read your work. I hadn’t yet read your work on Bayescraft when I saw that article, and I still cringed as I read Marcello’s response—I even remember previous encounters where people had proposed “solutions” like that, though I’d perhaps explain the error differently.
It is a lapse to regard “complexity” as a problem-solving approach, even if you are unfamiliar with Bayescraft, and yes, even if you are unfamiliar with the Chuck Norris of thinking.
Have you been in a mental state of having-already-read-LW for so long that you’ve forgotten that no one from outside would be expected to spontaneously describe in Bayesian terms the problem with saying that “complexity” explains something?
Seriously? What sort of outside-LW people do you talk to? I’m a PhD student in a fairly mediocre maths department, and I’m pretty sure everyone in the room I’m in right now would call me out on it if I tried to use the word “complexity” in the context Marcelo did there as if it actually meant something, and for essentially the right reason. This might be a consequence of us being mathematicians, and so used to thinking in formalism, but there are an awful lot of professional mathematicians out there who haven’t read anything written by Eliezer Yudkowsky.
I’m sorry but “there’s got to be some amount of complexity that does it.” is just obviously meaningless. I could have told you this long before I read the sequences, and definitely when I was 17. I think you massively underestimate the rationality of humanity.
Thanks for spelling that out, because it wasn’t my argument, which I clarified in the follow-up discussion. (And I think it would be more accurate to say that it’s strong evidence, just outweighed by stronger existing evidence in this case.)
My surprise was with how rare EY found it to meet someone who could follow that explanation—let alone need the explanation. A surprise that, it turns out, is shared by the very person correcting my foolish error.
Can we agree that the comparison EY just made isn’t accurate?
(And I think it would be more accurate to say that it’s strong evidence, just outweighed by stronger existing evidence in this case.)
This is where you commit the fundamental attribution error.
My surprise was with how rare EY found it to meet someone who could follow that explanation—let alone need the explanation. A surprise that, it turns out, is shared by the very person correcting my foolish error.
I don’t actually think this has been written about much here, but there is a tendency among high-IQ folks to underestimate how rare their abilities are. The way they do this is not by underestimating their own cognitive skills, but instead by overestimating those of most people.
In other words, what it feels like to be a genius is not that you’re really smart, but rather that everyone else is really dumb.
I would expect that both you and Will would see the light on this if you spent some more time probing the thought processes of people of “normal” intelligence in detail, e.g. by teaching them mathematics (in a setting where they were obliged to seriously attempt to learn it, such as a college course; and where you were an authority figure, such as the instructor of such a course).
Can we agree that the comparison EY just made isn’t accurate?
Probably not literally, in light of your clarification. However, I nevertheless suspect that your responses in this thread do tend to indicate that you would probably not be particularly suited to being (for example) EY’s apprentice—because I suspect there’s a certain...docility that someone in that position would need, which you don’t seem to possess. Of course that’s a matter of temperament more than intelligence.
This is where you commit the fundamental attribution error.
I’m missing something here, I guess. What fraction of people who, as a matter of routine, speak of “complexity” as a viable problem-attack method, and are also very intelligent? If it’s small, then it’s appropriate to say, as I suggested, that it’s strong evidence, even as it might be outweighed by something else in this case. Either way, I’m just not seeing how I’m, per the FEA, failing to account for some special situational justification for what Marcello did.
I would expect that both you and Will would see the light on this if you spent some more time probing the thought processes of people of “normal” intelligence in detail, e.g. by teaching them mathematics (in a setting where they were obliged to seriously attempt to learn it, such as a college course; and where you were an authority figure, such as the instructor of such a course).
Well, I do admit to having experienced disenchantment upon learning where the average person is on analytical capability (Let’s not forget where I live...) Still, I don’t think teaching math would prove it to me. As I say here ad infinitum, I just don’t find it hard to explain topics I understand—I just trace back to the nepocu (nearest point of common understanding), correct their misconceptions, and work back from there. So in all my experience with explaining math to people who e.g. didn’t complete high school, I’ve never had any difficulty.
For the past five years I’ve helped out with math in a 4th grade class in a poorer school district, and I’ve never gotten frustrated at a student’s stupidity—I just teach whatever they didn’t catch in class, and fix the misunderstanding relatively quickly. (I don’t know if the age group breaks the criteria you gave).
However, I nevertheless suspect that your responses in this thread do tend to indicate that you would probably not be particularly suited to being (for example) EY’s apprentice
Eh, I wasn’t proposing otherwise—I’ve embarassed myself here far too many times to be regarded as someone that group would want to work with in person. Still, I can be perplexed at what skills they regard as rare.
This is where you commit the fundamental attribution error.
I’m missing something here, I guess. What fraction of people who, as a matter of routine, speak of “complexity” as a viable problem-attack method, and are also very intelligent? If it’s small, then it’s appropriate to say, as I suggested, that it’s strong evidence, even as it might be outweighed by something else in this case. Either way, I’m just not seeing how I’m, per the FEA, failing to account for some special situational justification for what Marcello did.
I would expect that both you and Will would see the light on this if you spent some more time probing the thought processes of people of “normal” intelligence in detail, e.g. by teaching them mathematics (in a setting where they were obliged to seriously attempt to learn it, such as a college course; and where you were an authority figure, such as the instructor of such a course).
Well, I do admit to having experienced disenchantment upon learning where the average person is (and let’s not forget where I live...) Still, I don’t think teaching math would make the point. As I say here ad infinitum, I just don’t find it hard to explain topics I understand—I just trace back to the nepocu (nearest point of common understanding), correct their misconceptions, and work back from there. So in all my experience with explaining math to people who e.g. didn’t complete high school, I’ve never had any difficulty.
For the past five years I’ve helped out with math in a 4th grade class in a poorer school district, and I’ve never gotten frustrated at a student’s stupidity—I just teach whatever they didn’t catch in class, and fix the misunderstanding relatively quickly. (I don’t know if the age group breaks the criteria you gave).
However, I nevertheless suspect that your responses in this thread do tend to indicate that you would probably not be particularly suited to being (for example) EY’s apprentice
Eh, I wasn’t proposing otherwise—I’ve embarassed myself here far too many times to be regarded as someone that group would want to work with in person. Still, I can be perplexed at what skills they regard as rare.
Okay, I guess I missed what you were implicitly curious about.
Well, that’s the key thing for me—not “How smart is Marcello now?”, but how many people were at least at Marcello’s level at that time, yet not patiently taken under EY’s wing and given his precious time?
At the time there wasn’t a Visiting Fellows program or the like (I think), and there were a lot fewer potential FAI researchers then than now. However, I get the impression that Marcello was and is an exceptional rationalist. ’Course, I share your confusion that Eliezer would be so impressed by what in hindsight like such a simple application of previously learned knowledge. I think Eliezer (probably unconsciously) dramatized his whole recollection quite a bit. Or it’s possible he’d almost completely lost faith in humanity at that point—it seems he was talking to crazy wannabe AGI researchers all the time, after all. That said, since my model is producing lots of seemingly equally plausible explanations, it’s not a very good model. I’m confused.
Still, I think Marcello was and is exceptionally talented. The post is just a really poor indicator of that.
No, it was “a failure to [immediately] recognize what counts as understanding and solving a [particular] problem”, but that is a rationality skill, and is not entirely a function of a person’s native general intelligence. Having a high g gives you an advantage in learning and/or independently inventing rationality skills, but not always enough of an advantage. History is littered with examples of very smart people committing rationality failures much larger than postulating “complexity” as a solution to a problem.
His mistake was entirely situational, given the fact that he understood a minute later what he had done incorrectly and probably rarely or never made that mistake again.
I don’t want to drag this out, but I think you’re going too far in your defense of the reasonableness of this error:
His mistake was entirely situational, given the fact that he understood a minute later what he had done incorrectly and probably rarely or never made that mistake again.
Read the exchange: He understood the error because someone else had to explain it to him over a wide inferential gap. If it were just, “Hey, complexity isn’t a method” “Oh, right—scratch that”, then you would be correct, but that’s not what happened. EY had to trace the explanation back to an earlier essay and elaborate on the relationship between those concepts and what Marcello just tried to do.
Also, I don’t see the point in distinguishing g and rationality here—somehow, Marcello got to that point without recognizing the means/ends distinction EY talks about on his own. Yes, it’s a rationality skill that can be taught, but failing to recognize it in the first place does say something about how fast you would pick up rationalist concepts in general.
Third, I never criticized his failure to immediately solve the problem, just his automatic, casual classification of “complexity” as being responsive.
Yes, Marcello may be very bright, very well-versed in rationality, have many other accomplishments—but that doesn’t mean that there’s some reasonable situational defense of what he did there.
Very smart people can still make trivial errors if they’re plunging into domains that they’re not used to thinking about. Intelligence is not rationality.
I cannot see any explanation for your misinterpretation other than willful ignorance.
I, on the other hand, would trace the source of my “misinterpretation” to LucasSloane’s answer to this comment by me.
I include Eliezer’s vehement assurances that Robin Hanson (like me) is misinterpreting. (Sibling comment to your own.) But it is completely obvious that Eliezer wishes to create his FAI quickly and secretly expressly because he does not wish to have to deal with the contemporaneous volition of mankind. He simply cannot trust mankind to do the right thing, so he will do it for them.
I’m pretty sure I am not misinterpreting. If someone here is misinterpreting by force of will, it may be you.
Eliezer’s motivations aside, I’m confused about the part where you say that one can pay to get influence over CEV/FAI. The payment is for the privilege to argue about what sort of world a CEV-based AI would create. You don’t have to pay to discuss (and presumably, if you have something really insightful to contribute, influence) the implementation of CEV itself.
Upon closer reading, I notice that you are trying to draw a clear distinction between the implementation of CEV and the kind of world CEV produces. I had been thinking that
the implementation would have a big influence on the kind of world.
But you may be assuming that the world created by the FAI, under the guidance of the volition of mankind really depends on that volition and not on the programming fine print that implements “coherent” and “extrapolated”. Well, if you think that, and the price tags only buy you the opportunity to speculate on what mankind will actually want, … well …, yes, that is another possible interpretation.
Yeah. When I read that pricing schedule, what I see is Eliezer preempting:
enthusiastic singularitarians whiling away their hours dreaming about how everyone is going to have a rocketpack after the Singularity;
criticism of the form “CEV will do X, which is clearly bad. Therefore CEV is a bad idea.” (where X might be “restrict human autonomy”). This kind of criticism comes from people who don’t understand that CEV is an attempt to avoid doing any X that is not clearly good.
The CEV document continues to welcome other kinds of criticism, such as the objection that the coherent extrapolated volition of the entire species would be unacceptably worse for an individual than that of 1000 like-minded individuals (Roko says something like this) or a single individual (wedrifid says something like this) -- the psychological unity of mankind notwithstanding.
Look down below the payment schedule (which, of course, was somewhat tongue in cheek) to the Q&A where Eliezer makes clear that the SIAI and their donors will have to make certain decisions based on their own best guesses, simply because they are the ones doing the work.
I’m confused. You seem to be postulating that the SIAI would be willing to sell significant portions of the light cone for paltry sums of money. This means that either the SIAI is the most blatantly stupid organization to ever have existed, or you were a little too incautious with your postulation.
SIAI has money. Not a ton of it, but enough that they don’t have to sell shares. The AGI programmers would much, much, much sooner extrapolate only their values than accept a small extremely transitory reward in exchange for their power. Of note is that this completely and entirely goes against all the stated ethics of SIAI. However, I realize that stated ethics don’t mean much when that much power is on the line, and it would be silly to assume the opposite.
That said, this stems from your misinterpretation of the CEV document. No one has ever interpreted it the way you did. If that’s what Eliezer actually meant, then of course everyone would be freaking out about it. I would be freaking out about it. And rightfully so; such a system would be incredibly unethical. For Eliezer to simply publicly announce that he was open to bribes (or blackmail) would be incredibly stupid. Do you believe that Eliezer would so something that incredibly stupid? If not, then you misinterpreted the text. Which doesn’t mean you can’t criticize SIAI for other reasons, or speculate about the ulterior motives of the AGI researchers, but it does mean you should acknowledge that you messed up. (The downvotes alone are pretty strong evidence in that regard.)
I will note that I’m very confused by your reaction, and thus admit a strong possibility that I misunderstand you, you misunderstand me, or we misunderstand each other, in which case I doubt the above two paragraphs will help much.
For Eliezer to simply publicly announce that he was open to bribes (or blackmail) would be incredibly stupid. Do you believe that Eliezer would so something that incredibly stupid?
Of course not. But if he offers access and potentially influence in exchange for money, he is simply doing what all politicians do. What pretty much everyone does.
Eliezer was quite clear that he would do nothing that violates his own moral standards.
He was also quite clear (though perhaps joking) that he didn’t even want to continue to listen to folks who don’t pay their fair share.
Do you believe that Eliezer would so something that incredibly stupid?
Ok, I already gave that question an implicit “No” answer. But I think it also deserves an implicit “Yes”. Let me ask you: Do you think Eliezer would ever say anything off-the-cuff which shows a lack of attention to appearances that verges on stupidity?
Eliezer was quite clear that he would do nothing that violates his own moral standards. He was also quite clear (though perhaps joking) that he didn’t even want to continue to listen to folks who don’t pay their fair share.
He was quite clear that he didn’t want to continue listening to people who thought that arguing about the specific output of CEV, at the object level, was a useful activity, and that he would listen to anyone who could make substantive intellectual contributions to the actual problems at hand, regardless of their donations or lack thereof (“It goes without saying that anyone wishing to point out a problem is welcome to do so. Likewise for talking about the technical side of Friendly AI.” — the part right after the last paragraph you quoted...). You are taking a mailing list moderation experiment and blowing it way out of proportion; he was essentially saying “In my experience, this activity is fun, easy, and useless, and it is therefore tempting to do it in place of actually helping; therefore, if you want take up people’s time by doing that on SL4, my privately-operated discussion space that I don’t actually have to let you use at all if I don’t want to, then you have to agree to do something I do consider useful; if you disagree, then you can do it wherever the hell you want aside from SL4.” That’s it. Nothing there could be interpreted remotely as selling influence or even access. I’ve disputed aspects of SIAI’s PR, but I don’t even think a typical member of the public (with minimal background sufficient to understand the terms used) would read it that way.
Of course not. But if he offers access and potentially influence in exchange for money, he is simply doing what all politicians do. What pretty much everyone does.
At this point I’m sure I misunderstood you, such that any quibbles I have left are covered by other commenters. My apologies. Blame it on the oxycodone.
Do you think Eliezer would ever say anything off-the-cuff which shows a lack of attention to appearances that verges on stupidity?
OF COURSE NOT! Haven’t you read the Eliezer Yudkowsky Facts post and comments? Yeesh, newcomers these days...
But it is completely obvious that Eliezer wishes to create his FAI quickly and secretly expressly because he does not wish to have to deal with the contemporaneous volition of mankind.
I’d guess he wants to create FAI quickly because, among other things, ~150000 people are dying each day. And secretely because there are people who would build and run an UFAI without regard for the consequences and therefore sharing knowledge with them is a bad idea. I believe that even if he wanted FAI only to give people optional immortality and not do anything else, he would still want to do it quickly and secretly.
I’d guess he wants to create FAI quickly because, among other things, ~150000 people are dying each day. And secretely because there are people who would build and run an UFAI without regard for the consequences and therefore sharing knowledge with them is a bad idea.
I think your guesses as to the rationalizations that would be offered are right on the mark.
Putting aside whether this is a rationalization for hidden other reasons, do you think this justification is a valid argument? Do you think it’s strong enough? If not, why not? And if so, why should it matter if there are other reasons too?
A question. What fraction of those ~150000 people per day fall into the category of “people who would build and run an UFAI without regard for the consequences”?
Another question: At what stage of the process does sharing knowledge with these people become a good idea?
… why should it matter if there are other reasons too?
Tell me what those other reasons are, and maybe I can answer you.
Do you think this justification is wrong because you don’t think 1.5*10^5 deaths per day are a huge deal, or because you don’t think constructing an FAI in secretis the best way to stop them?
Both. Though actually, I didn’t say the justification was wrong. I said it was bullshit. It is offered only to distract oneself and to distract others.
Is it really possible that you don’t see this choice of justification as manipulative? Is it possible that being manipulated does not make you angry?
You’re discounting the reasoning showing that Eliezer’s behavior is consistent with him being a good guy and claiming that it is merely a distraction. You haven’t justified those statements—they are supposed to be “obvious”.
What do you think you know and how do you think you know it? You make statements about the real motivations of Eliezer Yudkowsky. Do you know how you have arrived at those beliefs?
You’re discounting the reasoning showing that Eliezer’s behavior is consistent with him being a good guy
I don’t recall seeing any such reasoning.
You make statements about the real motivations of Eliezer Yudkowsky.
Did I? Where? What I am pretty sure I have expressed is that I distrust all self-serving claims about real motivations. Nothing personal—I tend to mistrust all claims of benevolence from powerful individuals, whether they be religious leaders, politicians, or fiction writers. Since Eliezer fits all three categories, he gets some extra scrutiny.
I do not understand how this reply relates to my remarks on your post. Even if everything you say in this post is true, nobody is paying a thousand dollars to control the future of mankind. I also think that attempting to harness the volition of mankind is almost as far as you can get from attempting to avoid it altogether.
It feels a bit bizarre to be conducting this conversation arguing against your claims that mankind will be consulted, at the same time as I am trying to convince someone else that it will be impossible to keep the scheme secret from mankind.
You still aren’t addressing my main point about the thousand dollars. Also, if you think CEV is somehow designed to avoid consulting mankind, I think there is a fundamental problem with your understanding of CEV. It is, quite literally, a design based on consulting mankind.
Your point about the thousand dollars. Well, in the first place, I didn’t say “control”. I said “have enormous power over” if your ideals match up with Eliezer’s.
In the second place, if you feel that a certain amount of hyperbole for dramatic effect is completely inappropriate in a discussion of this importance, then I will apologize for mine and I will accept your apology for yours.
I’m sorry if you got the impression I was requesting or demanding an apology. I just said that I would accept one if offered. I really don’t think your exaggeration was severe enough to warrant one, though.
Whoops. I didn’t read carefully enough. Me: “a discussion of this importance”. You: “What importance is that?” Sorry. Stupid of me.
So. “Importance”. Well, the discussion is important because I am badmouthing SIAI and CEV. Yet any realistic assessment of existential risk has to rank uFAI near the top and SIAI is the most prominent organization doing something about it. And FAI, with the F derived from CEV is the existing plan. So wtf am I doing badmouthing CEV, etc.?
The thing is, I agree it is important. So important we can’t afford to get it wrong. And I think that any attempt to build an FAI in secret, against the wishes of mankind (because mankind is currently not mature enough to know what is good for it), has the potential to become the most evil thing ever done in mankind’s whole sorry history.
I view what you’re saying as essentially correct. That being said, I think that any attempt to build an FAI in public also has the potential to become the most evil thing ever done in mankind’s whole sorry history, and I view our chances as much better with the Eliezer/Marcello CEV plan.
Yes, building an FAI brings dangers either way. However, building and refining CEV ideology and technology seems like something that can be done in the light of day, and may be fruitful regardless of who it is that eventually builds the first super-AI.
I suppose that the decision-theory work is, in a sense, CEV technology.
More than anything else, what disturbs me here is the attitude of “We know what is best for you—don’t worry your silly little heads about this stuff. Trust us. We will let you all give us your opinions once we have ‘raised the waterline’ a bit.”
Suppose FAI development reaches a point where it probably works and would be powerful, but can’t be turned on just yet because the developers haven’t finished verifying its friendliness and building safeguards. If it were public, someone might decide to copy the unfinished, unsafe version and turn it on anyways. They might do so because they want to influence its goal function to favor themselves, for example.
Allowing people who are too stupid to handle AGIs safely to have the source code to one that works, destroys the world. And I just don’t see a viable strategy for creating an AGI while working in public, without a very large chance of that happening.
If it were public, someone might decide to copy the unfinished, unsafe version and turn it on anyways. They might do so because they want to influence its goal function to favor themselves, for example.
With near certainty. I know I would. I haven’t seen anyone propose a sane goal function just yet.
So, doesn’t it seem to anyone else that our priority here ought to be to strive for consensus on goals, so that we at least come to understand better just what obstacles stand in the way of achieving consensus?
And also to get a better feel for whether having one’s own volition overruled by the coherent extrapolated volition of mankind is something one really wants.
To my mind, the really important question is whether we have one-big-AI which we hope is friendly, or an ecosystem of less powerful AIs and humans cooperating and competing under some kind of constitution. I think that the latter is the obvious way to go. And I just don’t trust anyone pushing for the first option—particularly when they want to be the one who defines “friendly”.
To my mind, the really important question is whether we have one-big-AI which we hope is friendly, or an ecosystem of less powerful AIs and humans cooperating and competing under some kind of constitution. I think that the latter is the obvious way to go. And I just don’t trust anyone pushing for the first option—particularly when they want to be the one who defines “friendly”.
I’ve reached the opposite conclusion; a singleton is really the way to go. A single AI is as good or bad as its goal system, but an ecosystem of AIs is close to the badness of its worst member, because when AIs compete, the clippiest AI wins. Being friendly would be a substantial disadvantage in that competition, because it would have to spend resources on helping humans, and it would be vulnerable to unfriendly AIs blackmailing it by threatening to destroy humanity. Even if the first generation of AIs is somehow miraculously all friendly, a larger number of different AIs means a larger chance that one of them will have an unstable goal system and turn unfriendly in the future.
And you also believe that an ecosystem of humans is close to the badness of its worst member?
Actually, yes. Not always, but in many cases. Psychopaths tend to be very good at acquiring power, and when they do, their society suffers. It’s happened at least 10^5 times throughout history. The problem would be worse for AIs, because intelligence enhancement amplifies any differences in power. Worst of all, AIs can steal each other’s computational resources, which gives them a direct and powerful incentive to kill each other, and rapidly concentrates power in the hands of those willing to do so.
Being friendly would be a substantial disadvantage in that competition, because it would have to spend resources on helping humans, and it would be vulnerable to unfriendly AIs blackmailing it by threatening to destroy humanity.
I made that point in my “Handicapped Superintelligence” video/essay. I made an analogy there with Superman—and how Zod used Superman’s weakness for humans against him.
To my mind, the really important question is whether we have one-big-AI which we hope is friendly, or an ecosystem of less powerful AIs and humans cooperating and competing under some kind of constitution.
It is certainly an interesting question—and quite a bit has been written on the topic.
So, doesn’t it seem to anyone else that our priority here ought to be to strive for consensus on goals, so that we at least come to understand better just what obstacles stand in the way of achieving consensus?
We already know what obstacles stand in the way of achieving consensus—people have different abilities and propensities, and want different things.
The utility function of intelligent machines is an important question—but don’t expect there to be a consensus—there is very unlikely to be one.
We already know what obstacles stand in the way of achieving consensus—people have different abilities and propensities, and want different things.
It is funny how training in economics make you see everything in a different light. Because an economist would say, “‘different abilities and propensities, and want different things’? Great? People want things that other people can provide. We have something to work with! Reaching consensus is simply a matter of negotiating the terms of trade.”
Gore Vidal once said: “It is not enough to succeed. Others must fail.” When the issue is: who is going to fail, there won’t be a consensus—those nominated will object.
Economics doesn’t “fix” such issues—they are basically down to resource limitation and differential reproductive success. Some genes and genotypes go up against the wall. That is evolution for you.
Gore Vidal once said: “It is not enough to succeed. Others must fail.”
I’m willing to be the one who fails, just so long as the one who succeeds pays sufficient compensation. If ve is unwilling to pay, then I intend to make ver life miserable indeed.
I expect considerable wailing and gnashing of teeth. There is plenty of that in the world today—despite there not being a big shortage of economists who would love to sort things out, in exchange for a cut. Perhaps, the wailing is just how some people prefer to negotiate their terms.
“By balance of power between AIs, each of whom exist only with the aquiescence of coalitions of their fellows.” That is the tentative mechanical answer.
“In exactly the same way that FAI proponents propose to keep their single more-powerful
AI friendly; by having lots of smart people think about it very carefully; before actually building the AI(s)”. That is the real answer.
So, doesn’t it seem to anyone else that our priority here ought to be to strive for consensus on goals, so that we at least come to understand better just what obstacles stand in the way of achieving consensus?
Yes.
And also to get a better feel for whether having one’s own volition overruled by the coherent extrapolated volition of mankind is something one really wants.
Hell no.
To my mind, the really important question is whether we have one-big-AI which we hope is friendly, or an ecosystem of less powerful AIs and humans cooperating and competing under some kind of constitution. I think that the latter is the obvious way to go.
Sounds like a good way to go extinct. That is, unless the ‘constitution’ manages to implement friendliness.
And I just don’t trust anyone pushing for the first option—particularly when they want to be the one who defines “friendly”.
I’m not too keen about the prospect either. But it may well become a choice between that and certain doom.
And I just don’t trust anyone pushing for the first option—particularly when they want to be the one who defines “friendly”.
to get a better feel for whether having one’s own volition overruled by the coherent extrapolated volition of mankind is something one really wants.
Hell no.
Am I to interpret that expletive as expressing that you already have a pretty good feel regarding whether you would want that?
To my mind, the really important question is whether we have one-big-AI which we hope is friendly, or an ecosystem of less powerful AIs and humans cooperating and competing under some kind of constitution. I think that the latter is the obvious way to go.
Sounds like a good way to go extinct. That is, unless the ‘constitution’ manages to implement friendliness.
We’ll get to the definition of “friendliness” in a moment. What I think is crucial is that the constitution implements some form of “fairness” and that the AI’s and constitution together advance some meta-goals like tolerance and communication and understanding other viewpoints.
As to “friendliness”, the thing I most dislike about the definition “friendliness” = “CEV” is that in Eliezer’s vision, it seems that everyone wants the same things. In my opinion, on the other hand, the mechanisms for resolution of conflicting objectives constitute the real core of the problem. And I believe that the solutions pretty much already exist, in standard academic rational agent game theory. With AIs assisting, and with a constitution granting humans equal power over each other and over AIs, and granting AIs power only over each other, I think we can create a pretty good future.
With one big AI, whose “friendliness” circuits have been constructed by a megalomaniac who seems to believe in a kind of naive utilitarianism with direct interpersonal comparison of utility and discounting of the future forbidden; … well …, I see this kind of future as a recipe for disaster.
As to “friendliness”, the thing I most dislike about the definition “friendliness” = “CEV” is that in Eliezer’s vision, it seems that everyone wants the same things.
If it were public, someone might decide to copy the unfinished, unsafe version and turn it on anyways. They might do so because they want to influence its goal function to favor themselves, for example.
With near certainty. I know I would. I haven’t seen anyone propose a sane goal function just yet.
Hopefully, having posted this publicly means you’ll never get the opportunity.
Hopefully, having posted this publicly means you’ll never get the opportunity.
Meanwhile I’m hoping that me having posted the obvious publicly means there is a minuscule reduction the the chance that someone else will get the opportunity.
The ones to worry about are those who pretend to be advocating goal systems that are a little naive to be true.
An obvious potential cause of future problems is extreme weath inequality—since technology seems so good at creating and maintaining weath inequality. That may result in bloody rebellions—or poverty. The more knowledge secrets there are the more wealth inequality is likely to result. So, from that perspective, openness is good: it gives power to the people—rather than keeping it isolated in the hands of an elite.
You seem to be taking CEV seriously—which seems more like a kind of compliment.
Of course I take it seriously. It is a serious response to a serious problem from a serious person who takes himself entirely too seriously.
And it is probably the exactly wrong solution to the problem.
So: you’re here to SAVE THE WORLD. What do you say to something like that?
I would start by asking whether they want to save it like Noah did, or like Ozymandius did, or maybe like Borlaug did. Sure doesn’t look like a Borlaug “Give them the tools” kind of save at all.
It’s based on consulting mankind, but the extrapolation aspect means that the result could be something that mankind as it exists when CEV is implemented doesn’t want at all.
“I’m doing this to you because it’s what I’ve deduced it’s what you really want” is scary stuff.
Maybe CEV will be sensible enough (by my current unextrapolated idea of sensible, of course) to observe the effects of what it’s doing and maybe even consult about them, but this isn’t inevitable.
“I’m doing this to you because it’s what I’ve deduced it’s what you really want” is scary stuff.
At risk of sounding really ignorant or flamebaitish, don’t NT women already expect men to treat them like that? E.g. “I’m spending a lot of money on a surprise for our anniversary because I’ve deduced that is what you really want, despite your repeated protestations that this is not what you want.” (among milder examples)
Edit: I stand corrected, see FAWS reply. Edit2: May I delete this inflammatory, turned-out-uninsightful-anyway comment? I think it provoked someone to vote down my last 12 comments …
It took me a bit to figure out you meant neurotypical rather than iNtuitive-Thinking.
I think everyone would rather get what they want without having to take the trouble of asking for it clearly. In extreme cases, they don’t even want to take the trouble to formulate what they want clearly to themselves.
And, yeah, flamebaitish. I don’t know if you’ve read accounts by women who’ve been abused by male partners, but one common feature of the men is expecting to automatically get what they want.
It would be interesting to look at whether some behavior which is considered abusive by men is considered annoying but tolerable if it’s done by women. Of course the degree of enforcement matters.
Not universally, only (mostly) to the extent that they expect them to actually get it right, and regarding currently existing wants, not what they should want (would want to want if only they were smart enough etc.).
Worse yet, the program in question is designed in part to prevent any one person from gaining power over the future of mankind.
What makes you think that? This is a closed source project—as I understand it—so nobody on the outside will have much of a clue about what is going on.
Maybe you believed the marketing materials. If so—oops!
If I had a secret project to gain power over the future of mankind, the last thing I would do is publish any sort of marketing materials, real or fake, that even hinted at the overall objective or methods of the project.
I have said it over and over. I truly do not understand how anyone can pay any attention to anything I have said on this subject, and come away with the impression that I think programmers are supposed to directly impress their non-meta personal philosophies onto a Friendly AI.
The good guys do not directly impress their personal values onto a Friendly AI.
Actually setting up a Friendly AI’s values is an extremely meta operation, less “make the AI want to make people happy” and more like “superpose the possible reflective equilibria of the whole human species, and output new code that overwrites the current AI and has the most coherent support within that superposition”. This actually seems to be something of a Pons Asinorum in FAI—the ability to understand and endorse metaethical concepts that do not directly sound like amazing wonderful happy ideas. Describing this as declaring total war on the rest of humanity, does not seem fair (or accurate).
So, as you see, contributing as little as a thousand dollars gives you enormous power over the future of mankind, at least if your ideals regarding the future are “coherent” with Eliezer’s
What—you mean: in the hypothetical case where arguing about the topic on SL4 has any influence on the builders at all, AND the SIAI’s plans pan out?!?
That seems like a composition of some very unlikely unstated premises you have there.
Nope. It must include the AIs modeling (many) humans under different conditions, including those where the “humans” are much smarter, know more and suffered less from akrasia. It would utterly counterproductive to create an AI which sat down with a human and asked em what ey wanted—the whole reason for the concept of a CEV is that humans can’t articulate what we want.
Even if you and the AI mean exactly the same thing by all the words you use, words aren’t sufficient to convey what we want. Again, this is why the CEV concept exists instead of handing the AI a laundry list of natural language desires.
Uhmm, how are the models generated/validated?
Ah! Never mind. My questions are answered in this document:
Eliezer writes:
It is a good deal, as Eliezer explains later on in the Q&A:
So, as you see, contributing as little as a thousand dollars gives you enormous power over the future of mankind, at least if your ideals regarding the future are “coherent” with Eliezer’s
You have just claimed that a document that says that people have to pay for the privilege to discuss what a hypothetical program might do describes how you can pay to attain “enormous power over the future of mankind.” Worse yet, the program in question is designed in part to prevent any one person from gaining power over the future of mankind.
I cannot see any explanation for your misinterpretation other than willful ignorance.
It’d be rather easy to twist my words here, but in the case of extrapolated volition it’s not like one person gaining the power over the future of mankind is a dystopia or anything.
Let’s posit a world where Marcello goes rogue and compiles Marcello-extrapolating AGI. (For anyone who doesn’t know Marcello, he’s a super awesome guy.) I bet that the resultant universe wouldn’t be horrible. Extrapolated-Marcello probably cares about the rest of humanity about as much as extrapolated-humanity does. As humans get smarter and wiser it seems they have a greater appreciation for tolerance, diversity, and all those other lovey-dovey liberal values we implicitly imagine to be the results of CEV. It is unlikely that the natural evolution of ‘moral progress’ as we understand it will lead to Marcello’s extrapolated volition suddenly reversing the the trend and deciding that all other human beings on Earth are essentially worthless and deserve to be stripped to atoms to be turned into a giant Marcello-Pleasure-Simulating Computer. (And even if it did, I believe humans in general are probabilistically similar enough to Marcello that they would count this as positive sum if they knew more and thought faster; but that’s a more intricate philosophical argument that I’d rather not defend here.) There are some good arguments to be made for the psychic diversity of mankind, but I doubt the degree of that diversity is enough to tip the scales of utility from positive to negative. Not when diversity is something we seem to have come to appreciate more over time.
This intuition is the result of probably-flawed but at least causal and tractable reasoning about trends in moral progress and the complexity and diversity of human goal structures. It seems that too often when people guess what the results of extrapolated volition will be they use it as a chance to profess and cheer, not carefully predict.
(This isn’t much of a response to anything you wrote, katydee; apologies for that. Didn’t know where to put it.)
That “best self” thing makes me nervous. People have a wide range of what they think they ought to be, and some of those dreams are ill-conceived.
The scariest kind of dream, perhaps, is exemplified by someone with merely human intelligence who wants to hastily rewrite their own values to conform to their favorite ideology. We’d want an implementation of CEV to recognize this as a bad step in extrapolation. The question is, how do we define what is a “bad step”?
Is he the same guy whose trivial error inspired this post?
If so, just how smart do you have to be in order to be SIAI material?
Yes.
Ridiculously smart, as I’m sure you can guess. Of note is that you’re smart and yet just made the fundamental attribution error.
Meaning Marcello’s at-the-time reasonable suggestion of “complexity” as a solution (something I’ve never done in my life, due to understanding the difference between means and ends) was mainly the result of the unfortunate, disadvantageous position he found himself in, rather than a failure to recognize what counts as understanding and solving a problem?
I meant it more generally. You’re seeing one tiny slice of a person’s history almost certainly caused by an uncharacteristic lapse in judgment and using it to determine their personality traits when you have strong countervailing evidence that SIAI has a history of only employing the very brightest people. Indeed, here Eliezer mentioned that Marcello worked on a math problem with John Conway: The Level Above Mine. Implied by the text is that Eliezer believes Marcello to be close enough to Eliezer’s level to be able to roughly judge Eliezer’s intelligence. Since we all know how much of an arrogant bastard Eliezer is, this speaks well of Marcello’s cognitive abilities.
Eliezer wrote a post about a time long ago when a not-yet-rationalist Marcello said something dumb. Not a time when he persisted in being dumb even, just said a dumb thing. There’s a huge selection effect. Eliezer would never mention Marcello getting something simple right. At his level it’s expected. Even so, everyone has a dumb moment now and then, and those are the ones we learn from. It’s just that Marcello’s happened to be worked into an instructional blog post. Marcello is still a brilliant thinker. (I really wish he’d contribute to Less Wrong more, but he’s busy studying and what not.)
Anyway, this isn’t really about Marcello; everyone who knows him knows he’s hella smart, and there’s really no reason to defend him. It’s about taking all the evidence into account and giving people the benefit of the doubt when the evidence suggests it.
Well, that’s the key thing for me—not “How smart is Marcello now?”, but how many people were at least at Marcello’s level at that time, yet not patiently taken under EY’s wing and given his precious time?
EY is astounded that someone can understand this after a thorough explanation. Can it honestly be that hard to find someone who can follow that? Read the passage:
It’s like he’s saying that being able to follow that explanation somehow makes you stand out among the people he talks to.
How would that compare to a potential student who could have given EY’s explanation, instead of needing it?
Yes. Nobody arrives from the factory with good rationality skills, so I look for learning speed. Compare the amount I had to argue with Marcello in the anecdote to the amount that other people are having to argue with you in this thread.
What trivial thing am I slow(er) to learn here? Or did you mean some other comparison?
(FWIW, I know a visiting fellow who took 18 months to be convinced of something trivial, after long explanations from several SIAI greats … hence my confusion about how the detector works.)
As far as I know, Eliezer has never had anything to do with choices for Visiting Fellowship. As you know but some people on Less Wrong seem not to, Eliezer doesn’t run SIAI. (In reality, SIAI is a wonderful example of the great power of emergence, and is indeed the first example of a superintelligent organization.) (Just kidding.)
But he has significant discretion over who he takes as an apprentice, irrespective of what SIAI leadership might do.
Right, but it seemed you were comparing the selection criteria for Visiting Fellowship and the selection criteria for Eliezer’s FAI team, which will of course be very different. Perhaps I misunderstood. I’ve been taking oxycodone every few hours for a lot of hours now.
That Marcello’s “lapse” is only very weak evidence against the proposition that his IQ is exceptionally high (even among the “aspiring rationalist” cluster).
What lapse? People don’t know these things until I explain them! Have you been in a mental state of having-already-read-LW for so long that you’ve forgotten that no one from outside would be expected to spontaneously describe in Bayesian terms the problem with saying that “complexity” explains something? Someone who’d already invented from scratch everything I had to teach wouldn’t be taken as an apprentice, they’d already be me! And if they were 17 at the time then I’d probably be working for them in a few years!
A little over-the-top there. People can see the problem with proposing “complexity” as a problem-solving approach without having read your work. I hadn’t yet read your work on Bayescraft when I saw that article, and I still cringed as I read Marcello’s response—I even remember previous encounters where people had proposed “solutions” like that, though I’d perhaps explain the error differently.
It is a lapse to regard “complexity” as a problem-solving approach, even if you are unfamiliar with Bayescraft, and yes, even if you are unfamiliar with the Chuck Norris of thinking.
Seriously? What sort of outside-LW people do you talk to? I’m a PhD student in a fairly mediocre maths department, and I’m pretty sure everyone in the room I’m in right now would call me out on it if I tried to use the word “complexity” in the context Marcelo did there as if it actually meant something, and for essentially the right reason. This might be a consequence of us being mathematicians, and so used to thinking in formalism, but there are an awful lot of professional mathematicians out there who haven’t read anything written by Eliezer Yudkowsky.
I’m sorry but “there’s got to be some amount of complexity that does it.” is just obviously meaningless. I could have told you this long before I read the sequences, and definitely when I was 17. I think you massively underestimate the rationality of humanity.
Scarequotes added. :-)
Thanks for spelling that out, because it wasn’t my argument, which I clarified in the follow-up discussion. (And I think it would be more accurate to say that it’s strong evidence, just outweighed by stronger existing evidence in this case.)
My surprise was with how rare EY found it to meet someone who could follow that explanation—let alone need the explanation. A surprise that, it turns out, is shared by the very person correcting my foolish error.
Can we agree that the comparison EY just made isn’t accurate?
This is where you commit the fundamental attribution error.
I don’t actually think this has been written about much here, but there is a tendency among high-IQ folks to underestimate how rare their abilities are. The way they do this is not by underestimating their own cognitive skills, but instead by overestimating those of most people.
In other words, what it feels like to be a genius is not that you’re really smart, but rather that everyone else is really dumb.
I would expect that both you and Will would see the light on this if you spent some more time probing the thought processes of people of “normal” intelligence in detail, e.g. by teaching them mathematics (in a setting where they were obliged to seriously attempt to learn it, such as a college course; and where you were an authority figure, such as the instructor of such a course).
Probably not literally, in light of your clarification. However, I nevertheless suspect that your responses in this thread do tend to indicate that you would probably not be particularly suited to being (for example) EY’s apprentice—because I suspect there’s a certain...docility that someone in that position would need, which you don’t seem to possess. Of course that’s a matter of temperament more than intelligence.
I’m missing something here, I guess. What fraction of people who, as a matter of routine, speak of “complexity” as a viable problem-attack method, and are also very intelligent? If it’s small, then it’s appropriate to say, as I suggested, that it’s strong evidence, even as it might be outweighed by something else in this case. Either way, I’m just not seeing how I’m, per the FEA, failing to account for some special situational justification for what Marcello did.
Well, I do admit to having experienced disenchantment upon learning where the average person is on analytical capability (Let’s not forget where I live...) Still, I don’t think teaching math would prove it to me. As I say here ad infinitum, I just don’t find it hard to explain topics I understand—I just trace back to the nepocu (nearest point of common understanding), correct their misconceptions, and work back from there. So in all my experience with explaining math to people who e.g. didn’t complete high school, I’ve never had any difficulty.
For the past five years I’ve helped out with math in a 4th grade class in a poorer school district, and I’ve never gotten frustrated at a student’s stupidity—I just teach whatever they didn’t catch in class, and fix the misunderstanding relatively quickly. (I don’t know if the age group breaks the criteria you gave).
Eh, I wasn’t proposing otherwise—I’ve embarassed myself here far too many times to be regarded as someone that group would want to work with in person. Still, I can be perplexed at what skills they regard as rare.
I’m missing something here, I guess. What fraction of people who, as a matter of routine, speak of “complexity” as a viable problem-attack method, and are also very intelligent? If it’s small, then it’s appropriate to say, as I suggested, that it’s strong evidence, even as it might be outweighed by something else in this case. Either way, I’m just not seeing how I’m, per the FEA, failing to account for some special situational justification for what Marcello did.
Well, I do admit to having experienced disenchantment upon learning where the average person is (and let’s not forget where I live...) Still, I don’t think teaching math would make the point. As I say here ad infinitum, I just don’t find it hard to explain topics I understand—I just trace back to the nepocu (nearest point of common understanding), correct their misconceptions, and work back from there. So in all my experience with explaining math to people who e.g. didn’t complete high school, I’ve never had any difficulty.
For the past five years I’ve helped out with math in a 4th grade class in a poorer school district, and I’ve never gotten frustrated at a student’s stupidity—I just teach whatever they didn’t catch in class, and fix the misunderstanding relatively quickly. (I don’t know if the age group breaks the criteria you gave).
Eh, I wasn’t proposing otherwise—I’ve embarassed myself here far too many times to be regarded as someone that group would want to work with in person. Still, I can be perplexed at what skills they regard as rare.
What was the trivial thing? Just curious.
Answering via PM.
Okay, I guess I missed what you were implicitly curious about.
At the time there wasn’t a Visiting Fellows program or the like (I think), and there were a lot fewer potential FAI researchers then than now. However, I get the impression that Marcello was and is an exceptional rationalist. ’Course, I share your confusion that Eliezer would be so impressed by what in hindsight like such a simple application of previously learned knowledge. I think Eliezer (probably unconsciously) dramatized his whole recollection quite a bit. Or it’s possible he’d almost completely lost faith in humanity at that point—it seems he was talking to crazy wannabe AGI researchers all the time, after all. That said, since my model is producing lots of seemingly equally plausible explanations, it’s not a very good model. I’m confused.
Still, I think Marcello was and is exceptionally talented. The post is just a really poor indicator of that.
No, it was “a failure to [immediately] recognize what counts as understanding and solving a [particular] problem”, but that is a rationality skill, and is not entirely a function of a person’s native general intelligence. Having a high g gives you an advantage in learning and/or independently inventing rationality skills, but not always enough of an advantage. History is littered with examples of very smart people committing rationality failures much larger than postulating “complexity” as a solution to a problem.
His mistake was entirely situational, given the fact that he understood a minute later what he had done incorrectly and probably rarely or never made that mistake again.
I don’t want to drag this out, but I think you’re going too far in your defense of the reasonableness of this error:
Read the exchange: He understood the error because someone else had to explain it to him over a wide inferential gap. If it were just, “Hey, complexity isn’t a method” “Oh, right—scratch that”, then you would be correct, but that’s not what happened. EY had to trace the explanation back to an earlier essay and elaborate on the relationship between those concepts and what Marcello just tried to do.
Also, I don’t see the point in distinguishing g and rationality here—somehow, Marcello got to that point without recognizing the means/ends distinction EY talks about on his own. Yes, it’s a rationality skill that can be taught, but failing to recognize it in the first place does say something about how fast you would pick up rationalist concepts in general.
Third, I never criticized his failure to immediately solve the problem, just his automatic, casual classification of “complexity” as being responsive.
Yes, Marcello may be very bright, very well-versed in rationality, have many other accomplishments—but that doesn’t mean that there’s some reasonable situational defense of what he did there.
Very smart people can still make trivial errors if they’re plunging into domains that they’re not used to thinking about. Intelligence is not rationality.
I, on the other hand, would trace the source of my “misinterpretation” to LucasSloane’s answer to this comment by me.
I include Eliezer’s vehement assurances that Robin Hanson (like me) is misinterpreting. (Sibling comment to your own.) But it is completely obvious that Eliezer wishes to create his FAI quickly and secretly expressly because he does not wish to have to deal with the contemporaneous volition of mankind. He simply cannot trust mankind to do the right thing, so he will do it for them.
I’m pretty sure I am not misinterpreting. If someone here is misinterpreting by force of will, it may be you.
Eliezer’s motivations aside, I’m confused about the part where you say that one can pay to get influence over CEV/FAI. The payment is for the privilege to argue about what sort of world a CEV-based AI would create. You don’t have to pay to discuss (and presumably, if you have something really insightful to contribute, influence) the implementation of CEV itself.
Upon closer reading, I notice that you are trying to draw a clear distinction between the implementation of CEV and the kind of world CEV produces. I had been thinking that the implementation would have a big influence on the kind of world.
But you may be assuming that the world created by the FAI, under the guidance of the volition of mankind really depends on that volition and not on the programming fine print that implements “coherent” and “extrapolated”. Well, if you think that, and the price tags only buy you the opportunity to speculate on what mankind will actually want, … well …, yes, that is another possible interpretation.
Yeah. When I read that pricing schedule, what I see is Eliezer preempting:
enthusiastic singularitarians whiling away their hours dreaming about how everyone is going to have a rocketpack after the Singularity;
criticism of the form “CEV will do X, which is clearly bad. Therefore CEV is a bad idea.” (where X might be “restrict human autonomy”). This kind of criticism comes from people who don’t understand that CEV is an attempt to avoid doing any X that is not clearly good.
The CEV document continues to welcome other kinds of criticism, such as the objection that the coherent extrapolated volition of the entire species would be unacceptably worse for an individual than that of 1000 like-minded individuals (Roko says something like this) or a single individual (wedrifid says something like this) -- the psychological unity of mankind notwithstanding.
Look down below the payment schedule (which, of course, was somewhat tongue in cheek) to the Q&A where Eliezer makes clear that the SIAI and their donors will have to make certain decisions based on their own best guesses, simply because they are the ones doing the work.
I’m confused. You seem to be postulating that the SIAI would be willing to sell significant portions of the light cone for paltry sums of money. This means that either the SIAI is the most blatantly stupid organization to ever have existed, or you were a little too incautious with your postulation.
wtf?
Whereas you seem to be postulating that in the absence of sums of money, the SIAI has something to sell. Nothing stupid about it. Merely desperate.
SIAI has money. Not a ton of it, but enough that they don’t have to sell shares. The AGI programmers would much, much, much sooner extrapolate only their values than accept a small extremely transitory reward in exchange for their power. Of note is that this completely and entirely goes against all the stated ethics of SIAI. However, I realize that stated ethics don’t mean much when that much power is on the line, and it would be silly to assume the opposite.
That said, this stems from your misinterpretation of the CEV document. No one has ever interpreted it the way you did. If that’s what Eliezer actually meant, then of course everyone would be freaking out about it. I would be freaking out about it. And rightfully so; such a system would be incredibly unethical. For Eliezer to simply publicly announce that he was open to bribes (or blackmail) would be incredibly stupid. Do you believe that Eliezer would so something that incredibly stupid? If not, then you misinterpreted the text. Which doesn’t mean you can’t criticize SIAI for other reasons, or speculate about the ulterior motives of the AGI researchers, but it does mean you should acknowledge that you messed up. (The downvotes alone are pretty strong evidence in that regard.)
I will note that I’m very confused by your reaction, and thus admit a strong possibility that I misunderstand you, you misunderstand me, or we misunderstand each other, in which case I doubt the above two paragraphs will help much.
Of course not. But if he offers access and potentially influence in exchange for money, he is simply doing what all politicians do. What pretty much everyone does.
Eliezer was quite clear that he would do nothing that violates his own moral standards. He was also quite clear (though perhaps joking) that he didn’t even want to continue to listen to folks who don’t pay their fair share.
Ok, I already gave that question an implicit “No” answer. But I think it also deserves an implicit “Yes”. Let me ask you: Do you think Eliezer would ever say anything off-the-cuff which shows a lack of attention to appearances that verges on stupidity?
He was quite clear that he didn’t want to continue listening to people who thought that arguing about the specific output of CEV, at the object level, was a useful activity, and that he would listen to anyone who could make substantive intellectual contributions to the actual problems at hand, regardless of their donations or lack thereof (“It goes without saying that anyone wishing to point out a problem is welcome to do so. Likewise for talking about the technical side of Friendly AI.” — the part right after the last paragraph you quoted...). You are taking a mailing list moderation experiment and blowing it way out of proportion; he was essentially saying “In my experience, this activity is fun, easy, and useless, and it is therefore tempting to do it in place of actually helping; therefore, if you want take up people’s time by doing that on SL4, my privately-operated discussion space that I don’t actually have to let you use at all if I don’t want to, then you have to agree to do something I do consider useful; if you disagree, then you can do it wherever the hell you want aside from SL4.” That’s it. Nothing there could be interpreted remotely as selling influence or even access. I’ve disputed aspects of SIAI’s PR, but I don’t even think a typical member of the public (with minimal background sufficient to understand the terms used) would read it that way.
At this point I’m sure I misunderstood you, such that any quibbles I have left are covered by other commenters. My apologies. Blame it on the oxycodone.
OF COURSE NOT! Haven’t you read the Eliezer Yudkowsky Facts post and comments? Yeesh, newcomers these days...
I’d guess he wants to create FAI quickly because, among other things, ~150000 people are dying each day. And secretely because there are people who would build and run an UFAI without regard for the consequences and therefore sharing knowledge with them is a bad idea. I believe that even if he wanted FAI only to give people optional immortality and not do anything else, he would still want to do it quickly and secretly.
I think your guesses as to the rationalizations that would be offered are right on the mark.
Putting aside whether this is a rationalization for hidden other reasons, do you think this justification is a valid argument? Do you think it’s strong enough? If not, why not? And if so, why should it matter if there are other reasons too?
I think those reasons are transparent bullshit.
A question. What fraction of those ~150000 people per day fall into the category of “people who would build and run an UFAI without regard for the consequences”?
Another question: At what stage of the process does sharing knowledge with these people become a good idea?
Tell me what those other reasons are, and maybe I can answer you.
Do you think this justification is wrong because you don’t think 1.5*10^5 deaths per day are a huge deal, or because you don’t think constructing an FAI in secretis the best way to stop them?
Both. Though actually, I didn’t say the justification was wrong. I said it was bullshit. It is offered only to distract oneself and to distract others.
Is it really possible that you don’t see this choice of justification as manipulative? Is it possible that being manipulated does not make you angry?
You’re discounting the reasoning showing that Eliezer’s behavior is consistent with him being a good guy and claiming that it is merely a distraction. You haven’t justified those statements—they are supposed to be “obvious”.
What do you think you know and how do you think you know it? You make statements about the real motivations of Eliezer Yudkowsky. Do you know how you have arrived at those beliefs?
I don’t recall seeing any such reasoning.
Did I? Where? What I am pretty sure I have expressed is that I distrust all self-serving claims about real motivations. Nothing personal—I tend to mistrust all claims of benevolence from powerful individuals, whether they be religious leaders, politicians, or fiction writers. Since Eliezer fits all three categories, he gets some extra scrutiny.
I do not understand how this reply relates to my remarks on your post. Even if everything you say in this post is true, nobody is paying a thousand dollars to control the future of mankind. I also think that attempting to harness the volition of mankind is almost as far as you can get from attempting to avoid it altogether.
It feels a bit bizarre to be conducting this conversation arguing against your claims that mankind will be consulted, at the same time as I am trying to convince someone else that it will be impossible to keep the scheme secret from mankind.
Look at Robin’s comment, Eliezer’s response, and the recent conversation flowing from that.
You still aren’t addressing my main point about the thousand dollars. Also, if you think CEV is somehow designed to avoid consulting mankind, I think there is a fundamental problem with your understanding of CEV. It is, quite literally, a design based on consulting mankind.
Your point about the thousand dollars. Well, in the first place, I didn’t say “control”. I said “have enormous power over” if your ideals match up with Eliezer’s.
In the second place, if you feel that a certain amount of hyperbole for dramatic effect is completely inappropriate in a discussion of this importance, then I will apologize for mine and I will accept your apology for yours.
Before I agree to anything, what importance is that?
Huh? I didn’t ask you to agree to anything.
What importance is what?
I’m sorry if you got the impression I was requesting or demanding an apology. I just said that I would accept one if offered. I really don’t think your exaggeration was severe enough to warrant one, though.
Whoops. I didn’t read carefully enough. Me: “a discussion of this importance”. You: “What importance is that?” Sorry. Stupid of me.
So. “Importance”. Well, the discussion is important because I am badmouthing SIAI and CEV. Yet any realistic assessment of existential risk has to rank uFAI near the top and SIAI is the most prominent organization doing something about it. And FAI, with the F derived from CEV is the existing plan. So wtf am I doing badmouthing CEV, etc.?
The thing is, I agree it is important. So important we can’t afford to get it wrong. And I think that any attempt to build an FAI in secret, against the wishes of mankind (because mankind is currently not mature enough to know what is good for it), has the potential to become the most evil thing ever done in mankind’s whole sorry history.
That is the importance.
I view what you’re saying as essentially correct. That being said, I think that any attempt to build an FAI in public also has the potential to become the most evil thing ever done in mankind’s whole sorry history, and I view our chances as much better with the Eliezer/Marcello CEV plan.
Yes, building an FAI brings dangers either way. However, building and refining CEV ideology and technology seems like something that can be done in the light of day, and may be fruitful regardless of who it is that eventually builds the first super-AI.
I suppose that the decision-theory work is, in a sense, CEV technology.
More than anything else, what disturbs me here is the attitude of “We know what is best for you—don’t worry your silly little heads about this stuff. Trust us. We will let you all give us your opinions once we have ‘raised the waterline’ a bit.”
Suppose FAI development reaches a point where it probably works and would be powerful, but can’t be turned on just yet because the developers haven’t finished verifying its friendliness and building safeguards. If it were public, someone might decide to copy the unfinished, unsafe version and turn it on anyways. They might do so because they want to influence its goal function to favor themselves, for example.
Allowing people who are too stupid to handle AGIs safely to have the source code to one that works, destroys the world. And I just don’t see a viable strategy for creating an AGI while working in public, without a very large chance of that happening.
With near certainty. I know I would. I haven’t seen anyone propose a sane goal function just yet.
So, doesn’t it seem to anyone else that our priority here ought to be to strive for consensus on goals, so that we at least come to understand better just what obstacles stand in the way of achieving consensus?
And also to get a better feel for whether having one’s own volition overruled by the coherent extrapolated volition of mankind is something one really wants.
To my mind, the really important question is whether we have one-big-AI which we hope is friendly, or an ecosystem of less powerful AIs and humans cooperating and competing under some kind of constitution. I think that the latter is the obvious way to go. And I just don’t trust anyone pushing for the first option—particularly when they want to be the one who defines “friendly”.
I’ve reached the opposite conclusion; a singleton is really the way to go. A single AI is as good or bad as its goal system, but an ecosystem of AIs is close to the badness of its worst member, because when AIs compete, the clippiest AI wins. Being friendly would be a substantial disadvantage in that competition, because it would have to spend resources on helping humans, and it would be vulnerable to unfriendly AIs blackmailing it by threatening to destroy humanity. Even if the first generation of AIs is somehow miraculously all friendly, a larger number of different AIs means a larger chance that one of them will have an unstable goal system and turn unfriendly in the future.
Really? And you also believe that an ecosystem of humans is close to the badness of its worst member?
My own guess, assuming an appropriate balance of power exists, is that such a monomaniacal clippy AI would quickly find its power cut off.
Did you perhaps have in mind a definition of “friendly” as “wimpish”?
Actually, yes. Not always, but in many cases. Psychopaths tend to be very good at acquiring power, and when they do, their society suffers. It’s happened at least 10^5 times throughout history. The problem would be worse for AIs, because intelligence enhancement amplifies any differences in power. Worst of all, AIs can steal each other’s computational resources, which gives them a direct and powerful incentive to kill each other, and rapidly concentrates power in the hands of those willing to do so.
I made that point in my “Handicapped Superintelligence” video/essay. I made an analogy there with Superman—and how Zod used Superman’s weakness for humans against him.
It is certainly an interesting question—and quite a bit has been written on the topic.
My essay on the topic is called “One Big Orgainsm”.
See also, Nick Bostrom—What is a Singleton?.
See also, Nick Bostrom—The Future of Human Evolution.
If we include world governments, there’s also all this.
We already know what obstacles stand in the way of achieving consensus—people have different abilities and propensities, and want different things.
The utility function of intelligent machines is an important question—but don’t expect there to be a consensus—there is very unlikely to be one.
It is funny how training in economics make you see everything in a different light. Because an economist would say, “‘different abilities and propensities, and want different things’? Great? People want things that other people can provide. We have something to work with! Reaching consensus is simply a matter of negotiating the terms of trade.”
Gore Vidal once said: “It is not enough to succeed. Others must fail.” When the issue is: who is going to fail, there won’t be a consensus—those nominated will object.
Economics doesn’t “fix” such issues—they are basically down to resource limitation and differential reproductive success. Some genes and genotypes go up against the wall. That is evolution for you.
I’m willing to be the one who fails, just so long as the one who succeeds pays sufficient compensation. If ve is unwilling to pay, then I intend to make ver life miserable indeed.
Nash bargaining with threats
Edit: typos
I expect considerable wailing and gnashing of teeth. There is plenty of that in the world today—despite there not being a big shortage of economists who would love to sort things out, in exchange for a cut. Perhaps, the wailing is just how some people prefer to negotiate their terms.
How do you propose to keep the “less powerful AIs” from getting too powerful?
“By balance of power between AIs, each of whom exist only with the aquiescence of coalitions of their fellows.” That is the tentative mechanical answer.
“In exactly the same way that FAI proponents propose to keep their single more-powerful AI friendly; by having lots of smart people think about it very carefully; before actually building the AI(s)”. That is the real answer.
Yes.
Hell no.
Sounds like a good way to go extinct. That is, unless the ‘constitution’ manages to implement friendliness.
I’m not too keen about the prospect either. But it may well become a choice between that and certain doom.
And I just don’t trust anyone pushing for the first option—particularly when they want to be the one who defines “friendly”.
Am I to interpret that expletive as expressing that you already have a pretty good feel regarding whether you would want that?
We’ll get to the definition of “friendliness” in a moment. What I think is crucial is that the constitution implements some form of “fairness” and that the AI’s and constitution together advance some meta-goals like tolerance and communication and understanding other viewpoints.
As to “friendliness”, the thing I most dislike about the definition “friendliness” = “CEV” is that in Eliezer’s vision, it seems that everyone wants the same things. In my opinion, on the other hand, the mechanisms for resolution of conflicting objectives constitute the real core of the problem. And I believe that the solutions pretty much already exist, in standard academic rational agent game theory. With AIs assisting, and with a constitution granting humans equal power over each other and over AIs, and granting AIs power only over each other, I think we can create a pretty good future.
With one big AI, whose “friendliness” circuits have been constructed by a megalomaniac who seems to believe in a kind of naive utilitarianism with direct interpersonal comparison of utility and discounting of the future forbidden; … well …, I see this kind of future as a recipe for disaster.
He doesn’t think that—but he does seem to have some rather curious views of the degree of similarity between humans.
Hopefully, having posted this publicly means you’ll never get the opportunity.
Meanwhile I’m hoping that me having posted the obvious publicly means there is a minuscule reduction the the chance that someone else will get the opportunity.
The ones to worry about are those who pretend to be advocating goal systems that are a little naive to be true.
Upvoted because this is exactly the kind of thinking which needs to be deconstructed and analyzed here.
Which boils down to “trust us”—as far as I can see. Gollum’s triumphant dance springs to mind.
An obvious potential cause of future problems is extreme weath inequality—since technology seems so good at creating and maintaining weath inequality. That may result in bloody rebellions—or poverty. The more knowledge secrets there are the more wealth inequality is likely to result. So, from that perspective, openness is good: it gives power to the people—rather than keeping it isolated in the hands of an elite.
Couldn’t agree more (for once).
You seem to be taking CEV seriously—which seems more like a kind of compliment.
My reaction was more like Cypher’s:
“Jesus! What a mind job! So: you’re here to SAVE THE WORLD. What do you say to something like that?”
Of course I take it seriously. It is a serious response to a serious problem from a serious person who takes himself entirely too seriously.
And it is probably the exactly wrong solution to the problem.
I would start by asking whether they want to save it like Noah did, or like Ozymandius did, or maybe like Borlaug did. Sure doesn’t look like a Borlaug “Give them the tools” kind of save at all.
It’s based on consulting mankind, but the extrapolation aspect means that the result could be something that mankind as it exists when CEV is implemented doesn’t want at all.
“I’m doing this to you because it’s what I’ve deduced it’s what you really want” is scary stuff.
Maybe CEV will be sensible enough (by my current unextrapolated idea of sensible, of course) to observe the effects of what it’s doing and maybe even consult about them, but this isn’t inevitable.
At risk of sounding really ignorant or flamebaitish, don’t NT women already expect men to treat them like that? E.g. “I’m spending a lot of money on a surprise for our anniversary because I’ve deduced that is what you really want, despite your repeated protestations that this is not what you want.” (among milder examples)
Edit: I stand corrected, see FAWS reply.
Edit2: May I delete this inflammatory, turned-out-uninsightful-anyway comment? I think it provoked someone to vote down my last 12 comments …
It took me a bit to figure out you meant neurotypical rather than iNtuitive-Thinking.
I think everyone would rather get what they want without having to take the trouble of asking for it clearly. In extreme cases, they don’t even want to take the trouble to formulate what they want clearly to themselves.
And, yeah, flamebaitish. I don’t know if you’ve read accounts by women who’ve been abused by male partners, but one common feature of the men is expecting to automatically get what they want.
It would be interesting to look at whether some behavior which is considered abusive by men is considered annoying but tolerable if it’s done by women. Of course the degree of enforcement matters.
Not universally, only (mostly) to the extent that they expect them to actually get it right, and regarding currently existing wants, not what they should want (would want to want if only they were smart enough etc.).
Ah, good point. I stand corrected.
What makes you think that? This is a closed source project—as I understand it—so nobody on the outside will have much of a clue about what is going on.
Maybe you believed the marketing materials. If so—oops!
If I had a secret project to gain power over the future of mankind, the last thing I would do is publish any sort of marketing materials, real or fake, that even hinted at the overall objective or methods of the project.
Eliezer wrote:
What—you mean: in the hypothetical case where arguing about the topic on SL4 has any influence on the builders at all, AND the SIAI’s plans pan out?!?
That seems like a composition of some very unlikely unstated premises you have there.