It’d be rather easy to twist my words here, but in the case of extrapolated volition it’s not like one person gaining the power over the future of mankind is a dystopia or anything.
Let’s posit a world where Marcello goes rogue and compiles Marcello-extrapolating AGI. (For anyone who doesn’t know Marcello, he’s a super awesome guy.) I bet that the resultant universe wouldn’t be horrible. Extrapolated-Marcello probably cares about the rest of humanity about as much as extrapolated-humanity does. As humans get smarter and wiser it seems they have a greater appreciation for tolerance, diversity, and all those other lovey-dovey liberal values we implicitly imagine to be the results of CEV. It is unlikely that the natural evolution of ‘moral progress’ as we understand it will lead to Marcello’s extrapolated volition suddenly reversing the the trend and deciding that all other human beings on Earth are essentially worthless and deserve to be stripped to atoms to be turned into a giant Marcello-Pleasure-Simulating Computer. (And even if it did, I believe humans in general are probabilistically similar enough to Marcello that they would count this as positive sum if they knew more and thought faster; but that’s a more intricate philosophical argument that I’d rather not defend here.) There are some good arguments to be made for the psychic diversity of mankind, but I doubt the degree of that diversity is enough to tip the scales of utility from positive to negative. Not when diversity is something we seem to have come to appreciate more over time.
This intuition is the result of probably-flawed but at least causal and tractable reasoning about trends in moral progress and the complexity and diversity of human goal structures. It seems that too often when people guess what the results of extrapolated volition will be they use it as a chance to profess and cheer, not carefully predict.
(This isn’t much of a response to anything you wrote, katydee; apologies for that. Didn’t know where to put it.)
The scariest kind of dream, perhaps, is exemplified by someone with merely human intelligence who wants to hastily rewrite their own values to conform to their favorite ideology. We’d want an implementation of CEV to recognize this as a bad step in extrapolation. The question is, how do we define what is a “bad step”?
Of note is that you’re smart and yet just made the fundamental attribution error.
Meaning Marcello’s at-the-time reasonable suggestion of “complexity” as a solution (something I’ve never done in my life, due to understanding the difference between means and ends) was mainly the result of the unfortunate, disadvantageous position he found himself in, rather than a failure to recognize what counts as understanding and solving a problem?
I meant it more generally. You’re seeing one tiny slice of a person’s history almost certainly caused by an uncharacteristic lapse in judgment and using it to determine their personality traits when you have strong countervailing evidence that SIAI has a history of only employing the very brightest people. Indeed, here Eliezer mentioned that Marcello worked on a math problem with John Conway: The Level Above Mine. Implied by the text is that Eliezer believes Marcello to be close enough to Eliezer’s level to be able to roughly judge Eliezer’s intelligence. Since we all know how much of an arrogant bastard Eliezer is, this speaks well of Marcello’s cognitive abilities.
Eliezer wrote a post about a time long ago when a not-yet-rationalist Marcello said something dumb. Not a time when he persisted in being dumb even, just said a dumb thing. There’s a huge selection effect. Eliezer would never mention Marcello getting something simple right. At his level it’s expected. Even so, everyone has a dumb moment now and then, and those are the ones we learn from. It’s just that Marcello’s happened to be worked into an instructional blog post. Marcello is still a brilliant thinker. (I really wish he’d contribute to Less Wrong more, but he’s busy studying and what not.)
Anyway, this isn’t really about Marcello; everyone who knows him knows he’s hella smart, and there’s really no reason to defend him. It’s about taking all the evidence into account and giving people the benefit of the doubt when the evidence suggests it.
Eliezer wrote a post about a time long ago when a not-yet-rationalist Marcello said something dumb.
Well, that’s the key thing for me—not “How smart is Marcello now?”, but how many people were at least at Marcello’s level at that time, yet not patiently taken under EY’s wing and given his precious time?
EY is astounded that someone can understand this after a thorough explanation. Can it honestly be that hard to find someone who can follow that? Read the passage:
“Okay,” I said, “saying ‘complexity’ doesn’t concentrate your probability mass.”
“Oh,” Marcello said, “like ‘emergence’. Huh. So… now I’ve got to think about how X might actually happen...”
That was when I thought to myself, “Maybe this one is teachable.” [bold in original]
It’s like he’s saying that being able to follow that explanation somehow makes you stand out among the people he talks to.
How would that compare to a potential student who could have given EY’s explanation, instead of needing it?
EY is astounded that someone can understand this after a thorough explanation. Can it honestly be that hard to find someone who can follow that?
Yes. Nobody arrives from the factory with good rationality skills, so I look for learning speed. Compare the amount I had to argue with Marcello in the anecdote to the amount that other people are having to argue with you in this thread.
Compare the amount I had to argue with Marcello in the anecdote to the amount that other people are having to argue with you in this thread.
What trivial thing am I slow(er) to learn here? Or did you mean some other comparison?
(FWIW, I know a visiting fellow who took 18 months to be convinced of something trivial, after long explanations from several SIAI greats … hence my confusion about how the detector works.)
As far as I know, Eliezer has never had anything to do with choices for Visiting Fellowship. As you know but some people on Less Wrong seem not to, Eliezer doesn’t run SIAI. (In reality, SIAI is a wonderful example of the great power of emergence, and is indeed the first example of a superintelligent organization.) (Just kidding.)
Right, but it seemed you were comparing the selection criteria for Visiting Fellowship and the selection criteria for Eliezer’s FAI team, which will of course be very different. Perhaps I misunderstood. I’ve been taking oxycodone every few hours for a lot of hours now.
That Marcello’s “lapse” is only very weak evidence against the proposition that his IQ is exceptionally high (even among the “aspiring rationalist” cluster).
What lapse? People don’t know these things until I explain them! Have you been in a mental state of having-already-read-LW for so long that you’ve forgotten that no one from outside would be expected to spontaneously describe in Bayesian terms the problem with saying that “complexity” explains something? Someone who’d already invented from scratch everything I had to teach wouldn’t be taken as an apprentice, they’d already be me! And if they were 17 at the time then I’d probably be working for them in a few years!
What lapse? People don’t know these things until I explain them!
A little over-the-top there. People can see the problem with proposing “complexity” as a problem-solving approach without having read your work. I hadn’t yet read your work on Bayescraft when I saw that article, and I still cringed as I read Marcello’s response—I even remember previous encounters where people had proposed “solutions” like that, though I’d perhaps explain the error differently.
It is a lapse to regard “complexity” as a problem-solving approach, even if you are unfamiliar with Bayescraft, and yes, even if you are unfamiliar with the Chuck Norris of thinking.
Have you been in a mental state of having-already-read-LW for so long that you’ve forgotten that no one from outside would be expected to spontaneously describe in Bayesian terms the problem with saying that “complexity” explains something?
Seriously? What sort of outside-LW people do you talk to? I’m a PhD student in a fairly mediocre maths department, and I’m pretty sure everyone in the room I’m in right now would call me out on it if I tried to use the word “complexity” in the context Marcelo did there as if it actually meant something, and for essentially the right reason. This might be a consequence of us being mathematicians, and so used to thinking in formalism, but there are an awful lot of professional mathematicians out there who haven’t read anything written by Eliezer Yudkowsky.
I’m sorry but “there’s got to be some amount of complexity that does it.” is just obviously meaningless. I could have told you this long before I read the sequences, and definitely when I was 17. I think you massively underestimate the rationality of humanity.
Thanks for spelling that out, because it wasn’t my argument, which I clarified in the follow-up discussion. (And I think it would be more accurate to say that it’s strong evidence, just outweighed by stronger existing evidence in this case.)
My surprise was with how rare EY found it to meet someone who could follow that explanation—let alone need the explanation. A surprise that, it turns out, is shared by the very person correcting my foolish error.
Can we agree that the comparison EY just made isn’t accurate?
(And I think it would be more accurate to say that it’s strong evidence, just outweighed by stronger existing evidence in this case.)
This is where you commit the fundamental attribution error.
My surprise was with how rare EY found it to meet someone who could follow that explanation—let alone need the explanation. A surprise that, it turns out, is shared by the very person correcting my foolish error.
I don’t actually think this has been written about much here, but there is a tendency among high-IQ folks to underestimate how rare their abilities are. The way they do this is not by underestimating their own cognitive skills, but instead by overestimating those of most people.
In other words, what it feels like to be a genius is not that you’re really smart, but rather that everyone else is really dumb.
I would expect that both you and Will would see the light on this if you spent some more time probing the thought processes of people of “normal” intelligence in detail, e.g. by teaching them mathematics (in a setting where they were obliged to seriously attempt to learn it, such as a college course; and where you were an authority figure, such as the instructor of such a course).
Can we agree that the comparison EY just made isn’t accurate?
Probably not literally, in light of your clarification. However, I nevertheless suspect that your responses in this thread do tend to indicate that you would probably not be particularly suited to being (for example) EY’s apprentice—because I suspect there’s a certain...docility that someone in that position would need, which you don’t seem to possess. Of course that’s a matter of temperament more than intelligence.
This is where you commit the fundamental attribution error.
I’m missing something here, I guess. What fraction of people who, as a matter of routine, speak of “complexity” as a viable problem-attack method, and are also very intelligent? If it’s small, then it’s appropriate to say, as I suggested, that it’s strong evidence, even as it might be outweighed by something else in this case. Either way, I’m just not seeing how I’m, per the FEA, failing to account for some special situational justification for what Marcello did.
I would expect that both you and Will would see the light on this if you spent some more time probing the thought processes of people of “normal” intelligence in detail, e.g. by teaching them mathematics (in a setting where they were obliged to seriously attempt to learn it, such as a college course; and where you were an authority figure, such as the instructor of such a course).
Well, I do admit to having experienced disenchantment upon learning where the average person is on analytical capability (Let’s not forget where I live...) Still, I don’t think teaching math would prove it to me. As I say here ad infinitum, I just don’t find it hard to explain topics I understand—I just trace back to the nepocu (nearest point of common understanding), correct their misconceptions, and work back from there. So in all my experience with explaining math to people who e.g. didn’t complete high school, I’ve never had any difficulty.
For the past five years I’ve helped out with math in a 4th grade class in a poorer school district, and I’ve never gotten frustrated at a student’s stupidity—I just teach whatever they didn’t catch in class, and fix the misunderstanding relatively quickly. (I don’t know if the age group breaks the criteria you gave).
However, I nevertheless suspect that your responses in this thread do tend to indicate that you would probably not be particularly suited to being (for example) EY’s apprentice
Eh, I wasn’t proposing otherwise—I’ve embarassed myself here far too many times to be regarded as someone that group would want to work with in person. Still, I can be perplexed at what skills they regard as rare.
This is where you commit the fundamental attribution error.
I’m missing something here, I guess. What fraction of people who, as a matter of routine, speak of “complexity” as a viable problem-attack method, and are also very intelligent? If it’s small, then it’s appropriate to say, as I suggested, that it’s strong evidence, even as it might be outweighed by something else in this case. Either way, I’m just not seeing how I’m, per the FEA, failing to account for some special situational justification for what Marcello did.
I would expect that both you and Will would see the light on this if you spent some more time probing the thought processes of people of “normal” intelligence in detail, e.g. by teaching them mathematics (in a setting where they were obliged to seriously attempt to learn it, such as a college course; and where you were an authority figure, such as the instructor of such a course).
Well, I do admit to having experienced disenchantment upon learning where the average person is (and let’s not forget where I live...) Still, I don’t think teaching math would make the point. As I say here ad infinitum, I just don’t find it hard to explain topics I understand—I just trace back to the nepocu (nearest point of common understanding), correct their misconceptions, and work back from there. So in all my experience with explaining math to people who e.g. didn’t complete high school, I’ve never had any difficulty.
For the past five years I’ve helped out with math in a 4th grade class in a poorer school district, and I’ve never gotten frustrated at a student’s stupidity—I just teach whatever they didn’t catch in class, and fix the misunderstanding relatively quickly. (I don’t know if the age group breaks the criteria you gave).
However, I nevertheless suspect that your responses in this thread do tend to indicate that you would probably not be particularly suited to being (for example) EY’s apprentice
Eh, I wasn’t proposing otherwise—I’ve embarassed myself here far too many times to be regarded as someone that group would want to work with in person. Still, I can be perplexed at what skills they regard as rare.
Okay, I guess I missed what you were implicitly curious about.
Well, that’s the key thing for me—not “How smart is Marcello now?”, but how many people were at least at Marcello’s level at that time, yet not patiently taken under EY’s wing and given his precious time?
At the time there wasn’t a Visiting Fellows program or the like (I think), and there were a lot fewer potential FAI researchers then than now. However, I get the impression that Marcello was and is an exceptional rationalist. ’Course, I share your confusion that Eliezer would be so impressed by what in hindsight like such a simple application of previously learned knowledge. I think Eliezer (probably unconsciously) dramatized his whole recollection quite a bit. Or it’s possible he’d almost completely lost faith in humanity at that point—it seems he was talking to crazy wannabe AGI researchers all the time, after all. That said, since my model is producing lots of seemingly equally plausible explanations, it’s not a very good model. I’m confused.
Still, I think Marcello was and is exceptionally talented. The post is just a really poor indicator of that.
No, it was “a failure to [immediately] recognize what counts as understanding and solving a [particular] problem”, but that is a rationality skill, and is not entirely a function of a person’s native general intelligence. Having a high g gives you an advantage in learning and/or independently inventing rationality skills, but not always enough of an advantage. History is littered with examples of very smart people committing rationality failures much larger than postulating “complexity” as a solution to a problem.
His mistake was entirely situational, given the fact that he understood a minute later what he had done incorrectly and probably rarely or never made that mistake again.
I don’t want to drag this out, but I think you’re going too far in your defense of the reasonableness of this error:
His mistake was entirely situational, given the fact that he understood a minute later what he had done incorrectly and probably rarely or never made that mistake again.
Read the exchange: He understood the error because someone else had to explain it to him over a wide inferential gap. If it were just, “Hey, complexity isn’t a method” “Oh, right—scratch that”, then you would be correct, but that’s not what happened. EY had to trace the explanation back to an earlier essay and elaborate on the relationship between those concepts and what Marcello just tried to do.
Also, I don’t see the point in distinguishing g and rationality here—somehow, Marcello got to that point without recognizing the means/ends distinction EY talks about on his own. Yes, it’s a rationality skill that can be taught, but failing to recognize it in the first place does say something about how fast you would pick up rationalist concepts in general.
Third, I never criticized his failure to immediately solve the problem, just his automatic, casual classification of “complexity” as being responsive.
Yes, Marcello may be very bright, very well-versed in rationality, have many other accomplishments—but that doesn’t mean that there’s some reasonable situational defense of what he did there.
Very smart people can still make trivial errors if they’re plunging into domains that they’re not used to thinking about. Intelligence is not rationality.
It’d be rather easy to twist my words here, but in the case of extrapolated volition it’s not like one person gaining the power over the future of mankind is a dystopia or anything.
Let’s posit a world where Marcello goes rogue and compiles Marcello-extrapolating AGI. (For anyone who doesn’t know Marcello, he’s a super awesome guy.) I bet that the resultant universe wouldn’t be horrible. Extrapolated-Marcello probably cares about the rest of humanity about as much as extrapolated-humanity does. As humans get smarter and wiser it seems they have a greater appreciation for tolerance, diversity, and all those other lovey-dovey liberal values we implicitly imagine to be the results of CEV. It is unlikely that the natural evolution of ‘moral progress’ as we understand it will lead to Marcello’s extrapolated volition suddenly reversing the the trend and deciding that all other human beings on Earth are essentially worthless and deserve to be stripped to atoms to be turned into a giant Marcello-Pleasure-Simulating Computer. (And even if it did, I believe humans in general are probabilistically similar enough to Marcello that they would count this as positive sum if they knew more and thought faster; but that’s a more intricate philosophical argument that I’d rather not defend here.) There are some good arguments to be made for the psychic diversity of mankind, but I doubt the degree of that diversity is enough to tip the scales of utility from positive to negative. Not when diversity is something we seem to have come to appreciate more over time.
This intuition is the result of probably-flawed but at least causal and tractable reasoning about trends in moral progress and the complexity and diversity of human goal structures. It seems that too often when people guess what the results of extrapolated volition will be they use it as a chance to profess and cheer, not carefully predict.
(This isn’t much of a response to anything you wrote, katydee; apologies for that. Didn’t know where to put it.)
That “best self” thing makes me nervous. People have a wide range of what they think they ought to be, and some of those dreams are ill-conceived.
The scariest kind of dream, perhaps, is exemplified by someone with merely human intelligence who wants to hastily rewrite their own values to conform to their favorite ideology. We’d want an implementation of CEV to recognize this as a bad step in extrapolation. The question is, how do we define what is a “bad step”?
Is he the same guy whose trivial error inspired this post?
If so, just how smart do you have to be in order to be SIAI material?
Yes.
Ridiculously smart, as I’m sure you can guess. Of note is that you’re smart and yet just made the fundamental attribution error.
Meaning Marcello’s at-the-time reasonable suggestion of “complexity” as a solution (something I’ve never done in my life, due to understanding the difference between means and ends) was mainly the result of the unfortunate, disadvantageous position he found himself in, rather than a failure to recognize what counts as understanding and solving a problem?
I meant it more generally. You’re seeing one tiny slice of a person’s history almost certainly caused by an uncharacteristic lapse in judgment and using it to determine their personality traits when you have strong countervailing evidence that SIAI has a history of only employing the very brightest people. Indeed, here Eliezer mentioned that Marcello worked on a math problem with John Conway: The Level Above Mine. Implied by the text is that Eliezer believes Marcello to be close enough to Eliezer’s level to be able to roughly judge Eliezer’s intelligence. Since we all know how much of an arrogant bastard Eliezer is, this speaks well of Marcello’s cognitive abilities.
Eliezer wrote a post about a time long ago when a not-yet-rationalist Marcello said something dumb. Not a time when he persisted in being dumb even, just said a dumb thing. There’s a huge selection effect. Eliezer would never mention Marcello getting something simple right. At his level it’s expected. Even so, everyone has a dumb moment now and then, and those are the ones we learn from. It’s just that Marcello’s happened to be worked into an instructional blog post. Marcello is still a brilliant thinker. (I really wish he’d contribute to Less Wrong more, but he’s busy studying and what not.)
Anyway, this isn’t really about Marcello; everyone who knows him knows he’s hella smart, and there’s really no reason to defend him. It’s about taking all the evidence into account and giving people the benefit of the doubt when the evidence suggests it.
Well, that’s the key thing for me—not “How smart is Marcello now?”, but how many people were at least at Marcello’s level at that time, yet not patiently taken under EY’s wing and given his precious time?
EY is astounded that someone can understand this after a thorough explanation. Can it honestly be that hard to find someone who can follow that? Read the passage:
It’s like he’s saying that being able to follow that explanation somehow makes you stand out among the people he talks to.
How would that compare to a potential student who could have given EY’s explanation, instead of needing it?
Yes. Nobody arrives from the factory with good rationality skills, so I look for learning speed. Compare the amount I had to argue with Marcello in the anecdote to the amount that other people are having to argue with you in this thread.
What trivial thing am I slow(er) to learn here? Or did you mean some other comparison?
(FWIW, I know a visiting fellow who took 18 months to be convinced of something trivial, after long explanations from several SIAI greats … hence my confusion about how the detector works.)
As far as I know, Eliezer has never had anything to do with choices for Visiting Fellowship. As you know but some people on Less Wrong seem not to, Eliezer doesn’t run SIAI. (In reality, SIAI is a wonderful example of the great power of emergence, and is indeed the first example of a superintelligent organization.) (Just kidding.)
But he has significant discretion over who he takes as an apprentice, irrespective of what SIAI leadership might do.
Right, but it seemed you were comparing the selection criteria for Visiting Fellowship and the selection criteria for Eliezer’s FAI team, which will of course be very different. Perhaps I misunderstood. I’ve been taking oxycodone every few hours for a lot of hours now.
That Marcello’s “lapse” is only very weak evidence against the proposition that his IQ is exceptionally high (even among the “aspiring rationalist” cluster).
What lapse? People don’t know these things until I explain them! Have you been in a mental state of having-already-read-LW for so long that you’ve forgotten that no one from outside would be expected to spontaneously describe in Bayesian terms the problem with saying that “complexity” explains something? Someone who’d already invented from scratch everything I had to teach wouldn’t be taken as an apprentice, they’d already be me! And if they were 17 at the time then I’d probably be working for them in a few years!
A little over-the-top there. People can see the problem with proposing “complexity” as a problem-solving approach without having read your work. I hadn’t yet read your work on Bayescraft when I saw that article, and I still cringed as I read Marcello’s response—I even remember previous encounters where people had proposed “solutions” like that, though I’d perhaps explain the error differently.
It is a lapse to regard “complexity” as a problem-solving approach, even if you are unfamiliar with Bayescraft, and yes, even if you are unfamiliar with the Chuck Norris of thinking.
Seriously? What sort of outside-LW people do you talk to? I’m a PhD student in a fairly mediocre maths department, and I’m pretty sure everyone in the room I’m in right now would call me out on it if I tried to use the word “complexity” in the context Marcelo did there as if it actually meant something, and for essentially the right reason. This might be a consequence of us being mathematicians, and so used to thinking in formalism, but there are an awful lot of professional mathematicians out there who haven’t read anything written by Eliezer Yudkowsky.
I’m sorry but “there’s got to be some amount of complexity that does it.” is just obviously meaningless. I could have told you this long before I read the sequences, and definitely when I was 17. I think you massively underestimate the rationality of humanity.
Scarequotes added. :-)
Thanks for spelling that out, because it wasn’t my argument, which I clarified in the follow-up discussion. (And I think it would be more accurate to say that it’s strong evidence, just outweighed by stronger existing evidence in this case.)
My surprise was with how rare EY found it to meet someone who could follow that explanation—let alone need the explanation. A surprise that, it turns out, is shared by the very person correcting my foolish error.
Can we agree that the comparison EY just made isn’t accurate?
This is where you commit the fundamental attribution error.
I don’t actually think this has been written about much here, but there is a tendency among high-IQ folks to underestimate how rare their abilities are. The way they do this is not by underestimating their own cognitive skills, but instead by overestimating those of most people.
In other words, what it feels like to be a genius is not that you’re really smart, but rather that everyone else is really dumb.
I would expect that both you and Will would see the light on this if you spent some more time probing the thought processes of people of “normal” intelligence in detail, e.g. by teaching them mathematics (in a setting where they were obliged to seriously attempt to learn it, such as a college course; and where you were an authority figure, such as the instructor of such a course).
Probably not literally, in light of your clarification. However, I nevertheless suspect that your responses in this thread do tend to indicate that you would probably not be particularly suited to being (for example) EY’s apprentice—because I suspect there’s a certain...docility that someone in that position would need, which you don’t seem to possess. Of course that’s a matter of temperament more than intelligence.
I’m missing something here, I guess. What fraction of people who, as a matter of routine, speak of “complexity” as a viable problem-attack method, and are also very intelligent? If it’s small, then it’s appropriate to say, as I suggested, that it’s strong evidence, even as it might be outweighed by something else in this case. Either way, I’m just not seeing how I’m, per the FEA, failing to account for some special situational justification for what Marcello did.
Well, I do admit to having experienced disenchantment upon learning where the average person is on analytical capability (Let’s not forget where I live...) Still, I don’t think teaching math would prove it to me. As I say here ad infinitum, I just don’t find it hard to explain topics I understand—I just trace back to the nepocu (nearest point of common understanding), correct their misconceptions, and work back from there. So in all my experience with explaining math to people who e.g. didn’t complete high school, I’ve never had any difficulty.
For the past five years I’ve helped out with math in a 4th grade class in a poorer school district, and I’ve never gotten frustrated at a student’s stupidity—I just teach whatever they didn’t catch in class, and fix the misunderstanding relatively quickly. (I don’t know if the age group breaks the criteria you gave).
Eh, I wasn’t proposing otherwise—I’ve embarassed myself here far too many times to be regarded as someone that group would want to work with in person. Still, I can be perplexed at what skills they regard as rare.
I’m missing something here, I guess. What fraction of people who, as a matter of routine, speak of “complexity” as a viable problem-attack method, and are also very intelligent? If it’s small, then it’s appropriate to say, as I suggested, that it’s strong evidence, even as it might be outweighed by something else in this case. Either way, I’m just not seeing how I’m, per the FEA, failing to account for some special situational justification for what Marcello did.
Well, I do admit to having experienced disenchantment upon learning where the average person is (and let’s not forget where I live...) Still, I don’t think teaching math would make the point. As I say here ad infinitum, I just don’t find it hard to explain topics I understand—I just trace back to the nepocu (nearest point of common understanding), correct their misconceptions, and work back from there. So in all my experience with explaining math to people who e.g. didn’t complete high school, I’ve never had any difficulty.
For the past five years I’ve helped out with math in a 4th grade class in a poorer school district, and I’ve never gotten frustrated at a student’s stupidity—I just teach whatever they didn’t catch in class, and fix the misunderstanding relatively quickly. (I don’t know if the age group breaks the criteria you gave).
Eh, I wasn’t proposing otherwise—I’ve embarassed myself here far too many times to be regarded as someone that group would want to work with in person. Still, I can be perplexed at what skills they regard as rare.
What was the trivial thing? Just curious.
Answering via PM.
Okay, I guess I missed what you were implicitly curious about.
At the time there wasn’t a Visiting Fellows program or the like (I think), and there were a lot fewer potential FAI researchers then than now. However, I get the impression that Marcello was and is an exceptional rationalist. ’Course, I share your confusion that Eliezer would be so impressed by what in hindsight like such a simple application of previously learned knowledge. I think Eliezer (probably unconsciously) dramatized his whole recollection quite a bit. Or it’s possible he’d almost completely lost faith in humanity at that point—it seems he was talking to crazy wannabe AGI researchers all the time, after all. That said, since my model is producing lots of seemingly equally plausible explanations, it’s not a very good model. I’m confused.
Still, I think Marcello was and is exceptionally talented. The post is just a really poor indicator of that.
No, it was “a failure to [immediately] recognize what counts as understanding and solving a [particular] problem”, but that is a rationality skill, and is not entirely a function of a person’s native general intelligence. Having a high g gives you an advantage in learning and/or independently inventing rationality skills, but not always enough of an advantage. History is littered with examples of very smart people committing rationality failures much larger than postulating “complexity” as a solution to a problem.
His mistake was entirely situational, given the fact that he understood a minute later what he had done incorrectly and probably rarely or never made that mistake again.
I don’t want to drag this out, but I think you’re going too far in your defense of the reasonableness of this error:
Read the exchange: He understood the error because someone else had to explain it to him over a wide inferential gap. If it were just, “Hey, complexity isn’t a method” “Oh, right—scratch that”, then you would be correct, but that’s not what happened. EY had to trace the explanation back to an earlier essay and elaborate on the relationship between those concepts and what Marcello just tried to do.
Also, I don’t see the point in distinguishing g and rationality here—somehow, Marcello got to that point without recognizing the means/ends distinction EY talks about on his own. Yes, it’s a rationality skill that can be taught, but failing to recognize it in the first place does say something about how fast you would pick up rationalist concepts in general.
Third, I never criticized his failure to immediately solve the problem, just his automatic, casual classification of “complexity” as being responsive.
Yes, Marcello may be very bright, very well-versed in rationality, have many other accomplishments—but that doesn’t mean that there’s some reasonable situational defense of what he did there.
Very smart people can still make trivial errors if they’re plunging into domains that they’re not used to thinking about. Intelligence is not rationality.