IDK if this is relevant to much, but anyway, given the public record, saying that Scott Garrabrant isn’t a genius is just incorrect. Sam Eisenstat is also a genius. Also Jessica Taylor I think. (Pace other members of AF such as myself.)
Jessica I’m less sure about. Sam, from large quantities of insights in many conversations. If you want something more legible, I’m what, >300 ELO points better than you at math; Sam’s >150 ELO points better than me at math if I’m trained up, now probably more like >250 or something.
Apologies—when I said genius, I had a very high bar in mind, no more than a half dozen people alive today, who each have single-handedly created or materially advanced an entire field. And I certainly hold Scott in very high esteem, and while I don’t know Sam or Jessica personally, I expect they are within throwing distance—but I don’t think any of them meet this insanely high bar. And Scott’s views on this, at least from ca. 2015, was a large part of what informed my thinking about this; I can’t tell the difference between him and Terry Tao when speaking with them, but he can, and he said there is clearly a qualitative difference there. Similarly for other people clearly above my league, including a friend who worked with Thurston at Cornell back in 2003-5. (It’s very plausible that Scott Aaronson is in this bucket as well, albeit in a different areas, though I can’t tell personally, and have not heard people say this directly—but he’s not actually working on the key problems, and per him, he hasn’t really tried to work on agent foundations. Unfortunately.)
So to be clear, I think Scott is a genius, but not one of the level that is needed to single-handedly advance the field to the point where the problem might be solved this decade, if it is solvable. Yes, he’s brilliant, and yes, he has unarguably done a large amount of the most valuable work in the area in the past decade, albeit mostly more foundational that what is needed to solve the problem. So if we had another dozen people of his caliber at each of a dozen universities working on this, that would be at least similar in magnitude to what we have seen in fields that have made significant progress in a decade—though even then, not all fields like hat see progress.
But the Tao / Thurston level of genius, usually in addition to the above-mentioned 100+ top people working on the problem, is what has given us rapid progress in the past in fields where such progress was possible. This may not be one of those areas—but I certainly don’t expect that we can do much better than other areas with much less intellectual firepower, hence my above claim that humanity as a whole hasn’t managed even what I’d consider a half-assed semi-serious attempt at solving a problem that deserves an entire field of research working feverishly to try our best to actually not die—and not just a few lone brilliant researchers.
Oh ok lol. Ok on a quick read I didn’t see too much in this comment to disagree with.
(One possible point of disagreement is that I think you plausibly couldn’t gather any set of people alive today and solve the technical problem; plausibly you need many, like many hundreds, of people you call geniuses. Obviously “hundreds” is made up, but I mean to say that the problem, “come to understand minds—the most subtle/complex thing ever—at a pretty deep+comprehensive level”, is IMO extremely difficult, like it’s harder than anything humanity has done so far by a lot, not just an ordinary big science project. Possibly contra Soares, IDK.)
(Another disagreement would be
[Scott] has unarguably done a large amount of the most valuable work in the area in the past decade
I don’t actually think logical induction is that valuable for the AGI alignment problem, to the point where random philosophy is on par in terms of value to alignment, though I expect most people to disagree with this. It’s just a genius technical insight in general.)
I admitted that it’s possible the problem is practically unsolvable, or worse; you could have put the entire world on Russell and Whitehead’s goal of systematizing math, and you might have gotten to Gödel faster, but you’d probably just waste more time.
And on Scott’s contributions, I think they are solving or contributing towards solving parts of the problems that were posited initially as critical to alignment, and I haven’t seen anyone do more. (With the possible exception of Paul Christiano, who hasn’t been focusing on research for solving alignment as much recently.) I agree that the work doesn’t don’t do much other than establish better foundations, but that’s kind-of the point. (And it’s not just Logical induction—there’s his collaboration on Embedded Agency, and his work on finite factored sets.) But the fact that the work done to establish the base for the work is more philosophical and doesn’t align AGI seems like it is moving the goalposts, even if I agree it’s true.
IDK if this is relevant to much, but anyway, given the public record, saying that Scott Garrabrant isn’t a genius is just incorrect. Sam Eisenstat is also a genius. Also Jessica Taylor I think. (Pace other members of AF such as myself.)
Can you operationalize the standard you’re using for “genius” here? Do you mean “IQ > 150″?
Of course not. I mean, any reasonable standard? Garrabrant induction, bro. “Produces deep novel (ETA: important difficult) insight”
Have the others you listed produced insights on that level? What did you observe that leads you to call them geniuses, “by any reasonable standard”?
Jessica I’m less sure about. Sam, from large quantities of insights in many conversations. If you want something more legible, I’m what, >300 ELO points better than you at math; Sam’s >150 ELO points better than me at math if I’m trained up, now probably more like >250 or something.
Not by David’s standard though, lol.
Sam: https://www.lesswrong.com/posts/CvKnhXTu9BPcdKE4W/an-untrollable-mathematician-illustrated
Apologies—when I said genius, I had a very high bar in mind, no more than a half dozen people alive today, who each have single-handedly created or materially advanced an entire field. And I certainly hold Scott in very high esteem, and while I don’t know Sam or Jessica personally, I expect they are within throwing distance—but I don’t think any of them meet this insanely high bar. And Scott’s views on this, at least from ca. 2015, was a large part of what informed my thinking about this; I can’t tell the difference between him and Terry Tao when speaking with them, but he can, and he said there is clearly a qualitative difference there. Similarly for other people clearly above my league, including a friend who worked with Thurston at Cornell back in 2003-5. (It’s very plausible that Scott Aaronson is in this bucket as well, albeit in a different areas, though I can’t tell personally, and have not heard people say this directly—but he’s not actually working on the key problems, and per him, he hasn’t really tried to work on agent foundations. Unfortunately.)
So to be clear, I think Scott is a genius, but not one of the level that is needed to single-handedly advance the field to the point where the problem might be solved this decade, if it is solvable. Yes, he’s brilliant, and yes, he has unarguably done a large amount of the most valuable work in the area in the past decade, albeit mostly more foundational that what is needed to solve the problem. So if we had another dozen people of his caliber at each of a dozen universities working on this, that would be at least similar in magnitude to what we have seen in fields that have made significant progress in a decade—though even then, not all fields like hat see progress.
But the Tao / Thurston level of genius, usually in addition to the above-mentioned 100+ top people working on the problem, is what has given us rapid progress in the past in fields where such progress was possible. This may not be one of those areas—but I certainly don’t expect that we can do much better than other areas with much less intellectual firepower, hence my above claim that humanity as a whole hasn’t managed even what I’d consider a half-assed semi-serious attempt at solving a problem that deserves an entire field of research working feverishly to try our best to actually not die—and not just a few lone brilliant researchers.
Oh ok lol. Ok on a quick read I didn’t see too much in this comment to disagree with.
(One possible point of disagreement is that I think you plausibly couldn’t gather any set of people alive today and solve the technical problem; plausibly you need many, like many hundreds, of people you call geniuses. Obviously “hundreds” is made up, but I mean to say that the problem, “come to understand minds—the most subtle/complex thing ever—at a pretty deep+comprehensive level”, is IMO extremely difficult, like it’s harder than anything humanity has done so far by a lot, not just an ordinary big science project. Possibly contra Soares, IDK.)
(Another disagreement would be
I don’t actually think logical induction is that valuable for the AGI alignment problem, to the point where random philosophy is on par in terms of value to alignment, though I expect most people to disagree with this. It’s just a genius technical insight in general.)
I admitted that it’s possible the problem is practically unsolvable, or worse; you could have put the entire world on Russell and Whitehead’s goal of systematizing math, and you might have gotten to Gödel faster, but you’d probably just waste more time.
And on Scott’s contributions, I think they are solving or contributing towards solving parts of the problems that were posited initially as critical to alignment, and I haven’t seen anyone do more. (With the possible exception of Paul Christiano, who hasn’t been focusing on research for solving alignment as much recently.) I agree that the work doesn’t don’t do much other than establish better foundations, but that’s kind-of the point. (And it’s not just Logical induction—there’s his collaboration on Embedded Agency, and his work on finite factored sets.) But the fact that the work done to establish the base for the work is more philosophical and doesn’t align AGI seems like it is moving the goalposts, even if I agree it’s true.