I think this is a good analysis, and I’m really glad to see this kind of deep dive on an important crux. The most clarifying thing for me was connecting old and new arguments—they seem to have more common ground than I thought.
One thing I would appreciate is more in-text references. There are a bunch of claims here about e.g. history, evolution with no explicit reference. Maybe it seems like common knowledge, but I wasn’t sure whether to believe some things, e.g.
Evolution was optimizing for fitness, and driving increases in intelligence only indirectly and intermittently by optimizing for winning at social competition. What happened in human evolution is that it briefly switched to optimizing for increased intelligence, and as soon as that happened our intelligence grew very rapidly but continuously.
Could you clarify? I thought biological evolution always optimizes for inclusive genetic fitness.
I’m glad this changed someone’s mind about the connection between old/new views! The links in the text are references, and links before quotes go to the location of that quote—though there should be more and I’ll add more.
To clarify that section in particular, evolution is always optimizing for fitness (tautologically) but what specific traits evolution is promoting change all the time as selection pressures shift. What Paul Christiano argued is that evolution basically was not trying to make general intelligence until very recently, and that as soon as it did try it made continuous progress.
If we compare humans and chimps at the tasks chimps are optimized for, humans are clearly much better but the difference is not nearly as stark. Compare to the difference between chimps and gibbons, gibbons and lemurs, or lemurs and squirrels.
Relatedly, evolution changes what it is optimizing for over evolutionary time: as a creature and its environment change, the returns to different skills can change, and they can potentially change very quickly. So it seems easy for evolution to shift from “not caring about X” to “caring about X,” but nothing analogous will happen for AI projects. (In fact a similar thing often does happen while optimizing something with SGD, but it doesn’t happen at the level of the ML community as a whole.)
That argument was the one thing I researched that was most surprising to me, and I’m not sure why it hasn’t been more commonly discussed.
Thanks, that makes sense. To clarify, I realise there are references/links throughout. But I forgot that the takeoff speeds post was basically making the same claim as that quote, and so I was expecting a reference more from the biology space. And there are other places where I’m curious what informed you, e.g. the progress of guns, though that’s easier to read up on myself.
I think this is a good analysis, and I’m really glad to see this kind of deep dive on an important crux. The most clarifying thing for me was connecting old and new arguments—they seem to have more common ground than I thought.
One thing I would appreciate is more in-text references. There are a bunch of claims here about e.g. history, evolution with no explicit reference. Maybe it seems like common knowledge, but I wasn’t sure whether to believe some things, e.g.
Could you clarify? I thought biological evolution always optimizes for inclusive genetic fitness.
I’m glad this changed someone’s mind about the connection between old/new views! The links in the text are references, and links before quotes go to the location of that quote—though there should be more and I’ll add more.
To clarify that section in particular, evolution is always optimizing for fitness (tautologically) but what specific traits evolution is promoting change all the time as selection pressures shift. What Paul Christiano argued is that evolution basically was not trying to make general intelligence until very recently, and that as soon as it did try it made continuous progress.
That argument was the one thing I researched that was most surprising to me, and I’m not sure why it hasn’t been more commonly discussed.
Thanks, that makes sense. To clarify, I realise there are references/links throughout. But I forgot that the takeoff speeds post was basically making the same claim as that quote, and so I was expecting a reference more from the biology space. And there are other places where I’m curious what informed you, e.g. the progress of guns, though that’s easier to read up on myself.