Summary of my response:chimps are nearly useless because they aren’t optimized to be useful, not because evolution was trying to make something useful and wasn’t able to succeed until it got to humans.
So far as I can tell, the best one-line summary for why we should expect a continuous and not a fast takeoff comes from the interview Paul Christiano gave on the 80k podcast: ‘I think if you optimize AI systems for reasoning, it appears much, much earlier.’ Which is to say, the equivalent of the ‘chimp’ milestone on the road to human-level AI does not have approximately the economic utility of a chimp, but a decent fraction of the utility of something that is ‘human-level’. This strikes me as an important argument that he’s repeated here, and discussed here last april but other than that it seems to have gone largely unnoticed and I’m wondering why.
I have a theory about why this didn’t get discussed earlier—there is a much more famous bad argument against AGI being an existential risk, the ‘intelligence isn’t a superpower’ argument that sounds similar. From Chollet vs Yudkowsky:
Intelligence is not a superpower; exceptional intelligence does not, on its own, confer you with proportionally exceptional power over your circumstances.
…said the Homo sapiens, surrounded by countless powerful artifacts whose abilities, let alone mechanisms, would be utterly incomprehensible to the organisms of any less intelligent Earthly species.
I worry that in arguing against the claim that general intelligence isn’t a meaningful concept or can’t be used to compare different animals, some people have been implicitly assuming that evolution has been putting a decent amount of effort into optimizing for general intelligence. Alternatively, that arguing for one sounds like another, or that a lot of people have been arguing for both together and haven’t distinguished between them.
Claiming that you can meaningfully compare evolved minds on the generality of their intelligence needs to be distinguished from claiming that evolution has been optimizing for general intelligence reasonably hard before humans came about.
That part of the interview with Paul was super interesting to me, because the following were previously claims I’d heard from Nate and Eliezer in their explanations of how they think about fast takeoff:
[E]volution [hasn’t] been putting a decent amount of effort into optimizing for general intelligence. [...]
‘I think if you optimize AI systems for reasoning, it appears much, much earlier.’
Ditto things along the lines of this Paul quote from the same 80K interview:
It’s totally conceivable from our current perspective, I think, that an intelligence that was as smart as a crow, but was actually designed for doing science, actually designed for doing engineering, for advancing technologies rapidly as possible—it is quite conceivable that such a brain would actually outcompete humans pretty badly at those tasks.
I think that’s another important thing to have in mind, and then when we talk about when stuff goes crazy, I would guess humans are an upper bound for when stuff goes crazy. That is we know that if we had cheap simulated humans, that technological progress would be much, much faster than it is today. But probably stuff goes crazy somewhat before you actually get to humans.
This is part of why I don’t talk about “human-level” AI when I write things for MIRI.
If you think humans, corvids, etc. aren’t well-optimized for economically/pragmatically interesting feats, this predicts that timelines may be shorter and that “human-level” may be an especially bad way of thinking about the relevant threshold(s).
There still remains the question of whether the technological path to “optimizing messy physical environments” (or “science AI”, or whatever we want to call it) looks like a small number of “we didn’t know how to do this at all, and now we do know how to do this and can suddenly take much better advantage of available compute” events, vs. looking like a large number of individually low-impact events spread out over time.
If no one event is impactful enough, then a series of numerous S-curves ends up looking like a smooth slope when you zoom out; and large historical changes are usually made of many small changes that add up to one big effect. We don’t invent nuclear weapons, get hit by a super-asteroid, etc. every other day.
MIRI thinks that the fact Evolution hasn’t been putting much effort into optimizing for general intelligence is a reason to expect discontinuous progress? Apparently, Paul’s point is that once we realize evolution has been putting little effort into optimizing for general intelligence, we realize we can’t tell much about the likely course of AGI development from evolutionary history, which leaves us in the default position of ignorance. Then, he further argues that the default case is that progress is continuous.
So far as I can tell, Paul’s point is that absent specific reasons to think otherwise, the prima facie case that any time we are trying hard to optimize for some criteria, we should expect the ‘many small changes that add up to one big effect’ situation.
Then he goes on to argue that the specific arguments that AGI is a rare case where this isn’t true (like nuclear weapons) are either wrong or aren’t strong enough to make discontinuous progress plausible.
From what you just wrote, it seems like the folks at MIRI agree that we should have the prima facie expectation of continuous progress, and I’ve read elsewhere that Eliezer thinks the case for recursive self-improvement leading to a discontinuity is weaker or less central than it first seemed. So, are MIRI’s main reasons for disagreeing with Paul down to other arguments (hence the switch from the intelligence explosion hypothesis to the general idea of rapid capability gain)?
I would think the most likely place to disagree with Paul (if not on the intelligence explosion hypothesis) would be if you expected the right combination of breakthroughs exceeds to a ‘generality threshold’ (or ‘secret sauce’ as Paul calls it) that leads to a big jump in capability, but inadequate achievement on any one of the breakthroughs won’t do.
Stuart Russell gives a list of the elements he thinks will be necessary for the ‘secret sauce’ of general intelligence in Human Compatible: human-like language comprehension, cumulative learning, discovering new action sets and managing its own mental activity. (I would add that somebody making that list 30 years ago would have added perception and object recognition, and somebody making it 60 years ago would have also added efficient logical reasoning from known facts). Let’s go with Russell’s list, so we can be a bit more concrete. Perhaps this is your disagreement:
An AI with (e.g.) good perception and object recognition, language comprehension, cumulative learning capability and ability to discover new action sets but a merely adequate or bad ability to manage its mental activity would be (Paul thinks) reasonably capable compared to an AI that is good at all of these things, but (MIRI thinks) it would be much less capable. MIRI has conceptual arguments (to do with the nature of general intelligence) and empirical arguments (comparing human/chimp brains and pragmatic capabilities) in favour of this hypothesis, and Paul thinks the conceptual arguments are too murky and unclear to be persuasive and that the empirical arguments don’t show what MIRI thinks they show. Am I on the right track here?
So far as I can tell, the best one-line summary for why we should expect a continuous and not a fast takeoff comes from the interview Paul Christiano gave on the 80k podcast: ‘I think if you optimize AI systems for reasoning, it appears much, much earlier.’ Which is to say, the equivalent of the ‘chimp’ milestone on the road to human-level AI does not have approximately the economic utility of a chimp, but a decent fraction of the utility of something that is ‘human-level’. This strikes me as an important argument that he’s repeated here, and discussed here last april but other than that it seems to have gone largely unnoticed and I’m wondering why.
I have a theory about why this didn’t get discussed earlier—there is a much more famous bad argument against AGI being an existential risk, the ‘intelligence isn’t a superpower’ argument that sounds similar. From Chollet vs Yudkowsky:
I worry that in arguing against the claim that general intelligence isn’t a meaningful concept or can’t be used to compare different animals, some people have been implicitly assuming that evolution has been putting a decent amount of effort into optimizing for general intelligence. Alternatively, that arguing for one sounds like another, or that a lot of people have been arguing for both together and haven’t distinguished between them.
Claiming that you can meaningfully compare evolved minds on the generality of their intelligence needs to be distinguished from claiming that evolution has been optimizing for general intelligence reasonably hard before humans came about.
That part of the interview with Paul was super interesting to me, because the following were previously claims I’d heard from Nate and Eliezer in their explanations of how they think about fast takeoff:
Ditto things along the lines of this Paul quote from the same 80K interview:
This is part of why I don’t talk about “human-level” AI when I write things for MIRI.
If you think humans, corvids, etc. aren’t well-optimized for economically/pragmatically interesting feats, this predicts that timelines may be shorter and that “human-level” may be an especially bad way of thinking about the relevant threshold(s).
There still remains the question of whether the technological path to “optimizing messy physical environments” (or “science AI”, or whatever we want to call it) looks like a small number of “we didn’t know how to do this at all, and now we do know how to do this and can suddenly take much better advantage of available compute” events, vs. looking like a large number of individually low-impact events spread out over time.
If no one event is impactful enough, then a series of numerous S-curves ends up looking like a smooth slope when you zoom out; and large historical changes are usually made of many small changes that add up to one big effect. We don’t invent nuclear weapons, get hit by a super-asteroid, etc. every other day.
MIRI thinks that the fact Evolution hasn’t been putting much effort into optimizing for general intelligence is a reason to expect discontinuous progress? Apparently, Paul’s point is that once we realize evolution has been putting little effort into optimizing for general intelligence, we realize we can’t tell much about the likely course of AGI development from evolutionary history, which leaves us in the default position of ignorance. Then, he further argues that the default case is that progress is continuous.
So far as I can tell, Paul’s point is that absent specific reasons to think otherwise, the prima facie case that any time we are trying hard to optimize for some criteria, we should expect the ‘many small changes that add up to one big effect’ situation.
Then he goes on to argue that the specific arguments that AGI is a rare case where this isn’t true (like nuclear weapons) are either wrong or aren’t strong enough to make discontinuous progress plausible.
From what you just wrote, it seems like the folks at MIRI agree that we should have the prima facie expectation of continuous progress, and I’ve read elsewhere that Eliezer thinks the case for recursive self-improvement leading to a discontinuity is weaker or less central than it first seemed. So, are MIRI’s main reasons for disagreeing with Paul down to other arguments (hence the switch from the intelligence explosion hypothesis to the general idea of rapid capability gain)?
I would think the most likely place to disagree with Paul (if not on the intelligence explosion hypothesis) would be if you expected the right combination of breakthroughs exceeds to a ‘generality threshold’ (or ‘secret sauce’ as Paul calls it) that leads to a big jump in capability, but inadequate achievement on any one of the breakthroughs won’t do.
Stuart Russell gives a list of the elements he thinks will be necessary for the ‘secret sauce’ of general intelligence in Human Compatible: human-like language comprehension, cumulative learning, discovering new action sets and managing its own mental activity. (I would add that somebody making that list 30 years ago would have added perception and object recognition, and somebody making it 60 years ago would have also added efficient logical reasoning from known facts). Let’s go with Russell’s list, so we can be a bit more concrete. Perhaps this is your disagreement:
An AI with (e.g.) good perception and object recognition, language comprehension, cumulative learning capability and ability to discover new action sets but a merely adequate or bad ability to manage its mental activity would be (Paul thinks) reasonably capable compared to an AI that is good at all of these things, but (MIRI thinks) it would be much less capable. MIRI has conceptual arguments (to do with the nature of general intelligence) and empirical arguments (comparing human/chimp brains and pragmatic capabilities) in favour of this hypothesis, and Paul thinks the conceptual arguments are too murky and unclear to be persuasive and that the empirical arguments don’t show what MIRI thinks they show. Am I on the right track here?