Are there simple changes to chimps (or other animals) that would make them much better at accumulating culture?
Of course there are simple changes to chimps that make them much better at accumulating culture. How else did humans evolve?
(I suppose it is possible that evolution got one really lucky mutation that mutated 50 things at once and it needed all 50 of those mutations to create culture, and this was extraordinarily unlikely but anthropic effects can explain that, but I don’t think you’re arguing that.)
The question isn’t whether there are simple changes—it seems likely there were—the question is whether we should expect humans not to find these simple changes.
Will humans continually pursue all simple yet powerful changes to our AIs?
I feel like the answer to this question doesn’t tell me that much about takeoff speeds. Taking your quote from Paul:
If we step back from skills and instead look at outcomes we could say: “Evolution is always optimizing for fitness, and humans have now taken over the world.”
I see the key distinguishing feature of the argument as “evolution myopically optimized for fitness; culture is not myopically good for fitness; humans instead optimize for usefulness; culture is good for usefulness”. The key feature isn’t about whether humans are better optimizers than evolution, it’s about what the target of the optimization is. Even if humans were as “dumb” at optimization as evolution, but they continued to evaluate AI systems by how useful they are, I’d still expect a continuous takeoff.
(Also, you seem to be arguing that the dumber we expect human optimization to be, the more we should expect discontinuities. This seems kind of wild—surely with more understanding you can find more fundamental insights, and this leads to discontinuities? Sure seems like this was the story with nukes, for example.)
Hmm, let’s see. So the question I’m trying to ask here is: do other species lack proto-culture mainly because of an evolutionary oversight, or because proto-culture is not very useful until you’re close to human-level in other respects? In other words, is the discontinuity we’ve observed mainly because evolution took a weird path through the landscape of possible minds, or because the landscape is inherently quite discontinuous with respect to usefulness? I interpret Paul as claiming the former.
But if the former is true, then we should expect that there are many species (including chimpanzees) in which selection for proto-culture would be useful even in the absence of other changes like increased brain size or social skills, because proto-culture is a useful thing for them to have in ways that evolution has been ignoring. So by “simple changes” I mean something like: changes which could be induced by a relatively short period of medium-strength selection specifically for proto-culture (say, 100,000 years; much less than the human-chimp gap).
Another very similar question which is maybe more intuitive: suppose we take animals like monkeys, and evolve them by selecting the ones which seem like they’re making the most progress towards building a technological civilisation, until eventually they succeed. Would their progress be much more continuous than the human case, or fairly similar? Paul would say the former, I’m currently leaning (slightly) towards the latter. This version of the question doesn’t make so much sense with chimpanzees, since it may be the case that by the time we reach chimpanzees, we’ve “locked in” a pretty sharp discontinuity.
Both of these are proxies for the thing I’m actually interested in, which is whether more direct optimisation for reaching civilisation leads to much more continuous paths to civilisation than the one we took.
The question isn’t whether there are simple changes—it seems likely there were—the question is whether we should expect humans not to find these simple changes.
Both of these are interesting questions, if you interpret the former in the way I just described.
Separately, even if we concede that evolutionary progress could have been much more continuous if it had been “optimising for the right thing”, we can also question whether humans will “optimise for the right thing”.
You seem to be arguing that the dumber we expect human optimization to be, the more we should expect discontinuities. This seems kind of wild
Paul’s argument is that evolution was discontinuous specifically because evolution was dumb in certain ways. My claim is that AGI may be discontinuous specifically because humans are dumb in certain ways (i.e. taking a long time to notice big breakthroughs, during which an overhang builds up). There are other ways in which humans being dumb would make discontinuities less likely (e.g. if we’re incapable of big breakthroughs). That’s why I phrased the question as “Will humans continually pursue all simple yet powerful changes to our AIs?”, because I agree that humans are smart enough to find simple yet powerful changes if we’re looking in the right direction, but I think there will be long periods in which we’re looking in the wrong direction (i.e. not “continually pursuing” the most productive directions).
Thanks for the feedback. My responses are all things that I probably should have put in the original post. If they make sense to you (even if you disagree with them) then I’ll edit the post to add them in.
Oh, one last thing I should mention: a reason that this topic seems quite difficult for me to pin down is that the two questions seem pretty closely tied together. So if you think that the landscape of usefulness is really weird and discontinuous, then maybe humans can still find a continuous path by being really clever. Or maybe the landscape is actually pretty smooth, but humans are so much dumber than evolution that by default we’ll end up on a much more discontinuous path (because we accumulate massive hardware overhangs while waiting for the key insights). I don’t know how to pin down definitions for each of the questions which don’t implicitly depend on our expectations about the other question.
do other species lack proto-culture mainly because of an evolutionary oversight, or because proto-culture is not very useful until you’re close to human-level in other respects? In other words, is the discontinuity we’ve observed mainly because evolution took a weird path through the landscape of possible minds, or because the landscape is inherently quite discontinuous with respect to usefulness? I interpret Paul as claiming the former.
I think I disagree with the framing. Suppose I’m trying to be a great physicist, and I study a bunch of physics, which requires some relatively high-level understanding of math. At some point I want to do new research into general relativity, and so I do a deep dive into abstract algebra / category theory to understand tensors better. Thanks to my practice with physics, I’m able to pick it up much faster than a typical person who starts studying abstract algebra.
If you evaluate by “ability to do abstract algebra”, it seems like there was a sharp discontinuity, even though on “ability to do physics” there was not. But if I had started off trying to learn abstract algebra before doing any physics, then there would not have been such a discontinuity.
It seems wrong to say that my discontinuity in abstract algebra was “mainly because of an oversight in how I learned things”, or to say that “my learning took a weird path through the landscape of possible ways to learn fields”. Like, maybe those things would be true if you assume I had the goal of learning abstract algebra. But it’s far more natural and coherent to just say “Rohin wasn’t trying to learn abstract algebra, he was trying to learn physics”.
Similarly, I think you shouldn’t be saying that there were “evolutionary oversights” or “weird paths”, you should be saying “evolution wasn’t optimizing for proto-culture, it was optimizing for reproductive fitness”.
But if the former is true, then we should expect that there are many species (including chimpanzees) in which selection for proto-culture would be useful even in the absence of other changes like increased brain size or social skills, because proto-culture is a useful thing for them to have in ways that evolution has been ignoring.
What does “useful” mean here? If by “useful” you mean “improves an individual’s reproductive fitness”, then I disagree with the claim and I think that’s where the major disagreement is. (I also disagree that this is an implication of the argument that evolution wasn’t optimizing for proto-culture.)
If by “useful” you mean “helps in building a technological civilization”, then yes, I agree with the claim, but I don’t see why it has any relevance.
Another very similar question which is maybe more intuitive: suppose we take animals like monkeys, and evolve them by selecting the ones which seem like they’re making the most progress towards building a technological civilisation, until eventually they succeed. Would their progress be much more continuous than the human case, or fairly similar? Paul would say the former
Yes, I agree with this one (at least if we get to use a shaped reward, e.g. we get to select the ones that show signs of intelligence / culture, on the view that that is a necessary prereq to technological civilization).
I don’t know why you lean towards the latter.
Paul’s argument is that evolution was discontinuous specifically because evolution was dumb in certain ways.
No, the argument is that evolution wasn’t trying, not that it was dumb. (Really I want to taboo “trying”, “dumb”, “intelligent”, “optimization” and so on and talk only about what is and isn’t selected for—the argument is that evolution was not selecting for proto-culture / intelligence, whereas humans will select for proto-culture / intelligence. But if you must cast this in agent-oriented language, it’s that evolution wasn’t trying, not that it was dumb.)
My claim is that AGI may be discontinuous specifically because humans are dumb in certain ways
I’d find this a lot more believable if you convincingly argued for “humans are not selecting for agents with proto-culture / intelligence”, which is the analog to “evolution was not selecting for proto-culture / intelligence”.
a reason that this topic seems quite difficult for me to pin down is that the two questions seem pretty closely tied together. So if you think that the landscape of usefulness is really weird and discontinuous, then maybe humans can still find a continuous path by being really clever.
I agree that arguments about the landscape are going to be difficult. I don’t think that’s the crux of Paul’s original argument.
I think that, because culture is eventually very useful for fitness, you can either think of the problem as evolution not optimising for culture, or evolution optimising for fitness badly. And these are roughly equivalent ways of thinking about it, just different framings. Paul notes this duality in his original post:
If we step back from skills and instead look at outcomes we could say: “Evolution is always optimizing for fitness, and humans have now taken over the world.” On this perspective, I’m making a claim about the limits of evolution. First, evolution is theoretically optimizing for fitness, but it isn’t able to look ahead and identify which skills will be most important for your children’s children’s children’s fitness. Second, human intelligence is incredibly good for the fitness of groups of humans, but evolution acts on individual humans for whom the effect size is much smaller (who barely benefit at all from passing knowledge on to the next generation).
It seems like most of your response is an objection to this framing. I may need to think more about the relative advantages and disadvantages of each framing, but I don’t think either is outright wrong.
What does “useful” mean here? If by “useful” you mean “improves an individual’s reproductive fitness”, then I disagree with the claim and I think that’s where the major disagreement is.
Yes, I meant useful for reproductive fitness. Sorry for ambiguity.
I may need to think more about the relative advantages and disadvantages of each framing, but I don’t think either is outright wrong.
I agree it’s not wrong. I’m claiming it’s not a useful framing. If we must use this framing, I think humans and evolution are not remotely comparable on how good they are at long-term optimization, and I can’t understand why you think they are. (Humans may not be good at long-term optimization on some absolute scale, but they’re a hell of a lot better than evolution.)
I think in my example you could make a similar argument: looking at outcomes, you could say “Rohin is always optimizing for learning abstract algebra, and he has now become very good at abstract algebra.” It’s not wrong, it’s just not useful for predicting my future behavior, and doesn’t seem to carve reality at its joints.
(Tbc, I think this example is overstating the case, “evolution is always optimizing for fitness” is definitely more reasonable and more predictive than “Rohin is always optimizing for learning abstract algebra”.)
I really do think that the best thing is to just strip away agency, and talk about selection:
the argument is that evolution was not selecting for proto-culture / intelligence, whereas humans will select for proto-culture / intelligence
Re: usefulness:
Yes, I meant useful for reproductive fitness.
Suppose a specific monkey has some mutation and gets a little bit of proto-culture. Are you claiming that this will increase the number of children that monkey has?
Of course there are simple changes to chimps that make them much better at accumulating culture. How else did humans evolve?
(I suppose it is possible that evolution got one really lucky mutation that mutated 50 things at once and it needed all 50 of those mutations to create culture, and this was extraordinarily unlikely but anthropic effects can explain that, but I don’t think you’re arguing that.)
The question isn’t whether there are simple changes—it seems likely there were—the question is whether we should expect humans not to find these simple changes.
I feel like the answer to this question doesn’t tell me that much about takeoff speeds. Taking your quote from Paul:
I see the key distinguishing feature of the argument as “evolution myopically optimized for fitness; culture is not myopically good for fitness; humans instead optimize for usefulness; culture is good for usefulness”. The key feature isn’t about whether humans are better optimizers than evolution, it’s about what the target of the optimization is. Even if humans were as “dumb” at optimization as evolution, but they continued to evaluate AI systems by how useful they are, I’d still expect a continuous takeoff.
(Also, you seem to be arguing that the dumber we expect human optimization to be, the more we should expect discontinuities. This seems kind of wild—surely with more understanding you can find more fundamental insights, and this leads to discontinuities? Sure seems like this was the story with nukes, for example.)
Hmm, let’s see. So the question I’m trying to ask here is: do other species lack proto-culture mainly because of an evolutionary oversight, or because proto-culture is not very useful until you’re close to human-level in other respects? In other words, is the discontinuity we’ve observed mainly because evolution took a weird path through the landscape of possible minds, or because the landscape is inherently quite discontinuous with respect to usefulness? I interpret Paul as claiming the former.
But if the former is true, then we should expect that there are many species (including chimpanzees) in which selection for proto-culture would be useful even in the absence of other changes like increased brain size or social skills, because proto-culture is a useful thing for them to have in ways that evolution has been ignoring. So by “simple changes” I mean something like: changes which could be induced by a relatively short period of medium-strength selection specifically for proto-culture (say, 100,000 years; much less than the human-chimp gap).
Another very similar question which is maybe more intuitive: suppose we take animals like monkeys, and evolve them by selecting the ones which seem like they’re making the most progress towards building a technological civilisation, until eventually they succeed. Would their progress be much more continuous than the human case, or fairly similar? Paul would say the former, I’m currently leaning (slightly) towards the latter. This version of the question doesn’t make so much sense with chimpanzees, since it may be the case that by the time we reach chimpanzees, we’ve “locked in” a pretty sharp discontinuity.
Both of these are proxies for the thing I’m actually interested in, which is whether more direct optimisation for reaching civilisation leads to much more continuous paths to civilisation than the one we took.
Both of these are interesting questions, if you interpret the former in the way I just described.
Separately, even if we concede that evolutionary progress could have been much more continuous if it had been “optimising for the right thing”, we can also question whether humans will “optimise for the right thing”.
Paul’s argument is that evolution was discontinuous specifically because evolution was dumb in certain ways. My claim is that AGI may be discontinuous specifically because humans are dumb in certain ways (i.e. taking a long time to notice big breakthroughs, during which an overhang builds up). There are other ways in which humans being dumb would make discontinuities less likely (e.g. if we’re incapable of big breakthroughs). That’s why I phrased the question as “Will humans continually pursue all simple yet powerful changes to our AIs?”, because I agree that humans are smart enough to find simple yet powerful changes if we’re looking in the right direction, but I think there will be long periods in which we’re looking in the wrong direction (i.e. not “continually pursuing” the most productive directions).
Thanks for the feedback. My responses are all things that I probably should have put in the original post. If they make sense to you (even if you disagree with them) then I’ll edit the post to add them in.
Oh, one last thing I should mention: a reason that this topic seems quite difficult for me to pin down is that the two questions seem pretty closely tied together. So if you think that the landscape of usefulness is really weird and discontinuous, then maybe humans can still find a continuous path by being really clever. Or maybe the landscape is actually pretty smooth, but humans are so much dumber than evolution that by default we’ll end up on a much more discontinuous path (because we accumulate massive hardware overhangs while waiting for the key insights). I don’t know how to pin down definitions for each of the questions which don’t implicitly depend on our expectations about the other question.
I think I disagree with the framing. Suppose I’m trying to be a great physicist, and I study a bunch of physics, which requires some relatively high-level understanding of math. At some point I want to do new research into general relativity, and so I do a deep dive into abstract algebra / category theory to understand tensors better. Thanks to my practice with physics, I’m able to pick it up much faster than a typical person who starts studying abstract algebra.
If you evaluate by “ability to do abstract algebra”, it seems like there was a sharp discontinuity, even though on “ability to do physics” there was not. But if I had started off trying to learn abstract algebra before doing any physics, then there would not have been such a discontinuity.
It seems wrong to say that my discontinuity in abstract algebra was “mainly because of an oversight in how I learned things”, or to say that “my learning took a weird path through the landscape of possible ways to learn fields”. Like, maybe those things would be true if you assume I had the goal of learning abstract algebra. But it’s far more natural and coherent to just say “Rohin wasn’t trying to learn abstract algebra, he was trying to learn physics”.
Similarly, I think you shouldn’t be saying that there were “evolutionary oversights” or “weird paths”, you should be saying “evolution wasn’t optimizing for proto-culture, it was optimizing for reproductive fitness”.
What does “useful” mean here? If by “useful” you mean “improves an individual’s reproductive fitness”, then I disagree with the claim and I think that’s where the major disagreement is. (I also disagree that this is an implication of the argument that evolution wasn’t optimizing for proto-culture.)
If by “useful” you mean “helps in building a technological civilization”, then yes, I agree with the claim, but I don’t see why it has any relevance.
Yes, I agree with this one (at least if we get to use a shaped reward, e.g. we get to select the ones that show signs of intelligence / culture, on the view that that is a necessary prereq to technological civilization).
I don’t know why you lean towards the latter.
No, the argument is that evolution wasn’t trying, not that it was dumb. (Really I want to taboo “trying”, “dumb”, “intelligent”, “optimization” and so on and talk only about what is and isn’t selected for—the argument is that evolution was not selecting for proto-culture / intelligence, whereas humans will select for proto-culture / intelligence. But if you must cast this in agent-oriented language, it’s that evolution wasn’t trying, not that it was dumb.)
I’d find this a lot more believable if you convincingly argued for “humans are not selecting for agents with proto-culture / intelligence”, which is the analog to “evolution was not selecting for proto-culture / intelligence”.
I agree that arguments about the landscape are going to be difficult. I don’t think that’s the crux of Paul’s original argument.
I think that, because culture is eventually very useful for fitness, you can either think of the problem as evolution not optimising for culture, or evolution optimising for fitness badly. And these are roughly equivalent ways of thinking about it, just different framings. Paul notes this duality in his original post:
It seems like most of your response is an objection to this framing. I may need to think more about the relative advantages and disadvantages of each framing, but I don’t think either is outright wrong.
Yes, I meant useful for reproductive fitness. Sorry for ambiguity.
I agree it’s not wrong. I’m claiming it’s not a useful framing. If we must use this framing, I think humans and evolution are not remotely comparable on how good they are at long-term optimization, and I can’t understand why you think they are. (Humans may not be good at long-term optimization on some absolute scale, but they’re a hell of a lot better than evolution.)
I think in my example you could make a similar argument: looking at outcomes, you could say “Rohin is always optimizing for learning abstract algebra, and he has now become very good at abstract algebra.” It’s not wrong, it’s just not useful for predicting my future behavior, and doesn’t seem to carve reality at its joints.
(Tbc, I think this example is overstating the case, “evolution is always optimizing for fitness” is definitely more reasonable and more predictive than “Rohin is always optimizing for learning abstract algebra”.)
I really do think that the best thing is to just strip away agency, and talk about selection:
Re: usefulness:
Suppose a specific monkey has some mutation and gets a little bit of proto-culture. Are you claiming that this will increase the number of children that monkey has?