do other species lack proto-culture mainly because of an evolutionary oversight, or because proto-culture is not very useful until you’re close to human-level in other respects? In other words, is the discontinuity we’ve observed mainly because evolution took a weird path through the landscape of possible minds, or because the landscape is inherently quite discontinuous with respect to usefulness? I interpret Paul as claiming the former.
I think I disagree with the framing. Suppose I’m trying to be a great physicist, and I study a bunch of physics, which requires some relatively high-level understanding of math. At some point I want to do new research into general relativity, and so I do a deep dive into abstract algebra / category theory to understand tensors better. Thanks to my practice with physics, I’m able to pick it up much faster than a typical person who starts studying abstract algebra.
If you evaluate by “ability to do abstract algebra”, it seems like there was a sharp discontinuity, even though on “ability to do physics” there was not. But if I had started off trying to learn abstract algebra before doing any physics, then there would not have been such a discontinuity.
It seems wrong to say that my discontinuity in abstract algebra was “mainly because of an oversight in how I learned things”, or to say that “my learning took a weird path through the landscape of possible ways to learn fields”. Like, maybe those things would be true if you assume I had the goal of learning abstract algebra. But it’s far more natural and coherent to just say “Rohin wasn’t trying to learn abstract algebra, he was trying to learn physics”.
Similarly, I think you shouldn’t be saying that there were “evolutionary oversights” or “weird paths”, you should be saying “evolution wasn’t optimizing for proto-culture, it was optimizing for reproductive fitness”.
But if the former is true, then we should expect that there are many species (including chimpanzees) in which selection for proto-culture would be useful even in the absence of other changes like increased brain size or social skills, because proto-culture is a useful thing for them to have in ways that evolution has been ignoring.
What does “useful” mean here? If by “useful” you mean “improves an individual’s reproductive fitness”, then I disagree with the claim and I think that’s where the major disagreement is. (I also disagree that this is an implication of the argument that evolution wasn’t optimizing for proto-culture.)
If by “useful” you mean “helps in building a technological civilization”, then yes, I agree with the claim, but I don’t see why it has any relevance.
Another very similar question which is maybe more intuitive: suppose we take animals like monkeys, and evolve them by selecting the ones which seem like they’re making the most progress towards building a technological civilisation, until eventually they succeed. Would their progress be much more continuous than the human case, or fairly similar? Paul would say the former
Yes, I agree with this one (at least if we get to use a shaped reward, e.g. we get to select the ones that show signs of intelligence / culture, on the view that that is a necessary prereq to technological civilization).
I don’t know why you lean towards the latter.
Paul’s argument is that evolution was discontinuous specifically because evolution was dumb in certain ways.
No, the argument is that evolution wasn’t trying, not that it was dumb. (Really I want to taboo “trying”, “dumb”, “intelligent”, “optimization” and so on and talk only about what is and isn’t selected for—the argument is that evolution was not selecting for proto-culture / intelligence, whereas humans will select for proto-culture / intelligence. But if you must cast this in agent-oriented language, it’s that evolution wasn’t trying, not that it was dumb.)
My claim is that AGI may be discontinuous specifically because humans are dumb in certain ways
I’d find this a lot more believable if you convincingly argued for “humans are not selecting for agents with proto-culture / intelligence”, which is the analog to “evolution was not selecting for proto-culture / intelligence”.
a reason that this topic seems quite difficult for me to pin down is that the two questions seem pretty closely tied together. So if you think that the landscape of usefulness is really weird and discontinuous, then maybe humans can still find a continuous path by being really clever.
I agree that arguments about the landscape are going to be difficult. I don’t think that’s the crux of Paul’s original argument.
I think that, because culture is eventually very useful for fitness, you can either think of the problem as evolution not optimising for culture, or evolution optimising for fitness badly. And these are roughly equivalent ways of thinking about it, just different framings. Paul notes this duality in his original post:
If we step back from skills and instead look at outcomes we could say: “Evolution is always optimizing for fitness, and humans have now taken over the world.” On this perspective, I’m making a claim about the limits of evolution. First, evolution is theoretically optimizing for fitness, but it isn’t able to look ahead and identify which skills will be most important for your children’s children’s children’s fitness. Second, human intelligence is incredibly good for the fitness of groups of humans, but evolution acts on individual humans for whom the effect size is much smaller (who barely benefit at all from passing knowledge on to the next generation).
It seems like most of your response is an objection to this framing. I may need to think more about the relative advantages and disadvantages of each framing, but I don’t think either is outright wrong.
What does “useful” mean here? If by “useful” you mean “improves an individual’s reproductive fitness”, then I disagree with the claim and I think that’s where the major disagreement is.
Yes, I meant useful for reproductive fitness. Sorry for ambiguity.
I may need to think more about the relative advantages and disadvantages of each framing, but I don’t think either is outright wrong.
I agree it’s not wrong. I’m claiming it’s not a useful framing. If we must use this framing, I think humans and evolution are not remotely comparable on how good they are at long-term optimization, and I can’t understand why you think they are. (Humans may not be good at long-term optimization on some absolute scale, but they’re a hell of a lot better than evolution.)
I think in my example you could make a similar argument: looking at outcomes, you could say “Rohin is always optimizing for learning abstract algebra, and he has now become very good at abstract algebra.” It’s not wrong, it’s just not useful for predicting my future behavior, and doesn’t seem to carve reality at its joints.
(Tbc, I think this example is overstating the case, “evolution is always optimizing for fitness” is definitely more reasonable and more predictive than “Rohin is always optimizing for learning abstract algebra”.)
I really do think that the best thing is to just strip away agency, and talk about selection:
the argument is that evolution was not selecting for proto-culture / intelligence, whereas humans will select for proto-culture / intelligence
Re: usefulness:
Yes, I meant useful for reproductive fitness.
Suppose a specific monkey has some mutation and gets a little bit of proto-culture. Are you claiming that this will increase the number of children that monkey has?
I think I disagree with the framing. Suppose I’m trying to be a great physicist, and I study a bunch of physics, which requires some relatively high-level understanding of math. At some point I want to do new research into general relativity, and so I do a deep dive into abstract algebra / category theory to understand tensors better. Thanks to my practice with physics, I’m able to pick it up much faster than a typical person who starts studying abstract algebra.
If you evaluate by “ability to do abstract algebra”, it seems like there was a sharp discontinuity, even though on “ability to do physics” there was not. But if I had started off trying to learn abstract algebra before doing any physics, then there would not have been such a discontinuity.
It seems wrong to say that my discontinuity in abstract algebra was “mainly because of an oversight in how I learned things”, or to say that “my learning took a weird path through the landscape of possible ways to learn fields”. Like, maybe those things would be true if you assume I had the goal of learning abstract algebra. But it’s far more natural and coherent to just say “Rohin wasn’t trying to learn abstract algebra, he was trying to learn physics”.
Similarly, I think you shouldn’t be saying that there were “evolutionary oversights” or “weird paths”, you should be saying “evolution wasn’t optimizing for proto-culture, it was optimizing for reproductive fitness”.
What does “useful” mean here? If by “useful” you mean “improves an individual’s reproductive fitness”, then I disagree with the claim and I think that’s where the major disagreement is. (I also disagree that this is an implication of the argument that evolution wasn’t optimizing for proto-culture.)
If by “useful” you mean “helps in building a technological civilization”, then yes, I agree with the claim, but I don’t see why it has any relevance.
Yes, I agree with this one (at least if we get to use a shaped reward, e.g. we get to select the ones that show signs of intelligence / culture, on the view that that is a necessary prereq to technological civilization).
I don’t know why you lean towards the latter.
No, the argument is that evolution wasn’t trying, not that it was dumb. (Really I want to taboo “trying”, “dumb”, “intelligent”, “optimization” and so on and talk only about what is and isn’t selected for—the argument is that evolution was not selecting for proto-culture / intelligence, whereas humans will select for proto-culture / intelligence. But if you must cast this in agent-oriented language, it’s that evolution wasn’t trying, not that it was dumb.)
I’d find this a lot more believable if you convincingly argued for “humans are not selecting for agents with proto-culture / intelligence”, which is the analog to “evolution was not selecting for proto-culture / intelligence”.
I agree that arguments about the landscape are going to be difficult. I don’t think that’s the crux of Paul’s original argument.
I think that, because culture is eventually very useful for fitness, you can either think of the problem as evolution not optimising for culture, or evolution optimising for fitness badly. And these are roughly equivalent ways of thinking about it, just different framings. Paul notes this duality in his original post:
It seems like most of your response is an objection to this framing. I may need to think more about the relative advantages and disadvantages of each framing, but I don’t think either is outright wrong.
Yes, I meant useful for reproductive fitness. Sorry for ambiguity.
I agree it’s not wrong. I’m claiming it’s not a useful framing. If we must use this framing, I think humans and evolution are not remotely comparable on how good they are at long-term optimization, and I can’t understand why you think they are. (Humans may not be good at long-term optimization on some absolute scale, but they’re a hell of a lot better than evolution.)
I think in my example you could make a similar argument: looking at outcomes, you could say “Rohin is always optimizing for learning abstract algebra, and he has now become very good at abstract algebra.” It’s not wrong, it’s just not useful for predicting my future behavior, and doesn’t seem to carve reality at its joints.
(Tbc, I think this example is overstating the case, “evolution is always optimizing for fitness” is definitely more reasonable and more predictive than “Rohin is always optimizing for learning abstract algebra”.)
I really do think that the best thing is to just strip away agency, and talk about selection:
Re: usefulness:
Suppose a specific monkey has some mutation and gets a little bit of proto-culture. Are you claiming that this will increase the number of children that monkey has?