I’m much less STEM-oriented than most people here, so I could just be totally misunderstanding the points made in this post, but I tried reading it anyway, and a couple of things stood out to me as possibly mistaken:
Evolution applies very little direct optimization power to the middle level. E.g., evolution does not transfer the skills, knowledge, values, or behaviors learned by one generation to their descendants.
Am I missing something here, or is this just describing memetics? Granted, skills, knowledge, values, traditions, etc., are heritable in other ways than purely by lineal descent, but parents do also impart these to their children, and these are subject to evolution.
In contrast, I think we can explain humans’ tendency to like ice cream using the standard language of reinforcement learning. It doesn’t require that we adopt an entirely new paradigm before we can even get a handle on such issues.
But isn’t this solely because we have already studied our sensory organs and have a concept of taste buds, and hence flavors like sweet, etc. as primary categories of taste? It is not clear to me that we can do the same thing with regard to eg. ethics, even where humans are concerned. Does this not illustrate Yudkowsky’s point about inscrutable matrices?
Am I missing something here, or is this just describing memetics?
It is not describing memetics, which I regard as a mostly confused framework that primes people to misattribute the products of human intelligence to “evolution”. However, even if evolution meaningfully operates on the level of memes, the “Evolution” I’m referring to when I say “Evolution applies very little direct optimization power to the middle level” is strictly biological evolution over the genome, not memetic at all.
Memetic evolution in this context would not have inclusive genetic fitness as its “outer” objective, so whether memetic evolution can “transfer the skills, knowledge, values, or behaviors learned by one generation to their descendants” is irrelevant for the argument I was making in the post.
But isn’t this solely because we have already studied our sensory organs and have a concept of taste buds, and hence flavors like sweet, etc. as primary categories of taste?
Not really. The only way our understanding of the biology of taste impacts the story about humans coming to like ice cream is that we can infer that humans have sugar detecting reward circuitry, which ice cream activates in the modern environment. For AI systems, we actually have a better handle on how their reward circuitry works, as compared to the brain. E.g., we can just directly look at the reward counter during the AI’s training.
It is not describing memetics, which I regard as a mostly confused framework that primes people to misattribute the products of human intelligence to “evolution”.
New memes may arise either by being mutated from other memes or by invention ex nihilo—either of which involves some degree of human intelligence. However, if a meme becomes prevalent, it is not because all of its holders have invented it independently. It has rather spread because it is adapted both to the existing memetic ecosystem as well as to human intelligence. Of course, if certain memes reduce the likelihood of reproduction, that provides an evolutionary pressure for human intelligence to change to be more resistant to that particular kind of meme, so there are very complex interactions.
It is not a confused framework—at least not inherently—and it does not require us to ignore the role of human intelligence.
Memetic evolution in this context would not have inclusive genetic fitness as its “outer” objective, so whether memetic evolution can “transfer the skills, knowledge, values, or behaviors learned by one generation to their descendants” is irrelevant for the argument I was making in the post
My argument is that evolution selects simultaneously for genetic and memetic fitness, and that both genes and memes tend to be passed on from parent to child. Thus, evolution operates at a combined genetic-memetic level where it optimizes for inclusive genetic-memetic fitness. Though genes and memes correspond to entirely different mediums, they interact in complex ways when it comes to evolutionary fitness, so the mechanisms are not that straightforwardly separable. In addition, there are social network effects and geographic localization influencing what skills people are likely to acquire, such that skills have a tendency to be heritable in a manner that is not easily reducible to genetics, but which nevertheless influences evolutionary fitness. If we look aside from the fact that memes and skills can be transferred in manners other than heredity, then we can sorta model them as an extended genome.
Not really. The only way our understanding of the biology of taste impacts the story about humans coming to like ice cream is that we can infer that humans have sugar detecting reward circuitry, which ice cream activates in the modern environment. For AI systems, we actually have a better handle on how their reward circuitry works, as compared to the brain. E.g., we can just directly look at the reward counter during the AI’s training.
But the reason we can say that it is bad for humans to become addicted to ice cream is because we have an existing paradigm that provides us with a deep understanding of nutrition, and even here, subtle failures in the paradigm have notoriously done serious harms. Do you regard our understanding of morality as more reliable than our understanding of nutrition?
Remember, the context was Yudkowsky’s argument that we lack a paradigm to address systematic failures in which reward circuitry fails to correspond to good action. That is, specific understanding like that relating to sweetness and the scarcity of sugars in the ancestral environment, not just a general understanding that tastiness is not necessarily the same as healthiness. Without a clear understanding of the larger patterns to an AI’s perceptual categories—the inscrutable matrix problem—it is simply not possible to derive insights analogous to the one about sugar and ice cream.
I’m much less STEM-oriented than most people here, so I could just be totally misunderstanding the points made in this post, but I tried reading it anyway, and a couple of things stood out to me as possibly mistaken:
Am I missing something here, or is this just describing memetics? Granted, skills, knowledge, values, traditions, etc., are heritable in other ways than purely by lineal descent, but parents do also impart these to their children, and these are subject to evolution.
But isn’t this solely because we have already studied our sensory organs and have a concept of taste buds, and hence flavors like sweet, etc. as primary categories of taste? It is not clear to me that we can do the same thing with regard to eg. ethics, even where humans are concerned. Does this not illustrate Yudkowsky’s point about inscrutable matrices?
It is not describing memetics, which I regard as a mostly confused framework that primes people to misattribute the products of human intelligence to “evolution”. However, even if evolution meaningfully operates on the level of memes, the “Evolution” I’m referring to when I say “Evolution applies very little direct optimization power to the middle level” is strictly biological evolution over the genome, not memetic at all.
Memetic evolution in this context would not have inclusive genetic fitness as its “outer” objective, so whether memetic evolution can “transfer the skills, knowledge, values, or behaviors learned by one generation to their descendants” is irrelevant for the argument I was making in the post.
Not really. The only way our understanding of the biology of taste impacts the story about humans coming to like ice cream is that we can infer that humans have sugar detecting reward circuitry, which ice cream activates in the modern environment. For AI systems, we actually have a better handle on how their reward circuitry works, as compared to the brain. E.g., we can just directly look at the reward counter during the AI’s training.
New memes may arise either by being mutated from other memes or by invention ex nihilo—either of which involves some degree of human intelligence. However, if a meme becomes prevalent, it is not because all of its holders have invented it independently. It has rather spread because it is adapted both to the existing memetic ecosystem as well as to human intelligence. Of course, if certain memes reduce the likelihood of reproduction, that provides an evolutionary pressure for human intelligence to change to be more resistant to that particular kind of meme, so there are very complex interactions.
It is not a confused framework—at least not inherently—and it does not require us to ignore the role of human intelligence.
My argument is that evolution selects simultaneously for genetic and memetic fitness, and that both genes and memes tend to be passed on from parent to child. Thus, evolution operates at a combined genetic-memetic level where it optimizes for inclusive genetic-memetic fitness. Though genes and memes correspond to entirely different mediums, they interact in complex ways when it comes to evolutionary fitness, so the mechanisms are not that straightforwardly separable. In addition, there are social network effects and geographic localization influencing what skills people are likely to acquire, such that skills have a tendency to be heritable in a manner that is not easily reducible to genetics, but which nevertheless influences evolutionary fitness. If we look aside from the fact that memes and skills can be transferred in manners other than heredity, then we can sorta model them as an extended genome.
But the reason we can say that it is bad for humans to become addicted to ice cream is because we have an existing paradigm that provides us with a deep understanding of nutrition, and even here, subtle failures in the paradigm have notoriously done serious harms. Do you regard our understanding of morality as more reliable than our understanding of nutrition?
Remember, the context was Yudkowsky’s argument that we lack a paradigm to address systematic failures in which reward circuitry fails to correspond to good action. That is, specific understanding like that relating to sweetness and the scarcity of sugars in the ancestral environment, not just a general understanding that tastiness is not necessarily the same as healthiness. Without a clear understanding of the larger patterns to an AI’s perceptual categories—the inscrutable matrix problem—it is simply not possible to derive insights analogous to the one about sugar and ice cream.