The utility function is fitness: gene replication count (of the human defining genes)[1]. And by this measure, it is obvious that humans are enormously successful. If we normalize so that a utility score of 1 represents a mild success—the expectation of a typical draw of a great apes species, then humans’ score is >4 OOM larger, completely off the charts.[2]
Footnote 1 says:
Nitpick arguments about how you define this specifically are irrelevant and uninteresting.
Excuse me, what? This is not evolution’s utility function. It’s not optimizing for gene count. It does one thing, one thing only, and it does it well: it promotes genes that increase their RELATIVE FREQUENCY in the reproducing population.
The failure of alignment is witnessed by the fact that humans very very obviously fail to maximize the relative frequency of their genes in the next generation, given the opportunities available to them; and they are often aware of this; and they often choose to do so anyway. The whole argument in this post is totally invalid.
I don’t understand why you say promoting genes relative frequency is how it should be defined. Wouldn’t a gene drive like thing then max out that measure?
Also promoting genes that caused species extinction would also count as a win by that metric. I think that can happen sometimes—i.e. larger individuals are more mate-worthy and the species gets ever bigger (or suchlike) until it doesn’t fit its niche and goes extinct. These seem like failure modes rather than the utility function. Are there protections against gene drives in organizations populations?
IIUC a lot of DNA in a lot of species is gene-drive-like things.
These seem like failure modes rather than the utility function.
By what standard are you judging when something is a failure mode or a desired outcome? I’m saying that what evolution is, is a big search process for genes that increase their relative frequency given the background gene pool. When evolution built humans, it didn’t build agents that try to promote the relative frequency of the genes that they are carrying. Hence, inner misalignment and sharp left turn.
Yes, gene drives have very high (gene-level) fitness. Genes that don’t get carried along with the drive can improve their own gene-level fitness by preventing gene drives from taking hold, so I’d expect there to also be machinery to suppress gene drives, if that’s easy enough to evolve.
If gene drives having high gene-level fitness seems wrong to you, then read this: https://www.lesswrong.com/posts/gDNrpuwahdRrDJ9iY/evolving-to-extinction. Or, if you have more time, Dawkins’s The Selfish Gene is quite good. Evolution is not anthropomorphic, and doesn’t try to avoid “failure modes” like extinction, it’s just a result of selection on a reproducing population.
I have read selfish gene—I think the metric is about relative frequency doesn’t work. A gene that reduced the population from 1 million to 10 but increased is abundance to 100% would max out that metric. If it made the entire species go extinct, what even does the metric say?
Obviously the common understanding is that is an evolutionary failure, but the metric disagrees. Not sure what kind of argument you would accept against your metric capturing the essence of evolution.
If you’ve read The Selfish Gene, then would you agree that under Dawkins’s notion of gene-level fitness, the genes composing a gene drive have high gene-level fitness? If not, why?
Not sure what kind of argument you would accept against your metric capturing the essence of evolution.
Always a good question to ask. TekhneMakre gives a good example of a situation where the two metrics disagree in this comment. Quoting:
Say you have a species. Say you have two genes, A and B.
Gene A has two effects:
A1. Organisms carrying gene A reproduce slightly MORE than organisms not carrying A.
A2. For every copy of A in the species, every organism in the species (carrier or not) reproduces slightly LESS than it would have if not for this copy of A.
Gene B has two effects, the reverse of A:
B1. Organisms carrying gene B reproduce slightly LESS than organisms not carrying B.
B2. For every copy of B in the species, every organism in the species (carrier or not) reproduces slightly MORE than it would have if not for this copy of B.
If the relative frequency metric captures the essence of evolution, then gene A should be successful and gene B should be unsuccessful. Conversely, the total-abundance metric you suggest implies that gene B should be successful while gene A should be unsuccessful.
So one argument that would change my mind is if your showed (eg. via simulation, or just by a convincing argument) that gene B becomes prevalent in this hypothetical scenario.
Nitpick arguments about how you define this specifically are irrelevant and uninteresting.
Excuse me, what? This is not evolution’s utility function. It’s not optimizing for gene count. It does one thing, one thing only, and it does it well: it promotes genes that increase their RELATIVE FREQUENCY in the reproducing population.
I don’t see how this detail is relevant. The fact remains that humans are, in evolutionary terms, much more successful than most other mammals.
The failure of alignment is witnessed by the fact that humans very very obviously fail to maximize the relative frequency of their genes in the next generation, given the opportunities available to them; and they are often aware of this; and they often choose to do so anyway. The whole argument in this post is totally invalid.
Currently the world fertility rate is 2.3 children per woman. This is more than the often quoted 2.1 for a stable population. It is true that many developed countries are currently below that value. But this only means that these subpopulations will go extinct, while other subpopulations with higher fertility inherit the world. E.g. in Africa, Muslim countries, the Amish etc.
For example, in women high fertility is associated with, and likely caused by, low IQ and low education. The popular theory is that smart well-educated women have profitable and satisfying careers and consequently see much higher opportunity cost for having children instead, so they have fewer or none. And people with cognitive properties that cause them to have more children likely are currently getting more frequent in the population. Same as men who tend to forget to use condoms etc. These outlooks are not rosy from an ethical perspective, since properties like low intelligence and low conscientiousness are associated with a lot of human hardship such as poverty and violence, but they are totally in line with what evolution optimizes for.
Good question. I think a good approximation is total body mass of the population. By this measure, only cattle are more successful than humans, and obviously only because we deliberately breed them for food: https://xkcd.com/1338/
In other words, you’re pointing out that the people who have the most ability to choose how many children to have, choose on average to have fewer and therefore to reduce their genes’ relative frequencies in the next generation. They also have longer generation times, amplifying the effect. This is equivalent to “humans defect against evolution’s goals as soon as they have the opportunity to do so.”
Subpopulations which do this are expected to disappear relatively quickly in evolutionary time scales. Natural selection is error correcting. This can mean people get less intelligent again, or they start to really love getting children rather than enjoying sex.
Correct. In a few handfuls of generations the population shifts in those directions. This holds as long as evolutionary timescales are the important ones, but it it not at all clear to me that this is what matters today. If the subpopulations that do this move faster than even short evolutionary timescales, then the selection pressure is light enough that they can oppose it.
I have no problem with a world where people evolve towards wanting more children or caring less about sex itself. I think if it were easy for evolution to achieve that it would have happened a long time ago, which means I think in practice we’re selecting among subpopulations for cultural factors more than biological ones. That, in turn, means those subpopulations are susceptible to outside influence… which is mostly what we’ve been seeing for the whole timeline of fertility dropping as nations develop economically. Some communities are more resistant because they start with stronger beliefs on this front, but none are immune. And frankly I think the only way they could be immune is by enforcing kinds of rigidity that the larger world won’t want to permit, seeing as they’d be seen as abusive, especially to children.
As far as intelligence goes: a world where the average person gets dumber while a small elite becomes smarter and more powerful and wealthier (by starting companies, inventing technologies, controlling policy making, and adopting things like life extension and genetic screening as they become available and viable) is an unstable powder keg. Eventually there’s conflict. Who wins? That depends on how big the gap is. Today, the smart people would probably lose, overwhelmed by numbers and lack of coordination ability. In the future, when the boundaries between populations are stark and stable cultural divides? Well by then the smarter subpopulation has all sorts of options. Like a custom virus that targets genetic characteristics of the low-intelligence subpopulation and causes infertility. Or armies of robots to supplement their numbers. Or radically longer lives so that they at least increase in absolute numbers over time and can have more children per lifetime (just slower) if they want to (note: this would enable people who love having kids to have even more of them!). Or carefully targeted propaganda campaigns (memetic warfare) to break down the cultural wall that’s sustaining the differences.
Nitpick arguments about how you define this specifically are irrelevant and uninteresting.
Excuse me, what? This is not evolution’s utility function. It’s not optimizing for gene count. It does one thing, one thing only, and it does it well: it promotes genes that increase their RELATIVE FREQUENCY in the reproducing population.
It is completely irrelevant to my larger point unless you are claiming that changing out the specific detailed choice of evolutionary utility function would change the high level outcome of humans being successful (relatively high scoring) according to that utility function, or more importantly—simply not being a failure.
My analogy is between:
humans optimizing AI according to human utility function which actually results in extinction (0 score according to that utility function) due to inner misalignment
and
evolution optimizing brains according to some utility function which actually results in extinction (0 score according to that utility function) due to inner misalignment
No part of my argument depends on the specific nitpick detail of the evolution utility function, other than it outputs a non failure score for human history to date.
So are you actually arguing that humans are an evolutionary failure? Or that we simply got lucky as the development of early technoculture usually results in extinction? Any doomer predictions of our future extinction can’t be used as part of a doom argument—that’s just circular reasoning. The evidence to date is that brains were an enormous success according to any possible reasonable evolutionary utility function.
Or maybe you simply dislike the analogy because it doesn’t support your strongly held beliefs? Fine, but at least make that clear.
Evolution optimizes replicators whenever there is variation and selection over the replicated information. The units of replication are ultimately just bit sequences, some replicate more successfully than others, and replication count is thus the measure of replication success.
You can then consider individual genes or more complex distributions over gene sets that define species, but either way, some of these things replicate more successfully than others.
It’s not optimizing for gene count.
Sigh—what?
The distribution over genes changes over time, and the process driving that change is obviously an evolutionary optimization process, because there are 1.) mechanisms causing gene variation (mutation etc), and 2.) selection—some genes replicate more than others.
Thus changes which increase gene count are more frequent/likely, creating a clear direction favored by evolutionary optimization.
Say you have a species. Say you have two genes, A and B.
Gene A has two effects:
A1. Organisms carrying gene A reproduce slightly MORE than organisms not carrying A.
A2. For every copy of A in the species, every organism in the species (carrier or not) reproduces slightly LESS than it would have if not for this copy of A.
Gene B has two effects, the reverse of A:
B1. Organisms carrying gene B reproduce slightly LESS than organisms not carrying B.
B2. For every copy of B in the species, every organism in the species (carrier or not) reproduces slightly MORE than it would have if not for this copy of B.
So now what happens with this species? Answer: A is promoted to fixation, whether or not this causes the species to go extinct; B is eliminated from the gene pool. Evolution doesn’t search to increase total gene count, it searches to increase relative frequency. (Note that this is not resting specifically on the species being a sexually reproducing species. It does rest on the fixedness of the niche capacity. When the niche doesn’t have fixed capacity, evolution is closer to selecting for increasing gene count. But this doesn’t last long; the species grows to fill capacity, and then you’re back to zero-sum selection.)
Sure, genes and species (defined as distributions over gene packages) are separate replicators. Both replicate according to variation and selection, so both evolve.
Notice in your example gene A could actually fail in the long run if it’s too successful, which causes optimization pressure at the larger system/package level to protect against these misalignments (see all the various defenses against transposons, retroviruses, etc).
Ok so the point is that the vast vast majority of optimization power coming from {selection over variation in general} is coming more narrowly from {selection for genes that increase their relative frequency in the gene pool} and not from {selection between different species / other large groups}. In arguments about misalignment, evolution refers to {selection for genes that increase their relative frequency in the gene pool}.
If you run a big search process, and then pick a really extreme actual outcome X of the search process, and then go back and say “okay, the search process was all along a search for X”, then yeah, there’s no such thing as misalignment. But there’s still such a thing as a search process visibly searching for Y and getting some extreme and non-Y-ish outcome, and {selection for genes that increase their relative frequency in the gene pool} is an example.
Ok so the point is that the vast vast majority of optimization power coming from {selection over variation in general} is coming more narrowly from {selection for genes that increase their relative frequency in the gene pool} and not from {selection between different species / other large groups}.
No—Selection is over the distribution that defines the species set (and recursively over the fractal clusters within that down to individuals), and operates at the granularity of complete gene packages (individuals), not individual genes.
If you run a big search process, and then pick a really extreme actual outcome X of the search process, and then go back and say “okay, the search process was all along a search for X”, then yeah, there’s no such thing as misalignment.
The search process is just searching for designs that replicate well in environment. There could be misalignment in theory—as I discussed that would manifest as species tending to go extinct right around the early technocultural transition, when you have a massive sudden capability gain due to the exploding power of within lifetime learning/optimization.
So the misalignment is possible in theory, but we do not have evidence of that in the historical record. We don’t live in that world.
The search process is just searching for designs that replicate well in environment.
This is a retcon, as I described here:
If you run a big search process, and then pick a really extreme actual outcome X of the search process, and then go back and say “okay, the search process was all along a search for X”, then yeah, there’s no such thing as misalignment. But there’s still such a thing as a search process visibly searching for Y and getting some extreme and non-Y-ish outcome, and {selection for genes that increase their relative frequency in the gene pool} is an example.
You’re speaking as though humanity is the very first example of a species that reproduced a lot, but it’s always been the case that some species reproduced more than others and left more descendant species—the ancestor of mammals or eukaryotes, for example. This force has been constant and significant for as long as evolution has been a thing(more selection happens at the within-species level, sure, but that doesn’t mean between-species selection is completely unprecedented)
Within an organism there are various forms of viral genes which reproduce themselves largely at the expense of cells, organs, or the whole organism. A species is actually composed recursively of groups with geographically varying gene distributions and some
genes can grow within local populations at the expense of that local population. But that is counteracted by various mechanisms selecting at larger scales, and all of this is happening at many levels simultaneously well beyond that of just gene and species. A species decomposes fractally geographically into many diverse subgroups with some but limited gene flow (slowing the spread of faster ‘defecting’ viral like genes) and which are all in various forms of competition over time, but still can interbreed and thus are part of the same species.
Excuse me, what? This is not evolution’s utility function.
The world is composed of patterns. Some patterns replicate extensively, so pattern measure varies over many OOM. The patterns both mutate and variably replicate, so over time the distribution over patterns changes.
In fully generality, an optimization process applies computation to update the distribution over patterns. This process has a general temporal arrow—a direction/gradient. The ‘utility function’ is thus simply the function F such that dF/dt describes the gradient. For evolution in general, this is obviously pattern measure, and truly can not be anything else.
The failure of alignment is witnessed by the fact that humans very very obviously fail to maximize the relative frequency of their genes in the next generation,
Typically most individual members of a species will fail to replicate at all, but that is irrelevant on several levels: their genes are not limited to the one physical soma, and the species is not any individual regardless. In fact if every individual reproduced at the same rate, the species would not evolve—as selection requires differential individual failure to drive species success.
Alignment in my analogy has a precise definition and measurable outcome: species success. Any inner/outer alignment failure results in species extinction, or it wasn’t an alignment failure, period. This definition applies identically to the foom doom scenario (where AGI causes human extinction), and the historical analogy (where evolution of linguistic intelligence could cause a species to go extinct because they decide to stop reproducing).
For evolution in general, this is obviously pattern measure, and truly can not be anything else.
This sure sounds like my attempt elsewhere to describe your position:
There’s no such thing as misalignment. There’s one overarching process, call it evolution or whatever you like, and this process goes through stages of creating new things along new dimensions, but all the stages are part of the overall process. Anything called “misalignment” is describing the relationship of two parts or stages that are contained in the overarching process. The overarching process is at a higher level than that misalignment relationship, and the misalignment helps compute the overarching process.
One evolutionary process but many potential competing sub-components. Of course there is always misalignment.
The implied optimization gradient of any two different components of the system can never be perfectly aligned (as otherwise they wouldn’t be different).
The foom doom argument is that humanity and AGI will be very misaligned such that the latter’s rise results in the extinction of the former.
The analogy from historical evolution is the misalignment between human genes and human minds, where the rise of the latter did not result in extinction of the former. It plausibly could have, but that is not what we observe.
The analogy from historical evolution is the misalignment between human genes and human minds, where the rise of the latter did not result in extinction of the former. It plausibly could have, but that is not what we observe.
The analogy is that the human genes thing produces a thing (human minds) which wants stuff, but the stuff it wants is different from what what the human genes want. From my perspective you’re strawmanning and failing to track the discourse here to a sufficient degree that I’m bowing out.
The analogy is that the human genes thing produces a thing (human minds) which wants stuff, but the stuff it wants is different from what what the human genes want.
Not nearly different enough to prevent the human genes from getting what they want in excess.
If we apply your frame of the analogy to AGI, we have slightly misaligned AGI which doesn’t cause human extinction, and instead enormously amplifies our utility.
From my perspective you’re strawmanning and failing to track the discourse here to a sufficient degree that I’m bowing out
From my perspective you persistently ignore, misunderstand, or misrepresent my arguments, overfocus on pedantic details, and refuse to update or agree on basics.
You write:
Footnote 1 says:
Excuse me, what? This is not evolution’s utility function. It’s not optimizing for gene count. It does one thing, one thing only, and it does it well: it promotes genes that increase their RELATIVE FREQUENCY in the reproducing population.
The failure of alignment is witnessed by the fact that humans very very obviously fail to maximize the relative frequency of their genes in the next generation, given the opportunities available to them; and they are often aware of this; and they often choose to do so anyway. The whole argument in this post is totally invalid.
I don’t understand why you say promoting genes relative frequency is how it should be defined. Wouldn’t a gene drive like thing then max out that measure?
Also promoting genes that caused species extinction would also count as a win by that metric. I think that can happen sometimes—i.e. larger individuals are more mate-worthy and the species gets ever bigger (or suchlike) until it doesn’t fit its niche and goes extinct. These seem like failure modes rather than the utility function. Are there protections against gene drives in organizations populations?
IIUC a lot of DNA in a lot of species is gene-drive-like things.
By what standard are you judging when something is a failure mode or a desired outcome? I’m saying that what evolution is, is a big search process for genes that increase their relative frequency given the background gene pool. When evolution built humans, it didn’t build agents that try to promote the relative frequency of the genes that they are carrying. Hence, inner misalignment and sharp left turn.
Yes, gene drives have very high (gene-level) fitness. Genes that don’t get carried along with the drive can improve their own gene-level fitness by preventing gene drives from taking hold, so I’d expect there to also be machinery to suppress gene drives, if that’s easy enough to evolve.
If gene drives having high gene-level fitness seems wrong to you, then read this: https://www.lesswrong.com/posts/gDNrpuwahdRrDJ9iY/evolving-to-extinction. Or, if you have more time, Dawkins’s The Selfish Gene is quite good. Evolution is not anthropomorphic, and doesn’t try to avoid “failure modes” like extinction, it’s just a result of selection on a reproducing population.
I have read selfish gene—I think the metric is about relative frequency doesn’t work. A gene that reduced the population from 1 million to 10 but increased is abundance to 100% would max out that metric. If it made the entire species go extinct, what even does the metric say?
Obviously the common understanding is that is an evolutionary failure, but the metric disagrees. Not sure what kind of argument you would accept against your metric capturing the essence of evolution.
If you’ve read The Selfish Gene, then would you agree that under Dawkins’s notion of gene-level fitness, the genes composing a gene drive have high gene-level fitness? If not, why?
Always a good question to ask. TekhneMakre gives a good example of a situation where the two metrics disagree in this comment. Quoting:
If the relative frequency metric captures the essence of evolution, then gene A should be successful and gene B should be unsuccessful. Conversely, the total-abundance metric you suggest implies that gene B should be successful while gene A should be unsuccessful.
So one argument that would change my mind is if your showed (eg. via simulation, or just by a convincing argument) that gene B becomes prevalent in this hypothetical scenario.
Yes thanks, that thread goes over it in more detail than I could.
I don’t see how this detail is relevant. The fact remains that humans are, in evolutionary terms, much more successful than most other mammals.
Currently the world fertility rate is 2.3 children per woman. This is more than the often quoted 2.1 for a stable population. It is true that many developed countries are currently below that value. But this only means that these subpopulations will go extinct, while other subpopulations with higher fertility inherit the world. E.g. in Africa, Muslim countries, the Amish etc.
For example, in women high fertility is associated with, and likely caused by, low IQ and low education. The popular theory is that smart well-educated women have profitable and satisfying careers and consequently see much higher opportunity cost for having children instead, so they have fewer or none. And people with cognitive properties that cause them to have more children likely are currently getting more frequent in the population. Same as men who tend to forget to use condoms etc. These outlooks are not rosy from an ethical perspective, since properties like low intelligence and low conscientiousness are associated with a lot of human hardship such as poverty and violence, but they are totally in line with what evolution optimizes for.
What do you mean by “in evolutionary terms, much more successful”?
Good question. I think a good approximation is total body mass of the population. By this measure, only cattle are more successful than humans, and obviously only because we deliberately breed them for food: https://xkcd.com/1338/
In other words, you’re pointing out that the people who have the most ability to choose how many children to have, choose on average to have fewer and therefore to reduce their genes’ relative frequencies in the next generation. They also have longer generation times, amplifying the effect. This is equivalent to “humans defect against evolution’s goals as soon as they have the opportunity to do so.”
Subpopulations which do this are expected to disappear relatively quickly in evolutionary time scales. Natural selection is error correcting. This can mean people get less intelligent again, or they start to really love getting children rather than enjoying sex.
Correct. In a few handfuls of generations the population shifts in those directions. This holds as long as evolutionary timescales are the important ones, but it it not at all clear to me that this is what matters today. If the subpopulations that do this move faster than even short evolutionary timescales, then the selection pressure is light enough that they can oppose it.
I have no problem with a world where people evolve towards wanting more children or caring less about sex itself. I think if it were easy for evolution to achieve that it would have happened a long time ago, which means I think in practice we’re selecting among subpopulations for cultural factors more than biological ones. That, in turn, means those subpopulations are susceptible to outside influence… which is mostly what we’ve been seeing for the whole timeline of fertility dropping as nations develop economically. Some communities are more resistant because they start with stronger beliefs on this front, but none are immune. And frankly I think the only way they could be immune is by enforcing kinds of rigidity that the larger world won’t want to permit, seeing as they’d be seen as abusive, especially to children.
As far as intelligence goes: a world where the average person gets dumber while a small elite becomes smarter and more powerful and wealthier (by starting companies, inventing technologies, controlling policy making, and adopting things like life extension and genetic screening as they become available and viable) is an unstable powder keg. Eventually there’s conflict. Who wins? That depends on how big the gap is. Today, the smart people would probably lose, overwhelmed by numbers and lack of coordination ability. In the future, when the boundaries between populations are stark and stable cultural divides? Well by then the smarter subpopulation has all sorts of options. Like a custom virus that targets genetic characteristics of the low-intelligence subpopulation and causes infertility. Or armies of robots to supplement their numbers. Or radically longer lives so that they at least increase in absolute numbers over time and can have more children per lifetime (just slower) if they want to (note: this would enable people who love having kids to have even more of them!). Or carefully targeted propaganda campaigns (memetic warfare) to break down the cultural wall that’s sustaining the differences.
It is completely irrelevant to my larger point unless you are claiming that changing out the specific detailed choice of evolutionary utility function would change the high level outcome of humans being successful (relatively high scoring) according to that utility function, or more importantly—simply not being a failure.
My analogy is between:
humans optimizing AI according to human utility function which actually results in extinction (0 score according to that utility function) due to inner misalignment
and
evolution optimizing brains according to some utility function which actually results in extinction (0 score according to that utility function) due to inner misalignment
No part of my argument depends on the specific nitpick detail of the evolution utility function, other than it outputs a non failure score for human history to date.
So are you actually arguing that humans are an evolutionary failure? Or that we simply got lucky as the development of early technoculture usually results in extinction? Any doomer predictions of our future extinction can’t be used as part of a doom argument—that’s just circular reasoning. The evidence to date is that brains were an enormous success according to any possible reasonable evolutionary utility function.
Or maybe you simply dislike the analogy because it doesn’t support your strongly held beliefs? Fine, but at least make that clear.
Evolution optimizes replicators whenever there is variation and selection over the replicated information. The units of replication are ultimately just bit sequences, some replicate more successfully than others, and replication count is thus the measure of replication success.
You can then consider individual genes or more complex distributions over gene sets that define species, but either way, some of these things replicate more successfully than others.
Sigh—what?
The distribution over genes changes over time, and the process driving that change is obviously an evolutionary optimization process, because there are 1.) mechanisms causing gene variation (mutation etc), and 2.) selection—some genes replicate more than others.
Thus changes which increase gene count are more frequent/likely, creating a clear direction favored by evolutionary optimization.
Say you have a species. Say you have two genes, A and B.
Gene A has two effects:
A1. Organisms carrying gene A reproduce slightly MORE than organisms not carrying A.
A2. For every copy of A in the species, every organism in the species (carrier or not) reproduces slightly LESS than it would have if not for this copy of A.
Gene B has two effects, the reverse of A:
B1. Organisms carrying gene B reproduce slightly LESS than organisms not carrying B.
B2. For every copy of B in the species, every organism in the species (carrier or not) reproduces slightly MORE than it would have if not for this copy of B.
So now what happens with this species? Answer: A is promoted to fixation, whether or not this causes the species to go extinct; B is eliminated from the gene pool. Evolution doesn’t search to increase total gene count, it searches to increase relative frequency. (Note that this is not resting specifically on the species being a sexually reproducing species. It does rest on the fixedness of the niche capacity. When the niche doesn’t have fixed capacity, evolution is closer to selecting for increasing gene count. But this doesn’t last long; the species grows to fill capacity, and then you’re back to zero-sum selection.)
Sure, genes and species (defined as distributions over gene packages) are separate replicators. Both replicate according to variation and selection, so both evolve.
Notice in your example gene A could actually fail in the long run if it’s too successful, which causes optimization pressure at the larger system/package level to protect against these misalignments (see all the various defenses against transposons, retroviruses, etc).
Ok so the point is that the vast vast majority of optimization power coming from {selection over variation in general} is coming more narrowly from {selection for genes that increase their relative frequency in the gene pool} and not from {selection between different species / other large groups}. In arguments about misalignment, evolution refers to {selection for genes that increase their relative frequency in the gene pool}.
If you run a big search process, and then pick a really extreme actual outcome X of the search process, and then go back and say “okay, the search process was all along a search for X”, then yeah, there’s no such thing as misalignment. But there’s still such a thing as a search process visibly searching for Y and getting some extreme and non-Y-ish outcome, and {selection for genes that increase their relative frequency in the gene pool} is an example.
No—Selection is over the distribution that defines the species set (and recursively over the fractal clusters within that down to individuals), and operates at the granularity of complete gene packages (individuals), not individual genes.
The search process is just searching for designs that replicate well in environment. There could be misalignment in theory—as I discussed that would manifest as species tending to go extinct right around the early technocultural transition, when you have a massive sudden capability gain due to the exploding power of within lifetime learning/optimization.
So the misalignment is possible in theory, but we do not have evidence of that in the historical record. We don’t live in that world.
This is a retcon, as I described here:
You’re speaking as though humanity is the very first example of a species that reproduced a lot, but it’s always been the case that some species reproduced more than others and left more descendant species—the ancestor of mammals or eukaryotes, for example. This force has been constant and significant for as long as evolution has been a thing(more selection happens at the within-species level, sure, but that doesn’t mean between-species selection is completely unprecedented)
Within an organism there are various forms of viral genes which reproduce themselves largely at the expense of cells, organs, or the whole organism. A species is actually composed recursively of groups with geographically varying gene distributions and some genes can grow within local populations at the expense of that local population. But that is counteracted by various mechanisms selecting at larger scales, and all of this is happening at many levels simultaneously well beyond that of just gene and species. A species decomposes fractally geographically into many diverse subgroups with some but limited gene flow (slowing the spread of faster ‘defecting’ viral like genes) and which are all in various forms of competition over time, but still can interbreed and thus are part of the same species.
The world is composed of patterns. Some patterns replicate extensively, so pattern measure varies over many OOM. The patterns both mutate and variably replicate, so over time the distribution over patterns changes.
In fully generality, an optimization process applies computation to update the distribution over patterns. This process has a general temporal arrow—a direction/gradient. The ‘utility function’ is thus simply the function F such that dF/dt describes the gradient. For evolution in general, this is obviously pattern measure, and truly can not be anything else.
Typically most individual members of a species will fail to replicate at all, but that is irrelevant on several levels: their genes are not limited to the one physical soma, and the species is not any individual regardless. In fact if every individual reproduced at the same rate, the species would not evolve—as selection requires differential individual failure to drive species success.
Alignment in my analogy has a precise definition and measurable outcome: species success. Any inner/outer alignment failure results in species extinction, or it wasn’t an alignment failure, period. This definition applies identically to the foom doom scenario (where AGI causes human extinction), and the historical analogy (where evolution of linguistic intelligence could cause a species to go extinct because they decide to stop reproducing).
This sure sounds like my attempt elsewhere to describe your position:
Which you dismissed.
One evolutionary process but many potential competing sub-components. Of course there is always misalignment.
The implied optimization gradient of any two different components of the system can never be perfectly aligned (as otherwise they wouldn’t be different).
The foom doom argument is that humanity and AGI will be very misaligned such that the latter’s rise results in the extinction of the former.
The analogy from historical evolution is the misalignment between human genes and human minds, where the rise of the latter did not result in extinction of the former. It plausibly could have, but that is not what we observe.
The analogy is that the human genes thing produces a thing (human minds) which wants stuff, but the stuff it wants is different from what what the human genes want. From my perspective you’re strawmanning and failing to track the discourse here to a sufficient degree that I’m bowing out.
Not nearly different enough to prevent the human genes from getting what they want in excess.
If we apply your frame of the analogy to AGI, we have slightly misaligned AGI which doesn’t cause human extinction, and instead enormously amplifies our utility.
From my perspective you persistently ignore, misunderstand, or misrepresent my arguments, overfocus on pedantic details, and refuse to update or agree on basics.