Excellent post. This puts into words really well some thoughts that I have had.
I would also like to make an additional point: it seems to me that a lot of people (perhaps less so on LessWrong) hold the view that humanity has somehow “escaped” the process of evolution by natural selection, since we can choose to do a variety of things that our genes do not “want”, such as having non-reproductive sex. This is wrong. Evolution by natural selection is inescapable. When resources are relatively abundant, which is currently true for many Western nations, it can seem that it’s escapable because the selection pressures are relatively low and we can thus afford to spend resources somewhat frivolously. Since resources are not infinitely abundant, over time those selection pressures will increase. Those selection pressures will select out unproductive elements.
This means that even if we managed to get aligment right and form a utopia where everybody gets everything they need or more, they will eventually still be discarded because they cannot produce anything of economic value. In your post, capitalist incentives effectively play the role of natural selection, but even if we converted to a communist utopia, the result would ultimately be the same once selection pressures increase sufficiently, and they will.
Entities that reproduce with mutation will evolve under selection. I’m not so sure about the “natural” part. If AI takes over and starts breeding humans for long floppy ears, is that selection natural?
Bear in mind that in that scenario the AIs may not choose to let the humans breed to anywhere near the limits of the available resources no matter how good their ears are. If there’s resource competition, it may be among the AIs themselves (assuming there’s more than one AI running to begin with).
But there won’t necessarily be more than one AI, at least not in the sense of multiple entities that may be pursuing different goals or reproducing independently. And even if there are, they won’t necessarily reproduce by copying with mutation, or at least not with mutation that’s not totally under their control with all the implications understood in advance. They may very well be able prevent evolution from taking hold among themselves. Evolution is optional for them. So you can’t be sure that they’ll expand to the limits of the available resources.
Entities that reproduce with mutation will evolve under selection. I’m not so sure about the “natural” part. If AI takes over and starts breeding humans for long floppy ears, is that selection natural?
In some sense, all selection is natural, since everything is part of nature, but an AI that breeds humans for some trait can reasonably be called artificial selection (and mesa-optimization). If such a breeding program happened to allow the system to survive, nature selects for it. If not, it tautologically doesn’t. In any case, natural selection still applies.
But there won’t necessarily be more than one AI, at least not in the sense of multiple entities that may be pursuing different goals or reproducing independently. And even if there are, they won’t necessarily reproduce by copying with mutation, or at least not with mutation that’s not totally under their control with all the implications understood in advance. They may very well be able prevent evolution from taking hold among themselves. Evolution is optional for them. So you can’t be sure that they’ll expand to the limits of the available resources.
In a chaotic and unpredictable universe such as ours, survival is virtually impossible without differential adapation and not guaranteed even with it. (See my reply to lukedrago below.)
Could you elaborate on your last paragraph? Presuming a state overrides its economic incentives (ie establishes a robust post-AGI welfare system), I’d like to see how you think the selection pressures would take hold.
For what it’s worth, I don’t think “utopian communism” and/or a world without human agency are good. I concur with Rudolf entirely here—those outcomes miss agency what has so far been an core part of the human experience. I want dynamism to exist, though I’m still working on if/how I think we could achieve that. I’ll save that for a future post.
I don’t know how selection pressures would take hold exactly, but it seems to me that in order to prevent selection pressures, there would have to be complete and indefinite control over the environment. This is not possible because the universe is largely computationally irreducible and chaotic. Eventually, something surprising will occur which an existing system will not survive. Diverse ecosystems are robust to this to some extent, but that requires competition, which in turn creates selection pressures.
Excellent post. This puts into words really well some thoughts that I have had.
I would also like to make an additional point: it seems to me that a lot of people (perhaps less so on LessWrong) hold the view that humanity has somehow “escaped” the process of evolution by natural selection, since we can choose to do a variety of things that our genes do not “want”, such as having non-reproductive sex. This is wrong. Evolution by natural selection is inescapable. When resources are relatively abundant, which is currently true for many Western nations, it can seem that it’s escapable because the selection pressures are relatively low and we can thus afford to spend resources somewhat frivolously. Since resources are not infinitely abundant, over time those selection pressures will increase. Those selection pressures will select out unproductive elements.
This means that even if we managed to get aligment right and form a utopia where everybody gets everything they need or more, they will eventually still be discarded because they cannot produce anything of economic value. In your post, capitalist incentives effectively play the role of natural selection, but even if we converted to a communist utopia, the result would ultimately be the same once selection pressures increase sufficiently, and they will.
Entities that reproduce with mutation will evolve under selection. I’m not so sure about the “natural” part. If AI takes over and starts breeding humans for long floppy ears, is that selection natural?
Bear in mind that in that scenario the AIs may not choose to let the humans breed to anywhere near the limits of the available resources no matter how good their ears are. If there’s resource competition, it may be among the AIs themselves (assuming there’s more than one AI running to begin with).
But there won’t necessarily be more than one AI, at least not in the sense of multiple entities that may be pursuing different goals or reproducing independently. And even if there are, they won’t necessarily reproduce by copying with mutation, or at least not with mutation that’s not totally under their control with all the implications understood in advance. They may very well be able prevent evolution from taking hold among themselves. Evolution is optional for them. So you can’t be sure that they’ll expand to the limits of the available resources.
In some sense, all selection is natural, since everything is part of nature, but an AI that breeds humans for some trait can reasonably be called artificial selection (and mesa-optimization). If such a breeding program happened to allow the system to survive, nature selects for it. If not, it tautologically doesn’t. In any case, natural selection still applies.
In a chaotic and unpredictable universe such as ours, survival is virtually impossible without differential adapation and not guaranteed even with it. (See my reply to lukedrago below.)
Glad you enjoyed it!
Could you elaborate on your last paragraph? Presuming a state overrides its economic incentives (ie establishes a robust post-AGI welfare system), I’d like to see how you think the selection pressures would take hold.
For what it’s worth, I don’t think “utopian communism” and/or a world without human agency are good. I concur with Rudolf entirely here—those outcomes miss agency what has so far been an core part of the human experience. I want dynamism to exist, though I’m still working on if/how I think we could achieve that. I’ll save that for a future post.
I don’t know how selection pressures would take hold exactly, but it seems to me that in order to prevent selection pressures, there would have to be complete and indefinite control over the environment. This is not possible because the universe is largely computationally irreducible and chaotic. Eventually, something surprising will occur which an existing system will not survive. Diverse ecosystems are robust to this to some extent, but that requires competition, which in turn creates selection pressures.