We would need to solve the general problem of avoiding value drift. Value drift is the phenomenon of changing cultural, social, personal, biological and political values over time. We have observed it in human history during every generation. Older generations vary on average in what they want and care about compared to younger generations. More broadly, species have evolved over time, with constant change on Earth as a result of competition, variation, and natural selection. While over short periods of time, value drift tends to look small, over long periods of time, it can seem enormous.
I don’t know what a good solution to this problem would look like, and some proposed solutions—such as a permanent, very strict global regime of coordination to prevent cultural, biological, and artificial evolution—might be worse than the disease they aim to cure. However, without a solution, our distant descendants will likely be very different from us in ways that we consider morally relevant.
How do you expect this value drift to occur in an environment where humans don’t actually have competitive mental/physical capital? Presumably if humans “drift” beyond some desired bounds, the coalition of people or AIs with the actual compute and martial superiority who are allowing the humans to have these social and political fights will intervene. It wouldn’t matter if there was one AI or twenty, because the N different optimizers are still around and optimizing for the same thing, and the equilibrium between them wouldn’t be meaningfully shifted by culture wars between the newly formed humans.
Value drift happens in almost any environment in which there is variation and selection among entities in the world over time. I’m mostly just saying that things will likely continue to change continuously over the long-term, and a big part of that is that the behavioral tendencies, desires and physical makeup of the relevant entities in the world will continue to evolve too, absent some strong countervailing force to prevent that. This feature of our world does not require that humans continue to have competitive mental and physical capital. On the broadest level, the change I’m referring to took place before humans ever existed.
Can you specify the problem(s) you think we might need to solve, in addition to alignment, in order to avoid this sterility outcome?
We would need to solve the general problem of avoiding value drift. Value drift is the phenomenon of changing cultural, social, personal, biological and political values over time. We have observed it in human history during every generation. Older generations vary on average in what they want and care about compared to younger generations. More broadly, species have evolved over time, with constant change on Earth as a result of competition, variation, and natural selection. While over short periods of time, value drift tends to look small, over long periods of time, it can seem enormous.
I don’t know what a good solution to this problem would look like, and some proposed solutions—such as a permanent, very strict global regime of coordination to prevent cultural, biological, and artificial evolution—might be worse than the disease they aim to cure. However, without a solution, our distant descendants will likely be very different from us in ways that we consider morally relevant.
How do you expect this value drift to occur in an environment where humans don’t actually have competitive mental/physical capital? Presumably if humans “drift” beyond some desired bounds, the coalition of people or AIs with the actual compute and martial superiority who are allowing the humans to have these social and political fights will intervene. It wouldn’t matter if there was one AI or twenty, because the N different optimizers are still around and optimizing for the same thing, and the equilibrium between them wouldn’t be meaningfully shifted by culture wars between the newly formed humans.
Value drift happens in almost any environment in which there is variation and selection among entities in the world over time. I’m mostly just saying that things will likely continue to change continuously over the long-term, and a big part of that is that the behavioral tendencies, desires and physical makeup of the relevant entities in the world will continue to evolve too, absent some strong countervailing force to prevent that. This feature of our world does not require that humans continue to have competitive mental and physical capital. On the broadest level, the change I’m referring to took place before humans ever existed.
I do not understand how you expect this general tendency to continue against the will of one or more nearby gpu clusters.