In thinking about what alignment should be aiming towards, the phrase I’ve been coming back to a lot recently is “sustainable cooperative harmony”.
“Sustainable” because an aligned future shouldn’t exploit all resources until collapse. “Cooperative” because an aligned future should be about solving coordination problems, steering the goals of every subsystem into alignment with those of every other subsystem. “Harmony” because an aligned future should be beautiful, like the trillions of cells working in concert to create a unified organism, or like the hundreds of diverse instruments playing diverse notes in a unified symphony.
Harmony in particular is key here. In music, the principles of harmony are not about following pre-defined paths for building a musical piece, as though this chord should always follow that chord or this note should always pair with that note. There is no fixed formula for creating good harmony. Instead, creating harmony is really about avoiding disharmony. Certain notes will sound discordant when played together, so as long as you avoid that, the symphony can theoretically have as much going on at once as you would like. (Melody is similar, except that it is harmony stretched across time in sequence rather than in parallel.)
Similarly, harmony at the level of future civilization should be about avoiding conflicts, not forcing everyone into lockstep with a certain ideal. Maximizing diversity of thought, behavior, values, and goals while still preventing the butting of heads. Basically the libertarian ideal of “your right to swing your fist ends at my face”, except more generalized.
There are innumerably many right ways to create Eutopia, but there are vastly more ways to create dystopia. Harmonization/alignment is about finding the thin manifold in civilization-space where disharmony/conflict/death/suffering is minimized. The key insight here, though, is that it is a manifold, not a single-point target. Either way, though, there is almost no way that human minds could find our way there on our own.
If we could build an AGI that is structurally oriented around creating sustainable cooperative harmony in any system it’s involved in (whether repairing a human body, creating prosocial policies for human civilization, or improving the ecological structure of the biosphere), then we would have a shot at creating a future that’s capable of evolving into something far beyond what we could design ourselves.
Well, yes, but I was speaking on the short term of the next few trillion years. For that matter, we have the power to cause irreversible collapse today. I would prefer to see a future that is capable of sustaining something worthwhile for as long as physics allows.
Sustainable is a badly used term for this reason. Lots of inefficient things are sustainable for the short term of a few hundred thousand years. “Unsustainable” should be limited to things that are either collapsing now (climate change) or obviously short term, like low paid workers for essential jobs when demographically the population is slowly losing workers.
In thinking about what alignment should be aiming towards, the phrase I’ve been coming back to a lot recently is “sustainable cooperative harmony”.
“Sustainable” because an aligned future shouldn’t exploit all resources until collapse. “Cooperative” because an aligned future should be about solving coordination problems, steering the goals of every subsystem into alignment with those of every other subsystem. “Harmony” because an aligned future should be beautiful, like the trillions of cells working in concert to create a unified organism, or like the hundreds of diverse instruments playing diverse notes in a unified symphony.
Harmony in particular is key here. In music, the principles of harmony are not about following pre-defined paths for building a musical piece, as though this chord should always follow that chord or this note should always pair with that note. There is no fixed formula for creating good harmony. Instead, creating harmony is really about avoiding disharmony. Certain notes will sound discordant when played together, so as long as you avoid that, the symphony can theoretically have as much going on at once as you would like. (Melody is similar, except that it is harmony stretched across time in sequence rather than in parallel.)
Similarly, harmony at the level of future civilization should be about avoiding conflicts, not forcing everyone into lockstep with a certain ideal. Maximizing diversity of thought, behavior, values, and goals while still preventing the butting of heads. Basically the libertarian ideal of “your right to swing your fist ends at my face”, except more generalized.
There are innumerably many right ways to create Eutopia, but there are vastly more ways to create dystopia. Harmonization/alignment is about finding the thin manifold in civilization-space where disharmony/conflict/death/suffering is minimized. The key insight here, though, is that it is a manifold, not a single-point target. Either way, though, there is almost no way that human minds could find our way there on our own.
If we could build an AGI that is structurally oriented around creating sustainable cooperative harmony in any system it’s involved in (whether repairing a human body, creating prosocial policies for human civilization, or improving the ecological structure of the biosphere), then we would have a shot at creating a future that’s capable of evolving into something far beyond what we could design ourselves.
Well, yes, but I was speaking on the short term of the next few trillion years. For that matter, we have the power to cause irreversible collapse today. I would prefer to see a future that is capable of sustaining something worthwhile for as long as physics allows.
Sustainable is a badly used term for this reason. Lots of inefficient things are sustainable for the short term of a few hundred thousand years. “Unsustainable” should be limited to things that are either collapsing now (climate change) or obviously short term, like low paid workers for essential jobs when demographically the population is slowly losing workers.