I suspect our modelling of values would work better if we went away from the Gnostic approach of seeing values as some inner informational thing like a utility function. I’ve been playing around with an ecological approach. For instance, freedom is an important part of human values, and it’s probably the hardest value to model with utility functions, but it immediately falls out of an ecological approach: because a society built out of humans can make better use of each human’s agency if it allows everyone to take decisions, instead of forcing top-down command. The most classic case is the USA vs the USSR, with the former prospering and the latter stagnating.
The ecological approach is not very transhumanist-friendly. It tends to emphasize things like oxygen or food, which are not very relevant for e.g. uploads. In a sense, I think the lack of transhumanist-friendliness is actually a bonus because it puts the alignment challenges into much more clarity and crispness.
I suspect our modelling of values would work better if we went away from the Gnostic approach of seeing values as some inner informational thing like a utility function. I’ve been playing around with an ecological approach. For instance, freedom is an important part of human values, and it’s probably the hardest value to model with utility functions, but it immediately falls out of an ecological approach: because a society built out of humans can make better use of each human’s agency if it allows everyone to take decisions, instead of forcing top-down command. The most classic case is the USA vs the USSR, with the former prospering and the latter stagnating.
The ecological approach is not very transhumanist-friendly. It tends to emphasize things like oxygen or food, which are not very relevant for e.g. uploads. In a sense, I think the lack of transhumanist-friendliness is actually a bonus because it puts the alignment challenges into much more clarity and crispness.