One truth that is universal, relevant, and not taken into account, is that demanding everyone converge on the same utility function is suboptimal, probably even when evaluated from that utility function. To restate in terms of culture: If you convinced everybody around the world to wear blue jeans, drink Coke, and listen to rock and roll, I guarantee that 60 years in the future, your grandkids would get less enjoyment out of their world, even if they still love jeans, Coke, and rock and roll. Diversity is needed to generate memes very much like it is needed to generate genes.
Does not follow. We value diversity, but not diversity of utility functions. This allows diversity because some terms in the utility function are subjective (i.e. they depend on the agent’s brain). If I have “people should drink tasty beverages” as a terminal value, I have “MixedNuts should drink mango juice” and “Helen should drink tea”, and if a cosmic ray alters Helen’s brain, the latter will change. Helen.tasty is not an approximation of a mysterious Platonic essence of absolute tastiness, unlike Helen.prove_theorem.
(Maybe we also want some diversity in terminal values, but it’ll probably be very small—we aren’t going to sacrifice the galaxy to cheesecake makers.)
One truth that is universal, relevant, and not taken into account, is that demanding everyone converge on the same utility function is suboptimal, probably even when evaluated from that utility function. To restate in terms of culture: If you convinced everybody around the world to wear blue jeans, drink Coke, and listen to rock and roll, I guarantee that 60 years in the future, your grandkids would get less enjoyment out of their world, even if they still love jeans, Coke, and rock and roll. Diversity is needed to generate memes very much like it is needed to generate genes.
Does not follow. We value diversity, but not diversity of utility functions. This allows diversity because some terms in the utility function are subjective (i.e. they depend on the agent’s brain). If I have “people should drink tasty beverages” as a terminal value, I have “MixedNuts should drink mango juice” and “Helen should drink tea”, and if a cosmic ray alters Helen’s brain, the latter will change. Helen.tasty is not an approximation of a mysterious Platonic essence of absolute tastiness, unlike Helen.prove_theorem.
(Maybe we also want some diversity in terminal values, but it’ll probably be very small—we aren’t going to sacrifice the galaxy to cheesecake makers.)