Does not follow. We value diversity, but not diversity of utility functions. This allows diversity because some terms in the utility function are subjective (i.e. they depend on the agent’s brain). If I have “people should drink tasty beverages” as a terminal value, I have “MixedNuts should drink mango juice” and “Helen should drink tea”, and if a cosmic ray alters Helen’s brain, the latter will change. Helen.tasty is not an approximation of a mysterious Platonic essence of absolute tastiness, unlike Helen.prove_theorem.
(Maybe we also want some diversity in terminal values, but it’ll probably be very small—we aren’t going to sacrifice the galaxy to cheesecake makers.)
Does not follow. We value diversity, but not diversity of utility functions. This allows diversity because some terms in the utility function are subjective (i.e. they depend on the agent’s brain). If I have “people should drink tasty beverages” as a terminal value, I have “MixedNuts should drink mango juice” and “Helen should drink tea”, and if a cosmic ray alters Helen’s brain, the latter will change. Helen.tasty is not an approximation of a mysterious Platonic essence of absolute tastiness, unlike Helen.prove_theorem.
(Maybe we also want some diversity in terminal values, but it’ll probably be very small—we aren’t going to sacrifice the galaxy to cheesecake makers.)