Seems useful. I think there are a set of important intuitions you’re gesturing at here around naturality, some of which I may share. I have some take (which may or may not be related) like
utility is a measuring stick which is pragmatically useful in certain situations, because it helps corral your shards (e.g. dogs and diamonds) into executing macro-level sensible plans (where you aren’t throwing away resources which could lead to more dogs and/or diamonds) and not just activating incoherently.
but this doesn’t mean I instantly gain space-time-additive preferences about dogs and diamonds such that I use one utility function in all contexts, such that the utility function is furthermore over universe-histories (funny how I seem to care across Tegmark 4?).
Many observed values in humans and other mammals (see[4]) (e.g. fear, play/boredom, friendship/altruism, love, etc.) seem to be values that were instrumental for increasing inclusive genetic fitness (promoting survival, exploration, cooperation and sexual reproduction/survival of progeny respectively). Yet, humans and mammals seem to value these terminally and not because of their instrumental value on inclusive genetic fitness.
That the instrumentally convergent goals of evolution’s fitness criterion manifested as “terminal” values in mammals is IMO strong empirical evidence against the goals ontology and significant evidence in support of shard theory’s basic account of value formation.
Evolutionarily convergent terminal values, is something that’s underrated I think.
Seems useful. I think there are a set of important intuitions you’re gesturing at here around naturality, some of which I may share. I have some take (which may or may not be related) like
but this doesn’t mean I instantly gain space-time-additive preferences about dogs and diamonds such that I use one utility function in all contexts, such that the utility function is furthermore over universe-histories (funny how I seem to care across Tegmark 4?).
From the post:
Evolutionarily convergent terminal values, is something that’s underrated I think.