Oh, a thing I forgot to mention about the proposed formalization: if your distribution over utility functions includes some functions that are amenable to change via optimization (e.g. number of paperclips) and some that are not amenable to change via optimization (e.g. number of perpetual motion machines), then any optimization algorithm, including ones we’d naively call “value-neutral”, would lead to distributions of changes in attainable utility with large standard deviation. It might be possible to fix this through some sort of normalization scheme, though I’m not sure how.
Oh, a thing I forgot to mention about the proposed formalization: if your distribution over utility functions includes some functions that are amenable to change via optimization (e.g. number of paperclips) and some that are not amenable to change via optimization (e.g. number of perpetual motion machines), then any optimization algorithm, including ones we’d naively call “value-neutral”, would lead to distributions of changes in attainable utility with large standard deviation. It might be possible to fix this through some sort of normalization scheme, though I’m not sure how.