But a utility function can be lossily projected down into a bounded computational object by factoring it into a few shards, each representing a term in the utility function, each term conceptually chunked out of perceptual input.
I feel confused by the role of “perceptual input” here. Can you give an example of a situation where the utility function gets chunked in this way?
I had meant to suggest that your shards interface with a messy perceptual world of incoming retinal activations and the like, but are trained to nonetheless chunk out latent variables like “human flourishing” or “lollipops” in the input stream. That is, I was suggesting a rough shape for the link between the outside world as you observe it and the ontology your shards express their ends in.
If you formalized utility functions as orderings over possible worlds (or over other equivalent objects!), and your perception simply looked over the set of all possible worlds, then there wouldn’t be anything interesting to explain about perception and the ontology your values are framed in. For agents that can’t run impossibly large computations like that, though, I think you do have something to explain here.
I had meant to suggest that your shards interface with a messy perceptual world of incoming retinal activations and the like, but are trained to nonetheless chunk out latent variables like “human flourishing” or “lollipops” in the input stream. That is, I was suggesting a rough shape for the link between the outside world as you observe it and the ontology your shards express their ends in.
If you formalized utility functions as orderings over possible worlds (or over other equivalent objects!), and your perception simply looked over the set of all possible worlds, then there wouldn’t be anything interesting to explain about perception and the ontology your values are framed in. For agents that can’t run impossibly large computations like that, though, I think you do have something to explain here.