If you weight expected value by neurons
Ah, but now I can turn myself into a utility monster by artificially enlarging my brain! Game over.
We’re trying to work out how to make progress on moral questions today, not trying to lay down a rule for all eternity that future agents can’t game.
It was a joke.
Oops, sorry!
Or by having kids. Or copying your uploaded self. Or re-engineering your nervous system in other ways...
Ah, but now I can turn myself into a utility monster by artificially enlarging my brain! Game over.
We’re trying to work out how to make progress on moral questions today, not trying to lay down a rule for all eternity that future agents can’t game.
It was a joke.
Oops, sorry!
Or by having kids. Or copying your uploaded self. Or re-engineering your nervous system in other ways...