The main problem I see with it is that it implies wireheading as the optimal outcome.
Or the utilitronium shockwave, rather. Which doesn’t even require minds to wirehead anymore, but simply converts matter into maximally efficient bliss simulations. I used to find this highly counterintuitive, but after thinking about all the absurd implications of valuing preferences instead of actual states of the world, I’ve come to think of it as a perfectly reasonable thing.
Or the utilitronium shockwave, rather. Which doesn’t even require minds to wirehead anymore, but simply converts matter into maximally efficient bliss simulations. I used to find this highly counterintuitive, but after thinking about all the absurd implications of valuing preferences instead of actual states of the world, I’ve come to think of it as a perfectly reasonable thing.