I think a fair way to steelman that objection is interpreting it as meaning that doing what you describe isn’t just subtle and super difficult, it’s theoretically impossible, due to inherent contradictions in what we think we want, what different people want, inaccessibility of our true subjective experiences, etc.
That said, obviously, there’s a lot of leeway before getting to those limits. The world with no cancer sounds a lot more awesome than the world with it, and if getting there requires depriving a handful of researchers of the satisfaction of knowing they solved the problem and the consequent Nobel Prize, well, I’m sure we can manage.
I think a fair way to steelman that objection is interpreting it as meaning that doing what you describe isn’t just subtle and super difficult, it’s theoretically impossible, due to inherent contradictions in what we think we want, what different people want, inaccessibility of our true subjective experiences, etc.
That said, obviously, there’s a lot of leeway before getting to those limits. The world with no cancer sounds a lot more awesome than the world with it, and if getting there requires depriving a handful of researchers of the satisfaction of knowing they solved the problem and the consequent Nobel Prize, well, I’m sure we can manage.