I believe Steven didn’t imply that a significant number of people would approve or want such a future—indeed, the opposite, hence he called the scenario “dystopian”.
He basically meant that optimising surface signals of pleasure does not automatically lead to behaviours and plans congruent with reasonable ethics, so the surface elements of alignment suggested by LeCun in the paper are clearly insufficient.
I think many EAs/rationalists shouldn’t find this to be worse for humans than life today on the views they apparently endorse, because each human looks better off under standard approaches to intrapersonal aggregation: they get more pleasure, less suffering, more preference satisfaction (or we can imagine some kind of manipulation to achieve this), but at the cost of some important frustrated preferences.
EA/rationalists seems to me the community who gives more conscious attention to the problem of following your officially stated preferences down a cliff, a.k.a. Goodhart.
FWIW, I’m not sure if that’s true relative to the average person, and I’d guess non-consequentialist philosophers are more averse to biting bullets than the average EA and maybe rationalist.
I believe Steven didn’t imply that a significant number of people would approve or want such a future—indeed, the opposite, hence he called the scenario “dystopian”.
He basically meant that optimising surface signals of pleasure does not automatically lead to behaviours and plans congruent with reasonable ethics, so the surface elements of alignment suggested by LeCun in the paper are clearly insufficient.
I think many EAs/rationalists shouldn’t find this to be worse for humans than life today on the views they apparently endorse, because each human looks better off under standard approaches to intrapersonal aggregation: they get more pleasure, less suffering, more preference satisfaction (or we can imagine some kind of manipulation to achieve this), but at the cost of some important frustrated preferences.
EA/rationalists seems to me the community who gives more conscious attention to the problem of following your officially stated preferences down a cliff, a.k.a. Goodhart.
FWIW, I’m not sure if that’s true relative to the average person, and I’d guess non-consequentialist philosophers are more averse to biting bullets than the average EA and maybe rationalist.
I suspect that you read “conscious attention to the problem” in a different light than what I mean. To clarify:
Average person won’t go down a cliff on literal instructions, unless Moloch? Yes.
Average person will identify and understand such problem? No.
EA bites a bullet and does something weird? Yes.
EA bites a bullet because YAY BULLETS COOL? No.