Another way to state the problem is that it will be too easy for human preferences to get hijacked by AIs to value ~arbitrary things, because it’s too easy to persuade humans of things, and a whole lot of economic analysis assumes that you cannot change a consumer’s preferences, probably because if you could do that, a lot of economic conclusions fall apart.
We also see evidence for the proposition that humans are easy to persuade based on a randomized controlled trial to reduce conspiracy theory beliefs:
Another way to state the problem is that it will be too easy for human preferences to get hijacked by AIs to value ~arbitrary things, because it’s too easy to persuade humans of things, and a whole lot of economic analysis assumes that you cannot change a consumer’s preferences, probably because if you could do that, a lot of economic conclusions fall apart.
We also see evidence for the proposition that humans are easy to persuade based on a randomized controlled trial to reduce conspiracy theory beliefs:
https://arxiv.org/abs/2403.14380