Stuart_Armstrong comments on Large language models can provide “normative assumptions” for learning human preferences

[Stuart_Armstrong]

If the system that’s optimising is separate from the system that has the linguistic output, then there’s a huge issue with the optimising system manipulating or fooling the linguistic system—another kind of “symbol grounding failure”.