Mateusz Bagiński comments on AISC team report: Soft-optimization, Bayes and Goodhart

[Mateusz Bagiński]

People have already done a fair bit of work on this in RL in terms of ‘cautious’ RL which tries to take into account uncertainty in the world model to avoid accidentally falling into traps in the environment.

I would appreciate some pointers to resources