The thing you quoted doesn’t imply that there were any quotas or rewards, or metrics being Goodharted. (Definitely agree that quotas or rewards for “purging” is a terrible idea.)
I have the impression that it is a general problem with public prosecutors (source: cultural osmosis). So if one were to imagine creating someone whose job is analogous to prosecuting criminals, one would have to be very careful about the incentives one sets up for that person, and it’s not obvious that rationalists could walk in and immediately solve it better than society has managed to do with prosecutors.
The thing you quoted doesn’t imply that there were any quotas or rewards, or metrics being Goodharted. (Definitely agree that quotas or rewards for “purging” is a terrible idea.)
I have the impression that it is a general problem with public prosecutors (source: cultural osmosis). So if one were to imagine creating someone whose job is analogous to prosecuting criminals, one would have to be very careful about the incentives one sets up for that person, and it’s not obvious that rationalists could walk in and immediately solve it better than society has managed to do with prosecutors.