I have had two friends thrown out of the community without due process, and it was absolutely awful for them. For one of them, I was never told enough about the situation to make my own judgment, but the other one was thrown out basically just because a couple powerful people didn’t like his vibes.
I’ve also learned: It is easier to remove good people from your community than to remove people who are aggressive, manipulative, narcissistic, or psychopathic, because the former will remain reasonable throughout the process, while the latter will fight dirty.
This is why it’s so important to never, ever, EVER give people quotas for purging, or rewarding people in any way for racking large numbers of purge cases, no matter how good they make each purge case look.
Goodharting will typically take place if you stack the incentives like that.
The thing you quoted doesn’t imply that there were any quotas or rewards, or metrics being Goodharted. (Definitely agree that quotas or rewards for “purging” is a terrible idea.)
I have the impression that it is a general problem with public prosecutors (source: cultural osmosis). So if one were to imagine creating someone whose job is analogous to prosecuting criminals, one would have to be very careful about the incentives one sets up for that person, and it’s not obvious that rationalists could walk in and immediately solve it better than society has managed to do with prosecutors.
This is why it’s so important to never, ever, EVER give people quotas for purging, or rewarding people in any way for racking large numbers of purge cases, no matter how good they make each purge case look.
Goodharting will typically take place if you stack the incentives like that.
The thing you quoted doesn’t imply that there were any quotas or rewards, or metrics being Goodharted. (Definitely agree that quotas or rewards for “purging” is a terrible idea.)
I have the impression that it is a general problem with public prosecutors (source: cultural osmosis). So if one were to imagine creating someone whose job is analogous to prosecuting criminals, one would have to be very careful about the incentives one sets up for that person, and it’s not obvious that rationalists could walk in and immediately solve it better than society has managed to do with prosecutors.