Thank you for the answer! I thought the Goodheart effect was about metrics turning bad when they become the thing you are optimising for. Does the term apply to something else here?
The term “murder” refering to something immoral by definition makes sense, I should have thought about this more explicitely.
Your advice in the last paragraph resonates with me. My experience has also been that people often seem to prioritize signaling. I am not a big fan of Jordan Peterson, but this reminded me of a quote of his: “If you can’t understand why someone is doing something, look at the consequences of their actions, whatever they might be, and then infer the motivations from their consequences.”
Goodhart effect is a truly deep insight IMO. Goodhart itself states that lots of great metrics and “signals”, and in general, communicatory material, will lose its value when adopted by social systems. This leads to second order effects, such as people and society defending against this effect. This can be seen in speech easily: if your friend did not depend on your speech for various purposes, your speech would be a truly great way to learn about your “true” self. But now that he does depend on your speech, he has to guard against you lying and optimizing your speech to manipulate him and shift risks unto him. So he cannot accept your noncommittal stance on, e.g., “murder,” and requires you to commit your position. I have heard there are studies (beware replication crisis and yadada) that women prefer men with virtue ethics over utilitarians; One reason might be that virtue ethics constrains people much more effectively than “consequentialism,” where you’re your own auditor.
Thank you for the answer! I thought the Goodheart effect was about metrics turning bad when they become the thing you are optimising for. Does the term apply to something else here?
The term “murder” refering to something immoral by definition makes sense, I should have thought about this more explicitely.
Your advice in the last paragraph resonates with me. My experience has also been that people often seem to prioritize signaling. I am not a big fan of Jordan Peterson, but this reminded me of a quote of his: “If you can’t understand why someone is doing something, look at the consequences of their actions, whatever they might be, and then infer the motivations from their consequences.”
Goodhart effect is a truly deep insight IMO. Goodhart itself states that lots of great metrics and “signals”, and in general, communicatory material, will lose its value when adopted by social systems. This leads to second order effects, such as people and society defending against this effect. This can be seen in speech easily: if your friend did not depend on your speech for various purposes, your speech would be a truly great way to learn about your “true” self. But now that he does depend on your speech, he has to guard against you lying and optimizing your speech to manipulate him and shift risks unto him. So he cannot accept your noncommittal stance on, e.g., “murder,” and requires you to commit your position. I have heard there are studies (beware replication crisis and yadada) that women prefer men with virtue ethics over utilitarians; One reason might be that virtue ethics constrains people much more effectively than “consequentialism,” where you’re your own auditor.