My expectation is that error and utility are both extremely heavy tailed, and arguably in the same order of magnitude for heavy tails.
But thanks for answering, the real answer is we can predict effectively nothing without independence, and thus we can justify virtually every outcome of real-life Goodhart.
Maybe it’s catastrophic, maybe it doesn’t matter, or maybe there’s anti-goodhart, but I don’t see a way to predict what will reasonably happen.
Also, why do you think that error is heavier tailed than utility?
Also, why do you think that error is heavier tailed than utility?
Goodhart’s Law is really common in the real world, and most things only work because we can observe our metrics, see when they stop correlating with what we care about, and iteratively improve them. Also the prevalence of reward hacking in RL often getting very high values.
If the reward model is as smart as the policy and is continually updated with data, maybe we’re in a different regime where errors are smaller than utility.
My expectation is that error and utility are both extremely heavy tailed, and arguably in the same order of magnitude for heavy tails.
But thanks for answering, the real answer is we can predict effectively nothing without independence, and thus we can justify virtually every outcome of real-life Goodhart.
Maybe it’s catastrophic, maybe it doesn’t matter, or maybe there’s anti-goodhart, but I don’t see a way to predict what will reasonably happen.
Also, why do you think that error is heavier tailed than utility?
Goodhart’s Law is really common in the real world, and most things only work because we can observe our metrics, see when they stop correlating with what we care about, and iteratively improve them. Also the prevalence of reward hacking in RL often getting very high values.
If the reward model is as smart as the policy and is continually updated with data, maybe we’re in a different regime where errors are smaller than utility.