Systems that aim to optimize a goal /almost always/ instead optimize the pretense of the goal, followed by reproduction pressures, followed by the actual goal itself.
It’s related. Goodhart’s Law says that using a measure for policy will decouple it from any pre-existing relationship with economic activity, but doesn’t predict how that decoupling will occur. The common story of Goodhart’s law tells us how the Soviet Union measured factory output in pounds of machinery, and got heavier but less efficient machinery. Formalizing the patterns tells us more about how this would change if, say, there had not been very strict and severe punishments for falsifying machinery weight production reports.
Sometimes this is a good thing : it’s why, for one example, companies don’t instantly implode into profit-maximizers just because we look at stock values (or at least take years to do so). But it does mean that following a good statistic well tends to cause worse outcomes that following a poor statistic weakly.
That said, while I’m convinced that’s the pattern, it’s not the only one or even the most obvious one, and most people seem to have different formalizations, and I can’t find the evidence to demonstrate it.
Isn’t this basically Goodhart’s law?
It’s related. Goodhart’s Law says that using a measure for policy will decouple it from any pre-existing relationship with economic activity, but doesn’t predict how that decoupling will occur. The common story of Goodhart’s law tells us how the Soviet Union measured factory output in pounds of machinery, and got heavier but less efficient machinery. Formalizing the patterns tells us more about how this would change if, say, there had not been very strict and severe punishments for falsifying machinery weight production reports.
Sometimes this is a good thing : it’s why, for one example, companies don’t instantly implode into profit-maximizers just because we look at stock values (or at least take years to do so). But it does mean that following a good statistic well tends to cause worse outcomes that following a poor statistic weakly.
That said, while I’m convinced that’s the pattern, it’s not the only one or even the most obvious one, and most people seem to have different formalizations, and I can’t find the evidence to demonstrate it.