Terence Tao has a comment on this paper on G+ that I quite liked:
Goodhart’s law can be formulated as “When a measure becomes a target, it ceases to be a good measure.” It initially arose in economics and is mostly applied to situations involving human agents, but as the article below illustrates with several anecdotes, the same law applies in AI research. My favorite is the AI that learned to win at a generalised form of tic-tac-toe by sending their moves to the AI opponent in a highly convoluted fashion that caused them to crash due to exceeding memory limitations.
In mathematics, the analogous phenomenon is that the argmax (or argmin) function—that takes a function F ranging over some parameter space and locates its maximum (or minimum) - can be very unstable. An approximation G to F that agrees well with F for “typical” cases may have a vastly different location for its global maximum (or minimum), due to “edge” case discrepancies. More generally, it can be dangerous to extrapolate average case behaviour of a function to draw any conclusions about worst case (or best case) behaviour.
Terence Tao has a comment on this paper on G+ that I quite liked:
I believe Terence is describing extremal goodhart in his second paragraph.