I suspect that many rationalists independently figure this issue out before learning of older (better?) names for it. In a discussion group in college we used the term “optimized signifier” for the core concept while talking about ways that groups could go off course in the pursuit of honestly valuable goals that were rather nebulous in their “authentic” form. Our standard examples of optimized signifiers included weight as a proxy for health, grades as a proxy for learning.
...which was first discussed by name in The Importance of Goodhart’s Law.
The issue has also come up in various other posts listed here in order of their current karma:
77 - Self-fulfilling correlations
32 - Is Google Paperclipping the Web? The Perils of Optimization by Proxy in Social Systems
Followup: 11 Beyond Optimization by Proxy
29 - The Shabbos goy
3 - Imperfect Levers
There was a forward link in the comments from Eliezer’s Resist the Happy Death Spiral to Goodhart’s Law as well.
I suspect that many rationalists independently figure this issue out before learning of older (better?) names for it. In a discussion group in college we used the term “optimized signifier” for the core concept while talking about ways that groups could go off course in the pursuit of honestly valuable goals that were rather nebulous in their “authentic” form. Our standard examples of optimized signifiers included weight as a proxy for health, grades as a proxy for learning.