Business practice and management studies might provide some useful examples, and possibly some useful insights, here. E.g. off the top of my head:
Fairly obviously, how shareholders try to align managers with their goals, and managers align their subordinates with hopefully the same or close enough goals (and so on down the management chain to the juniors). Incompetent/scheming middle managers with their own agendas (and who may do things like unintentionally or deliberately alter information passed via them up & down the management chain) are a common problem. As are incorrectly incentivized CEOs (not least because their incentive package is typically devised by a committee of fellow board directors, of whom only some may be shareholders).
Less obviously, recruitment as an example of searching for optimizers: how do shareholders find managers who are best able to optimize shareholders’ interests, how do managers recruit subordinates, how do shareholders ensure managers are recruiting subordinates aligned with the shareholders’ goals rather than some agenda of their own, how are recruiters themselves incentivized and recruited, are there relevant differences between internal & external recruiters (e.g. HR vs headhunters), etc.
There are a set of strategies for mitigating the problems, and I have a paper on this that is written but still needs to be submitted somewhere, tentatively titled “Building Less Flawed Metrics: Dodging Goodhart and Campbell’s Laws,” if anyone wants to see it they can message/email/tweet at me.
Abstract: Metrics are useful for measuring systems and motivating behaviors. Unfortunately, naive application of metrics to a system can distort the system in ways that undermine the original goal. The problem was noted independently by Campbell and Goodhart, and in some forms it is not only common, but unavoidable due to the nature of metrics. There are two distinct but interrelated problems that must be overcome in building better metrics; first, specifying metrics more closely related to the true goals, and second, preventing the recipients from gaming the difference between the reward system and the true goal. This paper describes several approaches to designing metrics, beginning with design considerations and processes, then discussing specific strategies including secrecy, randomization, diversification, and post-hoc specification. Finally, it will discuss important desiderata and the trade-offs involved in each approach.
(I should edit this comment to be a link once I have submitted and have a pre-print or publication.)
I didn’t think it likely that business has solved any of these problems, but I just wonder if it might include particular failures not found in other fields that help point towards a solution; or even if it has found one or two pieces of a solution.
Business may at least provide some instructive examples of how things go wrong. Because business involves lots of people & motives & cooperation & conflict, things which I suspect humans are particularly good at thinking & reasoning about (as opposed to more abstract things).
Business practice and management studies might provide some useful examples, and possibly some useful insights, here. E.g. off the top of my head:
Fairly obviously, how shareholders try to align managers with their goals, and managers align their subordinates with hopefully the same or close enough goals (and so on down the management chain to the juniors). Incompetent/scheming middle managers with their own agendas (and who may do things like unintentionally or deliberately alter information passed via them up & down the management chain) are a common problem. As are incorrectly incentivized CEOs (not least because their incentive package is typically devised by a committee of fellow board directors, of whom only some may be shareholders).
Less obviously, recruitment as an example of searching for optimizers: how do shareholders find managers who are best able to optimize shareholders’ interests, how do managers recruit subordinates, how do shareholders ensure managers are recruiting subordinates aligned with the shareholders’ goals rather than some agenda of their own, how are recruiters themselves incentivized and recruited, are there relevant differences between internal & external recruiters (e.g. HR vs headhunters), etc.
There is a literature on this, and it’s not great for the purposes here—principle-agent setups assume we can formalize the goal as a good metric, and the complexity of management is a fundamentally hard problem that we don’t have good answers for (see my essay on scaling companies here: https://www.ribbonfarm.com/2016/03/17/go-corporate-or-go-home/ ), and goodhart failures due to under-specified goals are fundamentally impossible (see my essay on this here: https://www.ribbonfarm.com/2016/09/29/soft-bias-of-underspecified-goals/ ).
There are a set of strategies for mitigating the problems, and I have a paper on this that is written but still needs to be submitted somewhere, tentatively titled “Building Less Flawed Metrics: Dodging Goodhart and Campbell’s Laws,” if anyone wants to see it they can message/email/tweet at me.
Abstract: Metrics are useful for measuring systems and motivating behaviors. Unfortunately, naive application of metrics to a system can distort the system in ways that undermine the original goal. The problem was noted independently by Campbell and Goodhart, and in some forms it is not only common, but unavoidable due to the nature of metrics. There are two distinct but interrelated problems that must be overcome in building better metrics; first, specifying metrics more closely related to the true goals, and second, preventing the recipients from gaming the difference between the reward system and the true goal. This paper describes several approaches to designing metrics, beginning with design considerations and processes, then discussing specific strategies including secrecy, randomization, diversification, and post-hoc specification. Finally, it will discuss important desiderata and the trade-offs involved in each approach.
(I should edit this comment to be a link once I have submitted and have a pre-print or publication.)
Thanks, I’ll read those with interest.
I didn’t think it likely that business has solved any of these problems, but I just wonder if it might include particular failures not found in other fields that help point towards a solution; or even if it has found one or two pieces of a solution.
Business may at least provide some instructive examples of how things go wrong. Because business involves lots of people & motives & cooperation & conflict, things which I suspect humans are particularly good at thinking & reasoning about (as opposed to more abstract things).