I didn’t notice until just recently that this post fits into a similar genre as (what I think) the Moral Mazes discussion is pointing at (which may be different from what Zvi thinks).
Where one of the takeaways from Moral Mazes might be: “if you want your company to stay aligned, try not to grow the levels of hierarchy too much, or be extremely careful when you do.”
“Don’t grow the layers of hierarchy” is (in practice) perhaps a similar injunction to “don’t grow the company too much at all” (since you need hierarchy to scale)
Immoral Mazes posits a specific failure due to middle managers being disconnected from reality, and evolving an internal ecosystem that then sets out to protect itself. This post points at an (upstream?) issue where, regardless of middle management, the fundamental reality is that people cannot rely on repeated-interactions to build trust.
I actually totally forgot until just now the final paragraph, and key point of this post
So if we want to e.g. reduce regulation, we should first focus on the underlying socioeconomic problem: fewer interactions. A world of Amazon and Walmart, where every consumer faces decisions between a million different products, is inevitably a world where consumers do not know producers very well. There’s just too many products and companies to keep track of the reputation of each. To reduce regulation, first focus on solving that problem, scalably. Think amazon reviews—it’s an imperfect system, but it’s far more flexible and efficient than formal regulation, and it scales.
Now for the real problem: online reviews are literally the only example I could come up with where technology offers a way to scale-up reputation-based systems, and maybe someday roll back centralized control structures or group identities. How can we solve these sorts of problems more generally? Please comment if you have ideas.
I think this maps to one of the key (according to me) problems raised by the Immoral Mazes sequence: we don’t know how to actually identify and reward competence among middle managers, so all we have are easily goodhartable metrics. (And in the case of middle management that there’s a deep warping that happens because the thing that got goodharted on was “office politics”)
Unfortunately… well, nobody commented with ideas on this post, and also I don’t know that anyone came up with any way to track competence of management either.
The actionable place where this matters in my local environment is EA grantmakers awarding researchers, which are often pretty hard to evaluate. I think this is a serious bottleneck to scaling efforts.
I notice that forecasting is one of few domains where rationalsphere-folk are experimenting with scalable solutions for evaluation. I’ve been somewhat pessimistic about forecasting, but I think this might have convinced me to allocate more attention to it.
...huh. I would not have expected this post to be closely associated in my head with amplifying generalist forecasting. But, now I think it is.
My guess is that part of what’s going on here is that in certain ways attempting to optimize for coordination is at greater risk for goodhart than other things. To take an example from the post to its limit, the freelancer who invests 100% in selling their services and 0% in being skilled at them or providing them is a fraud. But also the freelancer who invests 100% in skills but 0% in selling is out of business.
So there’s a need for some sort of dynamic balance.
But my guess is that for whatever reasons (documented in Moral Mazes no doubt) certain kinds of organizations put pressure on the managers to go all the way in one direction, rather than finding that balance.
I didn’t notice until just recently that this post fits into a similar genre as (what I think) the Moral Mazes discussion is pointing at (which may be different from what Zvi thinks).
Where one of the takeaways from Moral Mazes might be: “if you want your company to stay aligned, try not to grow the levels of hierarchy too much, or be extremely careful when you do.”
“Don’t grow the layers of hierarchy” is (in practice) perhaps a similar injunction to “don’t grow the company too much at all” (since you need hierarchy to scale)
Immoral Mazes posits a specific failure due to middle managers being disconnected from reality, and evolving an internal ecosystem that then sets out to protect itself. This post points at an (upstream?) issue where, regardless of middle management, the fundamental reality is that people cannot rely on repeated-interactions to build trust.
I actually totally forgot until just now the final paragraph, and key point of this post
I think this maps to one of the key (according to me) problems raised by the Immoral Mazes sequence: we don’t know how to actually identify and reward competence among middle managers, so all we have are easily goodhartable metrics. (And in the case of middle management that there’s a deep warping that happens because the thing that got goodharted on was “office politics”)
Unfortunately… well, nobody commented with ideas on this post, and also I don’t know that anyone came up with any way to track competence of management either.
The actionable place where this matters in my local environment is EA grantmakers awarding researchers, which are often pretty hard to evaluate. I think this is a serious bottleneck to scaling efforts.
I notice that forecasting is one of few domains where rationalsphere-folk are experimenting with scalable solutions for evaluation. I’ve been somewhat pessimistic about forecasting, but I think this might have convinced me to allocate more attention to it.
...huh. I would not have expected this post to be closely associated in my head with amplifying generalist forecasting. But, now I think it is.
My guess is that part of what’s going on here is that in certain ways attempting to optimize for coordination is at greater risk for goodhart than other things. To take an example from the post to its limit, the freelancer who invests 100% in selling their services and 0% in being skilled at them or providing them is a fraud. But also the freelancer who invests 100% in skills but 0% in selling is out of business.
So there’s a need for some sort of dynamic balance.
But my guess is that for whatever reasons (documented in Moral Mazes no doubt) certain kinds of organizations put pressure on the managers to go all the way in one direction, rather than finding that balance.