I wish I had a better model of how common it is to actually have people destroying large amounts of value on purpose, for reasons such as those in the OP. And if it is common, I wish I had a clearer model of why. (I suspect it’s common. I’ve suspected this since reading Atlas Shrugged ~a year ago. But I’m not sure, and I don’t have a good mechanistic mode, and in my ignorance of the ‘true names’ of this stuff it seems really easily to blatantly misunderstand.)
To try to pinpoint what I don’t understand:
I agree that we care about each others’ motives. And that we infer these from actions. And that we care about others’ models of our motives, and that this creates funny second-order incentives for our actions.
I also agree that there are scenarios, such as those in the OP, where these second-order incentives can lead a person to destroy a bunch of value (by failing to not-poison the river, etc.)
I’m uncertain of the frequency and distribution of such incentive-impacts. Are these second-order incentives mostly toward “actions with worse physical consequences”, or are they neutral or positive in expectation (while still negative in some instances)? (I agree there are straight-forward examples where they’re toward worse, and that Zvi lists some of these. But there are also examples the other way. Like, in Zvi’s widget-factory example, I could imagine the middle manager choosing the policy that will avoid poisoning the water (whether or not he directly cares about it) so that other people will think he is the sort of person who cares about good things, and will be more likely to ally with him in contexts where you want someone who cares about good things (marriage; friendships; some jobs).)
If the distribution does have a large amount of cases second-order incentives push toward destroying value — why, exactly?
Differently put: in the OP, Zvi writes “none of this assumes a zero-sum mentality. At all.” But, if we aren’t assuming a zero-sum mentality, why would the middle manager’s boss (in the widgets example) want to make sure he doesn’t care about the environment? Like, one possibility is that the boss thinks things are zero-sum, and expects a tradeoff between “is this guy liable to worry about random non-company stuff in future situations” and “is this guy doing what’ll help the company” in future cases. But that seems like an example of the boss expecting zero-sum situations, or at least of expecting tradeoffs. And Zvi is saying that this isn’t the thing.
(And one possibility for why such dynamics would be common, if they are, is if it is common to have zero-sum situations where “putting effort toward increasing X” would interfere with a person’s ability to increase Y. But I think this isn’t quite what Zvi is positing.)
Here is a different model (besides the zero-sum effort tradeoffs model) of why value-losses such as those in the OP might be common and large. The different model is something like “compartmentalization has large upsides for coordination/predictability/simplicity, and is also easier than most other ways of getting control/predictability”. Or in more detail: having components of a (person/organization/anything) that act on anything unexpected means having components of a (person/organization/anything) that are harder to control, which decreases the (person/organization/etc.)’s ability to pull off maneuvers that require predictability, and is costly. (I think this might be Zvi’s model from not-this-post, but I’m not sure, and I’d like to elaborate it in my own words regardless.)
Under this model, real value is actually created via enforcing this kind of predictability (at least, if the organization is being used to make value at all), although at real cost.
Examples/analogies that (correctly or not) are parts of why I find this “compartmentalization/simplicity has major upsides” model plausible:
A. I read/heard somewhere that most of our muscles are used to selectively inhibit other muscles, so as to be able to do fine motor coordination. And that this is one of the differences between humans and chimps, where humans have piles of muscles inhibiting each other to allow fine motor skill, and chimps went more for uncoordinated strength. (Can someone confirm whether this is true?) (The connection may be opaque here. But it seems to me that most of our psychologies are a bit like this — we could’ve had simple, strongly felt, drives and creative impulses, but civilized humans are instead bunches of macros that selectively track and inhibit other macros; and this seems to me to have been becoming more and more true across the last few thousand years in the West.)
B. If I imagine hiring someone for CFAR who has a history of activism along the lines of [redacted, sorry I’m a coward but at least it’s better than omitting the example entirely], I feel pause, not because of “what if the new staff member puts some of their effort into that instead of about CFAR’s goals” but because of “what it makes it more difficult and higher-overhead to coordinate within CFAR, and leaves us with a bunch of, um, what shows up on my internal radar as ‘messes we have to navigate’ all the time, where I have to somehow trick them into going along with the program, and the overhead of this makes it harder to think and talk and get things done together.” (To be clear, parts of this seem bad to me, and this isn’t how I would try to strategize toward me and CFAR doing things; in particular it seems to highlight some flaw in my current ontology to parse divergent opinions as ‘messes I have to navigate, to trick them into going along with the program’. I, um, do not want you to think I am endorsing this and to get to blame or attack me for it, but I do want to get to talk about it.)
C. I think a surgeon would typically be advised not to try to operate on their own child, because it is somehow harder to have steady hands and mind (highly predictable-to-oneself and coordinated behavior) if a strong desire/fear is activated (even one as aligned with “do good surgery on my child” as the desire/fear for one’s child’s life). (Is this true? I haven’t fact-checked it. I have heard poker players say that it’s harder to play well for high stakes. Also the book “The inner game of tennis” claims that wanting to win at tennis impairs most adults’ ability to learn tennis.)
D. In the OP’s “don’t ask what the wine costs, it would ruin the evening” example: it seems to me that there really is a dynamic where asking what the wine costs can at least mildly harm my own experience of the evening, and that for me (and I imagine quite a few others), the harm is not that asking the wine’s price reveals a stable, persistent fact that the asker cares about money. Rather, the harm is asking it breaks the compartmentalization that was part of how I knew how to be “in flow” for the evening. Like, after the asking, I’m thinking about money, or thinking about others thinking about money, and I’m somehow less good at candlelight and music and being with my and others’ experiences when that is happening. (This is why Zvi describes it as “slightly socially awkward” — awkwardness is what it feels like when a flow is disrupted.) (We can tell that the thing that’s up here in my experience of the evening isn’t about longer-term money-indicators, partly because I have no aversion to hearing the same people talk about caring about money in most other contexts.) (I’m sure straight money-signaling, as in Zvi’s interpretation, also happens with some people about the wine. But the different “compartmentalization is better for romantic evenings” dynamic I’m describing can happen too.)
E. This is the example I care most about, and am least likely to do justice to. Um: there’s a lot of pain/caring that I find myself dissociating from, most of the time. (Though I can only see it in flashes.) For example, it’s hard for me to think about death. Or about AI risk, probably because of the “death” part. Or about how much I love people. Or how I hope I have a good life, and how much I want children. I can think words about these things, but I tend to control my breathing while doing so, to become analytic, to look at things a bit from a distance, to sort of emulate the thoughts rather than have them.
It seems to me my dissociating here is driven less by raw pain/caring being unpleasant (although sometimes it is), and more by the fact that when I am experiencing raw pain/caring it is harder to predict/plan/control my own behavior, and that lack of predictability is at least somewhat scary and risky. Plus it is somehow tricky for other people to be around, such that I would usually feel impolite doing it and avoid visibly caring in certain ways for that reason. (See the example F.)
F. [Kind of like E, but as an interpersonal dynamic] When other people show raw caring, it’s hard for me to avoid dissociating. Especially if it’s to do with something where… the feeling inside my head is something like “I want this, I am this, but I can’t have this. It isn’t mine. Fear. Maybe I’m [inadequate/embarrassing/unable forever]?” Example: a couple days ago, some friends and I watched “It’s a Wonderful Life”, which I hadn’t seen before. And afterward a friend and I were raw and talking, and my friend was, I can’t remember, but talking about wanting to be warm and human or something. And it was really hard for me not to just dissociate — I kept having all kinds of nonsense arguments pop into my head for why I should think about my laundry, why I get analytic-and-in-control-of-the-conversation, why I should interrupt him. And later on, my friend was “triggered” about a different thing, and I noticed it was the same [fear/blankness/tendency-to-want-to-dissociate] in me, in response to those other active currents. And I commented on it to my friend, and we noticed that the thing I was instinctively doing in response to that fear in me, was kind of sending my friend “this is weird/bad what you’re doing” signals. So. Um. Maybe there’s a thing where, once people start keeping raw pain/caring/love/anything at distance, if they run into other people who aren’t, they send those people “you’re being crazy/bad” signals whenever those other people aren’t keeping their own raw at a distance. And so we socialize each other to dissociate.
(This connects still to the compartmentalization-as-an-aid-to-predictability thesis, because part of the trouble with e.g. somebody else talking about death, or being raw, or triggered, is that it makes it harder for me to dissociate, and so makes me less predicable/controllable to me.)
G. This brings me to an alternate possible mechanics of Zvi’s “carefully not going out of one’s way not to poison the river with the widget factory” example. If lots of people at WidgetCorp wanted to contribute to (the environment / good things broadly), but are dissociated from their desire, it might mess with their dissociation (and, thus, their control and predictability-to-themselves of their own behavior, and plus WidgetCorp’s ability to predict and control them) if anybody else visibly cares about the river (or even, visibly does a thing one could mistake as caring about the river). And so we get the pressure that Zvi mentions, here and in his “moral mazes” sequence. (And we can analogously derive a pressure not to be a “goody two-shoes” among kids who kind of want to be good still, but also kind of want to be free from that wanting. And the pressure not to be too vulnerably sincere in one’s romantic/sexual encounters, and to instead aspire to cynicism. And more generally (in the extreme, at least) to get attack anyone who acts from intact caring. Sort of like an anti-epistemology, but more exactly like an anti-caring.
Extending the E-F-G thing: perhaps we could say “every cause/movement/organization wants to become a pile of defanged pica and ostentatious normalcy (think: Rowling’s Dursleys) that won’t be disruptive to anyone”, as an complimentary/slightly-contrasting description to “every cause wants to be a cult”.
In the extreme, this “removing of all impulses that’ll interfere with predictability-and-control” is clearly not useful for anything. But in medium-sized amounts, I think predictability/controllability-via-compartmentalization can actually help with creating physical goods, as with the surgeon or poker player or tennis player who has an easier time when they are not in touch with an intense desire for a particular outcome. And I think we see it sometimes in large amounts — large enough that they are net-detrimental to the original goal of the person/cause/business/etc.
Maybe it’s something like:
Being able to predict and control one’s own actions, or one’s organization’s actions, is in fact useful. You can use this to e.g. take three coordinated actions in sequence that will collectively but not individually move you toward a desired outcome, such as putting on your shoes in order to walk to the store in order to be able to buy pasta in order to be able to cook it for dinner. (I do not think one can do this kind of multi-step action nearly as well without prediction-and-control of one’s behavior.)
Because it is useful, we build apparatuses that support it. (“Egos” within individual humans; structures of management and deferral and conformity within organizations and businesses and social movements.)
Even though prediction-and-control is genuinely useful, a central planning entity doing prediction-and-control will tend to overestimate the usefulness of its having more prediction-and-control, and to underestimate the usefulness of aspects of behavior that it does not control. This is because it can see what it’s trying to do, and can’t see what other people are trying to do. Also, its actions are specifically those that its own map says will help, and others’ actions are those which their own maps say will help, which will bring in winner’s curse-type dynamics. So central planning will tend to over-invest in increasing its own control, and to under-invest in allowing unpredictability/disruption/alternate pulls on behavior.
… ? [I think the above three bullet points are probably a real thing that happens. But it doesn’t seem to take my imagination all the way to full-on moral mazes (for organizations), or to individuals who are full-on trying to prop up their ego at the expense of everything. Maybe it does and I’m underestimating it? Or maybe there are added steps after my third bullet point of some sort?]
[Epistemic status: I’m not confident of any of this; I just want better models and am trying to articulate mine in case that helps. Also, all of my comments on this post are as much a response to the book “Moral Mazes” as to the OP.]
Let’s say that A is good, and that B is also good. (E.g. equality and freedom, or diversity and families, or current lives saved and rationality, or any of a huge number of things.) Let’s consider how the desire-for-A and the desire-for-B might avoid having their plans/goal-achievement disrupted by one another.
In principle, you could build a larger model that explains how to trade off between A and B — a model that subsumes A and B as special cases of a more general good. And then the A-desire and the B-desire could peacefully co-exist and share influence within this larger structure, without disrupting each others’ ability to predict-and-control, or to achieve their goals. (And thereby, they could both stably remain part of your psyche. Or part of your organization. Or part of your subcultural movement. Or part of your overarching civilization’s sense of moral decency. Or whatever. Without one part of your civilization’s sense of moral decency (or etc.) straining to pitch another part of that same sense of moral decency overboard.)
Building a larger model subsuming both the A-is-good and B-is-good models is hard, though. It requires a bunch of knowledge/wisdom/culture to kind a workable model of that sort. Especially if you want everybody to coordinate within the same larger model (so that the predict-and-control thing can keep working). A simpler thing you could attempt instead is to just ban desire B. Then desire-for-B won’t get in the way of your attempt to coordinate around achieving desire A. (Or, in more degenerate cases, it won’t get in the way of your attempt to coordinate around you-the-coordinator staying coordinating, with all specific goals mostly forgotten about.) This “just abolish desire B” thing is much simpler to design. So this simpler strategy (“disown and dissociate from one of the good things”) can be reinvented even in ignorance, and can also be shared/evangelized for pretty easily, without needing to share a whole culture.
Separately: once upon a time, there used to be a shared deep culture that gave all humans in a given tribe a whole bunch of shared assumptions about how everything fit together. In that context, it was easier to create/remember/invoke common scaffolds allowing A-desire and B-desire to work together without disrupting each others’ ability to do predictability-and-control. You did not have to build such scaffolds from scratch.
Printing presses and cities and travel/commerce/conversation between many different tribes, and individuals acquiring more tools for creating new thoughts/patterns/associations, and… social media… later made different people assume different things, or fewer things. It became extra-hard to create shared templates in which A-desire and B-desire can coordinate. And so we more often saw social movements / culture wars in which the teams (which each have some memory of some fragment of what’s good) are bent on destroying one another, lest one another destroy their ability to do prediction-and-control in preservation of their own fragment of what’s good. “Humpty Dumpty sat on a wall…”
(Because the ability to do the simpler “dissociate from desire B, ban desire B” move does not break down as quickly, with increasing cultural diversity/fragmentation, as the ability to do the more difficult “assimilate A and B into a common larger good” move.)
Also: it seems to me that “G” might be the generator of the thing Zvi calls as “Moloch’s Army.” Zvi writes:
Moloch’s Army
…. I still can’t find a way into this without sounding crazy. The result of this is that the sequence talks about maze behaviors and mazes as if their creation and operation are motivated by self-interest. That’s far from the whole picture.
There is mindset that instinctively and unselfishly opposes everything of value. This mindset is not only not doing calculations to see what it would prefer or might accomplish. It does not even believe in the concept of calculation (or numbers, or logic, or reason) at all. It cares about virtues and emotional resonances, not consequences. To do this is to have the maze nature. This mindset instinctively promotes others that share the mindset, and is much more common and impactful among the powerful than one would think. Among other things, the actions of those with this mindset are vital to the creation, support and strengthening mazes.
Until a proper description of that is finished, my job is not done. So far, it continues to elude me. I am not giving up.
For whatever it’s worth, I am also inclined to think that something like “Moloch’s Army” describes something important in the world. As sort-of-mentioned, Atlas Shrugged more or less convinced me of this by highlighting a bunch of psychological dynamics that, once highlighted, I seemed to see in myself and others. But I am still confused about it (whether it’s real; what it’s made of insofar as there is a real thing like that). And G is my best current attempt to derive it.
Here is my take: Value is a function of the entire state space, and can’t be neatly decomposed as a sum of subgames. Rather (dually), value on (“quotient”) subgames must be confluent with the total value on the joint game. Eg, there’s an “enjoying the restaurant food” game, and a “making your spouse happy” game, but the joint game of “enjoying a restaurant with your spouse” game has more moves available, and more value terms that don’t show up in either game, like “be a committed couple”. ”Confluence” here means that what you need to forget to zoom in on the “enjoying the restaurant food” subgame causes your value judgement of “enjoying the restaurant food” and “enjoying a restaurant with your spouse, ignoring everything except food” to agree. The individual subgames aren’t “closed”, they were never closed, their value only makes sense in a larger context, because the primitives used to define that value refer to the larger context. From the perspective of the larger game, no value is “destroyed”, it only appears that way when projecting into the subgames, which were only ever virtual.
This is just saying the coordination that results from the destruction of value is more valuable than the value destroyed, externalities disregarded. The post is about finding cheaper coordination strategies and internalizing more of the externalities.
I wish I had a better model of how common it is to actually have people destroying large amounts of value on purpose, for reasons such as those in the OP. And if it is common, I wish I had a clearer model of why. (I suspect it’s common. I’ve suspected this since reading Atlas Shrugged ~a year ago. But I’m not sure, and I don’t have a good mechanistic mode, and in my ignorance of the ‘true names’ of this stuff it seems really easily to blatantly misunderstand.)
To try to pinpoint what I don’t understand:
I agree that we care about each others’ motives. And that we infer these from actions. And that we care about others’ models of our motives, and that this creates funny second-order incentives for our actions.
I also agree that there are scenarios, such as those in the OP, where these second-order incentives can lead a person to destroy a bunch of value (by failing to not-poison the river, etc.)
I’m uncertain of the frequency and distribution of such incentive-impacts. Are these second-order incentives mostly toward “actions with worse physical consequences”, or are they neutral or positive in expectation (while still negative in some instances)? (I agree there are straight-forward examples where they’re toward worse, and that Zvi lists some of these. But there are also examples the other way. Like, in Zvi’s widget-factory example, I could imagine the middle manager choosing the policy that will avoid poisoning the water (whether or not he directly cares about it) so that other people will think he is the sort of person who cares about good things, and will be more likely to ally with him in contexts where you want someone who cares about good things (marriage; friendships; some jobs).)
If the distribution does have a large amount of cases second-order incentives push toward destroying value — why, exactly?
Differently put: in the OP, Zvi writes “none of this assumes a zero-sum mentality. At all.” But, if we aren’t assuming a zero-sum mentality, why would the middle manager’s boss (in the widgets example) want to make sure he doesn’t care about the environment? Like, one possibility is that the boss thinks things are zero-sum, and expects a tradeoff between “is this guy liable to worry about random non-company stuff in future situations” and “is this guy doing what’ll help the company” in future cases. But that seems like an example of the boss expecting zero-sum situations, or at least of expecting tradeoffs. And Zvi is saying that this isn’t the thing.
(And one possibility for why such dynamics would be common, if they are, is if it is common to have zero-sum situations where “putting effort toward increasing X” would interfere with a person’s ability to increase Y. But I think this isn’t quite what Zvi is positing.)
Here is a different model (besides the zero-sum effort tradeoffs model) of why value-losses such as those in the OP might be common and large. The different model is something like “compartmentalization has large upsides for coordination/predictability/simplicity, and is also easier than most other ways of getting control/predictability”. Or in more detail: having components of a (person/organization/anything) that act on anything unexpected means having components of a (person/organization/anything) that are harder to control, which decreases the (person/organization/etc.)’s ability to pull off maneuvers that require predictability, and is costly. (I think this might be Zvi’s model from not-this-post, but I’m not sure, and I’d like to elaborate it in my own words regardless.)
Under this model, real value is actually created via enforcing this kind of predictability (at least, if the organization is being used to make value at all), although at real cost.
Examples/analogies that (correctly or not) are parts of why I find this “compartmentalization/simplicity has major upsides” model plausible:
A. I read/heard somewhere that most of our muscles are used to selectively inhibit other muscles, so as to be able to do fine motor coordination. And that this is one of the differences between humans and chimps, where humans have piles of muscles inhibiting each other to allow fine motor skill, and chimps went more for uncoordinated strength. (Can someone confirm whether this is true?) (The connection may be opaque here. But it seems to me that most of our psychologies are a bit like this — we could’ve had simple, strongly felt, drives and creative impulses, but civilized humans are instead bunches of macros that selectively track and inhibit other macros; and this seems to me to have been becoming more and more true across the last few thousand years in the West.)
B. If I imagine hiring someone for CFAR who has a history of activism along the lines of [redacted, sorry I’m a coward but at least it’s better than omitting the example entirely], I feel pause, not because of “what if the new staff member puts some of their effort into that instead of about CFAR’s goals” but because of “what it makes it more difficult and higher-overhead to coordinate within CFAR, and leaves us with a bunch of, um, what shows up on my internal radar as ‘messes we have to navigate’ all the time, where I have to somehow trick them into going along with the program, and the overhead of this makes it harder to think and talk and get things done together.” (To be clear, parts of this seem bad to me, and this isn’t how I would try to strategize toward me and CFAR doing things; in particular it seems to highlight some flaw in my current ontology to parse divergent opinions as ‘messes I have to navigate, to trick them into going along with the program’. I, um, do not want you to think I am endorsing this and to get to blame or attack me for it, but I do want to get to talk about it.)
C. I think a surgeon would typically be advised not to try to operate on their own child, because it is somehow harder to have steady hands and mind (highly predictable-to-oneself and coordinated behavior) if a strong desire/fear is activated (even one as aligned with “do good surgery on my child” as the desire/fear for one’s child’s life). (Is this true? I haven’t fact-checked it. I have heard poker players say that it’s harder to play well for high stakes. Also the book “The inner game of tennis” claims that wanting to win at tennis impairs most adults’ ability to learn tennis.)
D. In the OP’s “don’t ask what the wine costs, it would ruin the evening” example: it seems to me that there really is a dynamic where asking what the wine costs can at least mildly harm my own experience of the evening, and that for me (and I imagine quite a few others), the harm is not that asking the wine’s price reveals a stable, persistent fact that the asker cares about money. Rather, the harm is asking it breaks the compartmentalization that was part of how I knew how to be “in flow” for the evening. Like, after the asking, I’m thinking about money, or thinking about others thinking about money, and I’m somehow less good at candlelight and music and being with my and others’ experiences when that is happening. (This is why Zvi describes it as “slightly socially awkward” — awkwardness is what it feels like when a flow is disrupted.) (We can tell that the thing that’s up here in my experience of the evening isn’t about longer-term money-indicators, partly because I have no aversion to hearing the same people talk about caring about money in most other contexts.) (I’m sure straight money-signaling, as in Zvi’s interpretation, also happens with some people about the wine. But the different “compartmentalization is better for romantic evenings” dynamic I’m describing can happen too.)
E. This is the example I care most about, and am least likely to do justice to. Um: there’s a lot of pain/caring that I find myself dissociating from, most of the time. (Though I can only see it in flashes.) For example, it’s hard for me to think about death. Or about AI risk, probably because of the “death” part. Or about how much I love people. Or how I hope I have a good life, and how much I want children. I can think words about these things, but I tend to control my breathing while doing so, to become analytic, to look at things a bit from a distance, to sort of emulate the thoughts rather than have them.
It seems to me my dissociating here is driven less by raw pain/caring being unpleasant (although sometimes it is), and more by the fact that when I am experiencing raw pain/caring it is harder to predict/plan/control my own behavior, and that lack of predictability is at least somewhat scary and risky. Plus it is somehow tricky for other people to be around, such that I would usually feel impolite doing it and avoid visibly caring in certain ways for that reason. (See the example F.)
F. [Kind of like E, but as an interpersonal dynamic] When other people show raw caring, it’s hard for me to avoid dissociating. Especially if it’s to do with something where… the feeling inside my head is something like “I want this, I am this, but I can’t have this. It isn’t mine. Fear. Maybe I’m [inadequate/embarrassing/unable forever]?” Example: a couple days ago, some friends and I watched “It’s a Wonderful Life”, which I hadn’t seen before. And afterward a friend and I were raw and talking, and my friend was, I can’t remember, but talking about wanting to be warm and human or something. And it was really hard for me not to just dissociate — I kept having all kinds of nonsense arguments pop into my head for why I should think about my laundry, why I get analytic-and-in-control-of-the-conversation, why I should interrupt him. And later on, my friend was “triggered” about a different thing, and I noticed it was the same [fear/blankness/tendency-to-want-to-dissociate] in me, in response to those other active currents. And I commented on it to my friend, and we noticed that the thing I was instinctively doing in response to that fear in me, was kind of sending my friend “this is weird/bad what you’re doing” signals. So. Um. Maybe there’s a thing where, once people start keeping raw pain/caring/love/anything at distance, if they run into other people who aren’t, they send those people “you’re being crazy/bad” signals whenever those other people aren’t keeping their own raw at a distance. And so we socialize each other to dissociate.
(This connects still to the compartmentalization-as-an-aid-to-predictability thesis, because part of the trouble with e.g. somebody else talking about death, or being raw, or triggered, is that it makes it harder for me to dissociate, and so makes me less predicable/controllable to me.)
G. This brings me to an alternate possible mechanics of Zvi’s “carefully not going out of one’s way not to poison the river with the widget factory” example. If lots of people at WidgetCorp wanted to contribute to (the environment / good things broadly), but are dissociated from their desire, it might mess with their dissociation (and, thus, their control and predictability-to-themselves of their own behavior, and plus WidgetCorp’s ability to predict and control them) if anybody else visibly cares about the river (or even, visibly does a thing one could mistake as caring about the river). And so we get the pressure that Zvi mentions, here and in his “moral mazes” sequence. (And we can analogously derive a pressure not to be a “goody two-shoes” among kids who kind of want to be good still, but also kind of want to be free from that wanting. And the pressure not to be too vulnerably sincere in one’s romantic/sexual encounters, and to instead aspire to cynicism. And more generally (in the extreme, at least) to get attack anyone who acts from intact caring. Sort of like an anti-epistemology, but more exactly like an anti-caring.
Extending the E-F-G thing: perhaps we could say “every cause/movement/organization wants to become a pile of defanged pica and ostentatious normalcy (think: Rowling’s Dursleys) that won’t be disruptive to anyone”, as an complimentary/slightly-contrasting description to “every cause wants to be a cult”.
In the extreme, this “removing of all impulses that’ll interfere with predictability-and-control” is clearly not useful for anything. But in medium-sized amounts, I think predictability/controllability-via-compartmentalization can actually help with creating physical goods, as with the surgeon or poker player or tennis player who has an easier time when they are not in touch with an intense desire for a particular outcome. And I think we see it sometimes in large amounts — large enough that they are net-detrimental to the original goal of the person/cause/business/etc.
Maybe it’s something like:
Being able to predict and control one’s own actions, or one’s organization’s actions, is in fact useful. You can use this to e.g. take three coordinated actions in sequence that will collectively but not individually move you toward a desired outcome, such as putting on your shoes in order to walk to the store in order to be able to buy pasta in order to be able to cook it for dinner. (I do not think one can do this kind of multi-step action nearly as well without prediction-and-control of one’s behavior.)
Because it is useful, we build apparatuses that support it. (“Egos” within individual humans; structures of management and deferral and conformity within organizations and businesses and social movements.)
Even though prediction-and-control is genuinely useful, a central planning entity doing prediction-and-control will tend to overestimate the usefulness of its having more prediction-and-control, and to underestimate the usefulness of aspects of behavior that it does not control. This is because it can see what it’s trying to do, and can’t see what other people are trying to do. Also, its actions are specifically those that its own map says will help, and others’ actions are those which their own maps say will help, which will bring in winner’s curse-type dynamics. So central planning will tend to over-invest in increasing its own control, and to under-invest in allowing unpredictability/disruption/alternate pulls on behavior.
… ? [I think the above three bullet points are probably a real thing that happens. But it doesn’t seem to take my imagination all the way to full-on moral mazes (for organizations), or to individuals who are full-on trying to prop up their ego at the expense of everything. Maybe it does and I’m underestimating it? Or maybe there are added steps after my third bullet point of some sort?]
[Epistemic status: I’m not confident of any of this; I just want better models and am trying to articulate mine in case that helps. Also, all of my comments on this post are as much a response to the book “Moral Mazes” as to the OP.]
Let’s say that A is good, and that B is also good. (E.g. equality and freedom, or diversity and families, or current lives saved and rationality, or any of a huge number of things.) Let’s consider how the desire-for-A and the desire-for-B might avoid having their plans/goal-achievement disrupted by one another.
In principle, you could build a larger model that explains how to trade off between A and B — a model that subsumes A and B as special cases of a more general good. And then the A-desire and the B-desire could peacefully co-exist and share influence within this larger structure, without disrupting each others’ ability to predict-and-control, or to achieve their goals. (And thereby, they could both stably remain part of your psyche. Or part of your organization. Or part of your subcultural movement. Or part of your overarching civilization’s sense of moral decency. Or whatever. Without one part of your civilization’s sense of moral decency (or etc.) straining to pitch another part of that same sense of moral decency overboard.)
Building a larger model subsuming both the A-is-good and B-is-good models is hard, though. It requires a bunch of knowledge/wisdom/culture to kind a workable model of that sort. Especially if you want everybody to coordinate within the same larger model (so that the predict-and-control thing can keep working). A simpler thing you could attempt instead is to just ban desire B. Then desire-for-B won’t get in the way of your attempt to coordinate around achieving desire A. (Or, in more degenerate cases, it won’t get in the way of your attempt to coordinate around you-the-coordinator staying coordinating, with all specific goals mostly forgotten about.) This “just abolish desire B” thing is much simpler to design. So this simpler strategy (“disown and dissociate from one of the good things”) can be reinvented even in ignorance, and can also be shared/evangelized for pretty easily, without needing to share a whole culture.
Separately: once upon a time, there used to be a shared deep culture that gave all humans in a given tribe a whole bunch of shared assumptions about how everything fit together. In that context, it was easier to create/remember/invoke common scaffolds allowing A-desire and B-desire to work together without disrupting each others’ ability to do predictability-and-control. You did not have to build such scaffolds from scratch.
Printing presses and cities and travel/commerce/conversation between many different tribes, and individuals acquiring more tools for creating new thoughts/patterns/associations, and… social media… later made different people assume different things, or fewer things. It became extra-hard to create shared templates in which A-desire and B-desire can coordinate. And so we more often saw social movements / culture wars in which the teams (which each have some memory of some fragment of what’s good) are bent on destroying one another, lest one another destroy their ability to do prediction-and-control in preservation of their own fragment of what’s good. “Humpty Dumpty sat on a wall…”
(Because the ability to do the simpler “dissociate from desire B, ban desire B” move does not break down as quickly, with increasing cultural diversity/fragmentation, as the ability to do the more difficult “assimilate A and B into a common larger good” move.)
Also: it seems to me that “G” might be the generator of the thing Zvi calls as “Moloch’s Army.” Zvi writes:
For whatever it’s worth, I am also inclined to think that something like “Moloch’s Army” describes something important in the world. As sort-of-mentioned, Atlas Shrugged more or less convinced me of this by highlighting a bunch of psychological dynamics that, once highlighted, I seemed to see in myself and others. But I am still confused about it (whether it’s real; what it’s made of insofar as there is a real thing like that). And G is my best current attempt to derive it.
Reminds me of The Costs of Reliability
Oh man; that article is excellent and I hadn’t seen it. If anyone’s wondering whether to click the link: highly recommend.
It’s currently up for review if anyone wants to write a review :)
Here is my take:
Value is a function of the entire state space, and can’t be neatly decomposed as a sum of subgames.
Rather (dually), value on (“quotient”) subgames must be confluent with the total value on the joint game.
Eg, there’s an “enjoying the restaurant food” game, and a “making your spouse happy” game, but the joint game of “enjoying a restaurant with your spouse” game has more moves available, and more value terms that don’t show up in either game, like “be a committed couple”.
”Confluence” here means that what you need to forget to zoom in on the “enjoying the restaurant food” subgame causes your value judgement of “enjoying the restaurant food” and “enjoying a restaurant with your spouse, ignoring everything except food” to agree.
The individual subgames aren’t “closed”, they were never closed, their value only makes sense in a larger context, because the primitives used to define that value refer to the larger context. From the perspective of the larger game, no value is “destroyed”, it only appears that way when projecting into the subgames, which were only ever virtual.
This is just saying the coordination that results from the destruction of value is more valuable than the value destroyed, externalities disregarded. The post is about finding cheaper coordination strategies and internalizing more of the externalities.