Either scenario clearly implies that these estimates are severely distorted and have to be interpreted as marketing copy designed to control your behavior, not unbiased estimates designed to improve the quality of your decisionmaking process.
While most things have at least some motives to control your behavior, I do think GiveWell outlines a pretty reasonable motivation here that they explained in detail in the exact blogpost that you linked (and I know that you critiqued that reasoning on your blog, though I haven’t found the arguments there particularly compelling). Even if their reasoning is wrong, they might still genuinely believe that their reasoning is right, which I do think is very important to distinguish from “marketing copy designed to control your behavior”.
I am often wrong and still try to explain to others why I am right. Sometimes this is the cause of bad external incentives, but sometimes it’s also just a genuine mistake. Humans are not perfect reasoners and they make mistakes for reasons other than to take advantage of other people (sometimes they are tired, or sometimes they haven’t invented physics yet and try to build planes anyway, or sometimes they haven’t figured out what good game theory actually looks like and try their best anyways).
For clarity, the claim Givewell-at-the-time made was:
For giving opportunities that are above the benchmark we’ve set but not “must-fund” opportunities, we want to recommend that Good Ventures funds 50%. It’s hard to say what the right long-term split should be between Good Ventures (a major foundation) and a large number of individual donors, and we’ve chosen 50% largely because we don’t want to engineer – or appear to be engineering – the figure around how much we project that individuals will give this year (which would create the problematic incentives associated with “funging” approaches). A figure of 50% seems reasonable for the split between (a) one major, “anchor” donor who has substantial resources and great conviction in a giving opportunity; (b) all other donors combined.
With the two claims I’ve heard about why the 50% split being:
1. There’s still more than $8 billion dollars worth of good to do, and they expect their last dollar to be worth more than current dollars.
(I agree that this is at least somewhat sketchy, esp. when you think about Gates Foundation and others, although I think the case is less strong than Benquo is presenting here)
2. Having a charity have most/all of their money come from a single donor creates some distorting effects, where the charity feels more beholden to that donor. Whereas if their donations are diversified the charity feels more free to make their own strategic choices. (I more recently heard someone from [OpenPhil or Givewell, can’t remember which], saying that they sometimes made offhand comments to an org like “hmm, would it make sense for you to do X?” and the org treated that like “OMG we need to do X in order to get GoodVentures money” and then ran off to implement X, when the OpenPhil researcher had meant that more as an offhand hypothesis.
This second point seems pretty plausible to me, and was the thing that ultimately updated me away from the “OpenPhil should just fund everything” hypothesis.
Benquo, I can’t remember if you have a post concretely addressing that – if so can you link to it?
The problem already exists on multiple levels, and the decision GiveWell made doesn’t really alleviate it much. We should expect that GiveWell / Open Philanthropy Project is already distorting its judgment to match its idea of what Good Ventures wants, and the programs it’s funding are already distorting their behavior to match their idea of what GiveWell / Open Philanthropy Project wants (since many of the “other” donors aren’t actually uncorrelated with GiveWell’s recommendations either!).
This line of thinking also seems like pretty much the opposite of the one that suggests that making a large grant to OpenAI in order to influence it would be a good idea, as I pointed out here. The whole arrangement is very much not what someone who was trying to avoid this kind of problem would build, so I don’t buy it as an ad hoc justification for this particular decision.
I find this general pattern (providing reasons for things, that if taken seriously would actually recommend a quite different course of action than the one being considered) pretty unfortunate, and I wish I saw a feasible way to insist on better behavior. What’s your model for how GiveWell should behave if they seriously wanted to avoid that sort of distortion? Why do you think it matches their revealed preferences?
This ended up taking awhile (and renewed some of my sympathy for the “I tried to discuss this all clearly and dispassionately and basically nobody listened” issue).
This section mostly summarized the bits I and Benquo covered in this thread, with Ben’s takeaways being:
[...]
[The issues of top charity independence are] partially mitigated by the indirect and time-delayed nature of GiveWell’s influence. In 2013, when AMF had been removed from the list of GiveWell top charities due to concerns about its room for more funding, it still received millions from GiveWell-influenced donors, suggesting that GiveWell donors are applying some amount of independent judgment. If this money all or mostly came from a single donor, it could exacerbate this problem, or lead top charities to hold some of the money in reserve.
If GiveWell is concerned about this effect, it could ask top charities how much money they would be willing to spend in a year from a single donor. Good Ventures could also negotiate a taper-down schedule for withdrawing funding, to reduce the potential costs of withdrawing a major source of funds.
I’m not sure I understand these suggestions yet, but they seem worth mulling over.
GiveWell independence
This section was fairly long (much longer than the previous one). I’m tempted to say “the thing I really cared about was the answer to the first problem”. But I’ve tried to build a habit where, when I ask a question and someone responds in a different frame, I try to grok why their frame is different since that’s often more illuminating (and at least seems like good form, building good will so that when I’m confident my frame makes more sense I can cash in and get others to try to understand mine)
Summarizing the section will take awhile and for now I think I just recommend people read the whole thing.
My off-the-cuff, high level response to the Givewell independence section + final conslusions (without having fully digested them) is:
Ben seems to be arguing that Givewell should either become much more independent from Good Ventures and OpenPhil (and probably move to a separate office), so that it can actually present the average donor will unbiased, relevant information (rather than information entangled with Good Venture’s goals/models)
or
The other viable option is for GiveWell to give up for now on most public-facing recommendations and become a fully-funded branch of Good Ventures, to demonstrate to the world what GiveWell-style methods can do when applied to a problem where it is comparatively easy to verify results.
I can see both of these as valid options to explore, and that going to either extreme would probably maximize particular values.
But it’s not obvious either of those maximize area-under-the-curve-of-total-values.
There’s value to people with deep models being able to share those models. Bell Labs worked by having people being able to bounce ideas off each other, casually run into each other, and explain things to each other iteratively. My current sense is that I wish there was more opportunity for people in the EA landscape to share models more deeply with each other on a casual, day-to-day basis, rather than less (while still sharing as much as possible with the general public so people in the general public can also get engaged)
This does come with tradeoffs of neither maximizing independent judgment nor maximizing output nor most easily avoiding particular epistemic and integrity pitfalls, but it’s where I expect the most total value to lie.
There’s value to people with deep models being able to share those models. Bell Labs worked by having people being able to bounce ideas off each other, casually run into each other, and explain things to each other iteratively. My current sense is that I wish there was more opportunity for people in the EA landscape to share models more deeply with each other on a casual, day-to-day basis, rather than less (while still sharing as much as possible with the general public so people in the general public can also get engaged)
Trying to build something kind of like Bell Labs would be great! I don’t see how it’s relevant to the current discussion, though.
Right now, we (maybe? I’m not sure) have something like a few different mini-Bell-labs, that each have their own paradigm (and specialists within that paradigm).
The world where Givewell, Good Ventures and OpenPhil share an office is more Bell Labs like than one where they all have different offices. (FHI and UK CEA is a similar situation, as is CFAR/MIRI/LW). One of your suggestions in the blogpost was specifically that they split up into different, fully separate entities.
I’m proposing that Bell Labs exists on a spectrum, that sharing office space is a mechanism to be more Bell Labs like, and that generally being more Bell Labs like is better (at least in a vacuum)
(My shoulder Benquo now says something like “but if you’re models are closely entangled with those of your funders, don’t pretend like you are offering neutral services.” Or maybe “it’s good to share office space with people thinking about physics, because that’s object level. It’s bad to share office space with the people funding you.” Which seems plausible but not overwhelmingly obvious given the other tradeoffs at play)
People working at Bell Labs were trying to solve technical problems, not marketing or political problems. Sharing ideas across different technical disciplines is potentially a good thing, and I can see how FHI and MIRI in particular are a little bit like this, though writing white papers is a very different even within a technical field from figuring out how to make a thing work. But it doesn’t seem like any of the other orgs substantially resemble Bell Labs at all, and the benefits of collocation for nontechnical projects are very different from the benefits for technical projects—they have more to do with narrative alignment (checking whether you’re selling the same story), and less to do with opportunities to learn things of value outside the context of a shared story.
Collocation of groups representing (others’) conflicting interests represents increased opportunity for corruption, not for generative collaboration.
Okay. I’m not sure whether I agree precisely but agree that that’s the valid hypothesis, which I hadn’t considered before in quite these terms, and updates my model a bit.
Collocation of groups representing (others’) conflicting interests represents increased opportunity for corruption, not for generative collaboration.
The version of this that I’d more obviously endorse goes:
Collocation of groups representing conflicting interests represents increased opportunity for corruption.
Collocation of people who are building models represents increased opportunity for generative collaboration.
Collocation of people who are strategizing together represents increased opportunity for working on complex goals that require shared complex models, and/or shared complex plans. (Again, as said elsethread, I agree that plans are models are different, but I think they are subject to a lot of the same forces, with plans being subject to some additional forces as well)
I also think “sharing a narrative” and “building technical social models” are different, although easily confused (both from the outside and inside – I’m not actually sure which confusion is easier). But you do actually need social models if you’re tackling social domains, which do actually benefit from interpersonal generativity.
My shoulder Benquo now says something like “but if you’re models are closely entangled with those of your funders, don’t pretend like you are offering neutral services.” Or maybe “it’s good to share office space with people thinking about physics, because that’s object level. It’s bad to share office space with the people funding you.”
I think these are a much stronger objection jointly than separately. If Cari Tuna wants to run her own foundation, then it’s probably good for her to collocate with the staff of that foundation.
(I do want to note that this is a domain where I’m quite confused about the right answer. I think I stand by the individual comments I made last night but somewhat regret posting them as quickly as I did without thinking about it more and it seems moderately likely that some pieces of my current take on the situation are incoherent)
Some further thoughts on that: I agree social-reality-distortions are a big problem, although I don’t think the werewolf/villager-distinction is the best frame. (In answer to Wei_Dai’s comment elsethread, “am I a werewolf” isn’t a very useful question. You almost certainly are at least slightly cognitively-distorted due to social reality, at least some of the time. You almost certainly sometimes employ obfuscatory techniques in order to give yourself room to maneuver, at least sort of, at least some times.)
But I think thinking in terms of villagers and werewolves leads you to ask the question ‘who is a werewolf’ moreso than ‘how do we systematically disincentivize obfuscatory or manipulative behavior’, which seems a more useful question.
I bring this all up in this particular subthread because I think it’s important that one thing that incentivizes obfuscatory behavior is giving away billions of dollars.
My sense (not backed up by much legible argument) is that a major source of inefficiencies of the Gates Foundation (and OpenPhil to a lesser degree) is that they’ve created an entire ecosystem, which both attracts people motivated by power/money/prestige (simply to staff the organization), as well as incentives for charities to goodhart themselves to become legibly-valuable to the Gates Foundation’s values.
Meanwhile, my experiencing reading OpenPhil articles is that they usually take pretty serious pains to say “don’t take our estimates literally, these are very rough, please actually look at the spreadsheets that generated them and plug in your own values.” AFAICT they’re making a pretty good faith effort actually just be able to talk about object-level stuff without their statements being enactive language, and it’s just really hard to get people to treat them that way.
(There are certainly older Givewell posts that seem to be concretely making the “drowning children everywhere” mistake, but AFAICT the current AMF page doesn’t even give a concrete final estimate at all, instead the various object level costs and then linking to a spreadsheet and a whole other blogpost about how they do cost estimates)
I do see patterns within the broader EA community that push towards taking the low-cost-per-lives-saved estimates literally, where there are lots of movement-building forces that really want to translate things into a simple, spreadable message. Some of this seems like it was caused or exacerbated by specific people at specific times, but it also seems like the movement-building-forces almost exist as a force in their own right that’s hard to stop.
It seems like there’s this general pattern, that occurs over and over, where people follow a path going:
1. Woah. Drowning child argument!
2. Woah. Lives are cheap!
3. Woah, obviously this is important to take action on and scale up now. Mass media! Get the message out!
4. Oh. This is more complicated.
5. Oh, I see, it’s even more complicated. (where complication can include moving from global poverty to x-risk as a major focus, as well as realizing that global poverty isn’t as simple to solve)
6. Person has transitioned into a more nuanced and careful thinker, and now is one of the people in charge of some kind of org or at least a local community somewhere. (for one example, see CEA’s article on shifting from mass media to higher fidelity methods of transition)
But, the mass media (and generally simplified types of thinking independent of strategy) are more memetically virulent than the more careful thinking, and new people keep getting excited about them in waves that are self-sustaining and hard to clarify (esp. since the original EA infrastructure was created by people at the earlier stages of thinking). So it keeps on being something that a newcomer will bump into most often in EA spaces.
CEA continues to actively make the kinds of claims implied by taking GiveWell’s cost per life saved numbers literally, as I pointed out in the post. Exact quote from the page I linked:
But I think thinking in terms of villagers and werewolves leads you to ask the question ‘who is a werewolf’ moreso than ‘how do we systematically disincentivize obfuscatory or manipulative behavior’, which seems a more useful question.
But I think thinking in terms of villagers and werewolves leads you to ask the question ‘who is a werewolf’ moreso than ‘how do we systematically disincentivize obfuscatory or manipulative behavior’, which seems a more useful question.
Clearly, the second question is also useful, but there is little hope of understanding, much less effectively counteracting, obfuscatory behavior, unless at least some people can see it as it happens, i.e. detect who is (locally) acting like a werewolf. (Note that the same person can act more/less obfuscatory at different times, in different contexts, about different things, etc)
Sure, I just think the right frame here is “detect and counteract obfuscatory behavior” rather than “detect werewolves.” I think the “detect werewolves”, or even “detect werewolf behavior” frame is more likely to collapse into tribal and unhelpful behavior at scale [edit: and possibly before then]
(This is for very similar reasons to why EA arguments often collapse into “donate all your money to help people”. It’s not that the nuances position isn’t there, it just gets outcompeted by simpler versions of itself)
In your previous comment you’re talking to Wei Dai, though. Do you think Wei Dai is going to misinterpret the werewolf concept in this manner? If so, why not link to the original post to counteract the possible misinterpretation, instead of implying that the werewolf frame itself is wrong?
(meta note: I’m worried here about the general pattern of people optimizing discourse for “the public” who is nonspecific and assumed to be highly uninformed / willfully misinterpreting / etc, in a way that makes it impossible for specific, informed people (such as you and Wei Dai) to communicate in a nuanced, high-information fashion)
[EDIT: also note that the frame you objected to (the villagers vs werewolf frame) contains important epistemic content that the “let’s incentivize non-obfuscatory behavior” frame doesn’t, as you agreed in your subsequent comment after I pointed it out. Which means I’m going to even more object to saying “the villagers/werewolf frame is bad” with the defense being that “people might misinterpret this”, without offering a frame that contains the useful epistemic content of the misinterpretable frame]
But I’m worried here about well-informed people caching ideas wrongly, not about the general public. More to say about this, but first want to note:
also note that the frame you objected to (the villagers vs werewolf frame) contains important epistemic content that the “let’s incentivize non-obfuscatory behavior” frame doesn’t, as you agreed in your subsequent comment after I pointed it out.
Huh—this just feels like a misinterpretation or reading odd things into what I said.
It had seemed obvious to me that to disincentivize obfuscatory behavior, you need people to be aware of what obfuscatory behavior looks like and what to do about it, and it felt weird that you saw that as something different.
It is fair that I may not have communicated that well, but that’s part of my point – communication is quite hard. Similarly, I don’t think the original werewolf post really communicates the thing it was meant to.
“Am I a werewolf” is not a particularly useful question to ask, and neither is “is so and so a werewolf?” because the answer is almost always “yes, kinda.” (and what exactly you mean by “kinda” is doing most of the work). But, nonetheless, this is the sort of question that the werewolf frame prompts people to ask.
I’m worried about this, concretely, because after reading Effective Altruism is Self Recommending a while, despite the fact that I thought lots about it, and wrote up detailed responses to it (some of which I posted and some of which I just thought about privately), and I ran a meetup somewhat inspired by taking it seriously...
...despite all that, a year ago when I tried to remember what it was about, all I could remember was “givewell == ponzi scheme == bad”, without any context of why the ponzi scheme metaphor mattered or how the principle was supposed to generalize. I’m similarly worried that a year from now, “werewolves == bad, hunt werewolves”, is going to be the thing I remember about this.
The five-word-limit isn’t just for the uninformed public, it’s for serious people trying to coordinate. The public can only coordinate around 5-word things. Serious people trying to be informed still have to ingest lots of information and form detailed models but those models are still going to have major bits that are compressed, out of pieces that end up being about five words. And this is a major part of why many people are confused about Effective Altruism and how to do it right in the first place.
I’m worried about this, concretely, because after reading Effective Altruism is Self Recommending a while, despite the fact that I thought lots about it, and wrote up detailed responses to it (some of which I posted and some of which I just thought about privately), and I ran a meetup somewhat inspired by taking it seriously...
...despite all that, a year ago when I tried to remember what it was about, all I could remember was “givewell == ponzi scheme == bad”, without any context of why the ponzi scheme metaphor mattered or how the principle was supposed to generalize. I’m similarly worried that a year from now, “werewolves == bad, hunt werewolves”, is going to be the thing I remember about this.
The five-word-limit isn’t just for the uninformed public, it’s for serious people trying to coordinate. The public can only coordinate around 5-word things. Serious people trying to be informed still have to ingest lots of information and form detailed models but those models are still going to have major bits that are compressed, out of pieces that end up being about five words. And this is a major part of whymany people are confused about Effective Altruism and how to do it right in the first place.
If that’s your outlook, it seems pointless to write anything longer than five words on any topic other than how to fix this problem.
I agree with the general urgency of the problem, although I think the frame of your comment is somewhat off. This problem seems… very information-theoretically-entrenched. I have some sense that you think of it as solvable in a way that it’s fundamentally not actually solvable, just improvable, like you’re trying to build a perpetual motion machine instead of a more efficient engine. There is only so much information people can process.
(This is based entirely off of reading between the lines of comments you’ve made, and I’m not confident what your outlook actually is here, and apologies for the armchair psychologizing).
I think you can make progress on it, which would look something like:
0) make sure people are aware of the problem
1) building better infrastructure (social or technological), probably could be grouped into a few goals:
nudge readers towards certain behavior
nudge writers towards certain behavior
provide tools that amplify readers capabilities
provide tools that amplify writer’s capabilities
2) meanwhile, as a writer, make sure that the concepts you create for the public discourse are optimized for the right kind of compression. Some ideas compress better than others. (I have thought about the details of this
This *is* my outlook, and that yes this that both I, as well as you and Jessica, should probably be taking some kind of action that takes this outlook strategically seriously if you aren’t already.
Distillation Technology
A major goal I have for LessWrong, which the team has talked about a lot, is improving distillation technology. It’s not what we’re currently working on because, well, there are *multiple* top priorities that all seem pretty urgent (and all seem like pieces of the same puzzle). But I think Distillation Tech is the sort of thing most likely to meaningfully improve the situation.
Right now the default mode people interact with LessWrong and many other blogging platforms is “write up a thing, post it, maybe change a few things in response to feedback.” But for ideas that are actually going to become building blocks of the intellectual commons, you need to continuously invest in improving them.
Arbital tried to do this, and it failed because the problem is hard in weird ways, many of them somewhat hard to anticipate.
http://distill.pub tackles a piece of this but not in a way that seems especially scalable.
Scott Alexander’s short story Ars Longa Vita Brevis is a fictional account of what seems necessary to me.
I do hope that by the end of this year the LW team will have made some concrete progress on this. I think it is plausibly a mistake that we haven’t focused on it already – we discussed switching gears towards it at our last retreat but it seemed to make more sense to finish Open Questions.
Trying to nudge others seems like an attempt to route around the problem rather than solve it. It seems like you tried pretty hard to integrate the substantive points in my “Effective Altruism is self-recommending” post, and even with pretty extensive active engagement, your estimate is that you only retained a very superficial summary. I don’t see how any compression tech for communication at scale can compete with what an engaged reader like you should be able to do for themselves while taking that kind of initiative.
We know this problem has been solved in the past in some domains—you can’t do a thing like the Apollo project or build working hospitals where cardiovascular surgery is regularly successful based on a series of atomic five-word commands; some sort of recursive general grammar is required, and at least some of the participants need to share detailed models.
One way this could be compatible with your observation is that people have somewhat recently gotten worse at this sort of skill; another is that credit-assignment is an unusually difficult domain to do this in. My recentblog posts have argued that at least the latter is true.
In the former case (lost literacy), we should be able to reconstruct older modes of coordination. In the latter (politics has always been hard to think clearly about), we should at least internally be able to learn from each other by learning to apply cognitive architectures we use in domains where we find this sort of thing comparatively easy.
I think I may have communicatedly somewhat poorly by phrasing this in terms of 5 words, rather than 5 chunks, and will try to write a new post sometime that presents a more formal theory of what’s going on.
Coordinated actions can’t take up more bandwidth than someone’s working memory (which is something like 7 chunks, and if you’re using all 7 chunks then they don’t have any spare chunks to handle weird edge cases).
A lot of coordination (and communication) is about reducing the chunk-size of actions. This is why jargon is useful, habits and training are useful (as well as checklists and forms and bureaucracy), since that can condense an otherwise unworkably long instruction into something people can manage.
And:
The “Go to the store” is four words. But “go” actually means “stand up. walk to the door. open the door. Walk to your car. Open your car door. Get inside. Take the key out of your pocket. Put the key in the ignition slot...” etc. (Which are in turn actually broken into smaller steps like “lift your front leg up while adjusting your weight forward”)
But, you are capable of taking all of that an chunking it as the concept “go somewhere” (as as well as the meta concept of “go to the place whichever way is most convenient, which might be walking or biking or taking a bus”), although if you have to use a form of transport you are less familiar with, remembering how to do it might take up a lot of working memory slots, leaving you liable to forget other parts of your plan.
I do in fact expect that the Apollo project worked via finding ways to cache things into manageable chunks, even for the people who kept the whole project in their head.
I’d be interested in figuring out how to operationalize this as a bet and check how the project actually worked. What I have heard (epistemic status: heard it from some guy on the internet) is that actually, most people on the project did not have all the pieces in their head, and the only people who did were the pilots.
My guess is that the pilots had a model of how to *use* and *repair* all the pieces of the ship, but couldn’t have built it themselves.
My guess it that “the people who actually designed and assembled the thing” had a model of how all the pieces fit together, but not as a deep a model of how and when to use it, and may have only understood the inputs and outputs of each piece.
And meanwhile, while I’m not quite sure how to operationalize the bet, I would bet maybe $50 that (conditional on us finding a good operationalization), that the number of people who had the full model or anything like it was quite small. (“You Have About Five Words” doesn’t claim you can’t have more than 5 words of nuance, it claims that you can’t coordinate large groups of people that depend on more than 5 words of nuance. I bet there were less than 100 people and probably closer to 10 who had anything like a full model of everything going on)
and will try to write a new post sometime that presents a more formal theory of what’s going on
I think I’m unclear on how this constrains anticipations, and in particular it seems like there’s substantial ambiguity as to what claim you’re making, such that it could be any of these:
You can’t communicate recursive structures or models with more than five total chunks via mass media such as writing.
You can’t get humans to act (or in particular to take initiative) based on such models, so you’re limited to direct commands when coordinating actions.
There exist such people, but they’re very few and stretched between very different projects and there’s nothing we can do about that.
I think there are two different anticipation-constraining-claims, similar but not quite what you said there:
Working Memory Learning Hypothesis – people can learn complex or recursive concepts, but each chunk that they learn cannot be composed of more than 7 other chunks. You can learn a 49 chunk concept but first must distill it into seven 7-chunk-concepts, learn each one, and then combine them together.
Coordination Nuance Hypothesis – there are limits to how nuanced a model you can coordinate around, at various scales of coordination. I’m not sure precisely what the limits are, but it seems quite clear that the more people you are coordinating the harder it is to get them to share a nuanced model or strategy. It’s easier to have a nuanced strategy with 10 people than 100, 1000, or 10,000.
I’m less confident of the Working Memory hypothesis (it’s an armchair inside view based on my understanding of how working memory works)
I’m fairly confident in the Coordination Nuance Hypothesis, which is based on observations about how people actually seem to coordinate at various scales and how much nuance they seem to preserve.
In both cases, there are tools available to improve your ability to learn (as an individual), disseminate information (as a communicator), and keep people organized (as a leader). But none of the tools changed the fundamental equation, just the terms.
Anticipation Constraints:
The anticipation-constraint of the WMLH is “if you try to learn a concept that requires more than 7 chunks, you will fail. If a concept requires 12 chunks, you will not successfully learn it (or will learn a simplified bastardization of it) until you find a way to compress the 12 chunks into 7. If you have to do this yourself, it will take longer than if an educator has optimized it for you in advance.”
The anticipation constraint of the CNH is that if you try to coordinate with 100 people of a given level of intelligence, the shared complexity of the plan that you are enacting will be lower than the complexity of the plan you could enact with 10 people. If you try to implement a more complex plan or orient around a more complex model, your organization will make mistakes due to distorted simplifications of the plan. And this gets worse as your organizations scales.
I agree they are different but think it is the case that with a larger group you have a harder time with either of them, for roughly the same reasons at roughly the same rate of increased difficulty.
The Working Memory Hypothesis says the Bell Labs is useful, in part, because whenever you need to combine multiple interdisciplinary concepts that are each complicated to invent a new concept…
instead of having to read a textbook that explains it one-particular-way (and, if it’s not your field, you’d need to get up to speed on the entire field in order to have any context at all) you can just walk down the hall and ask the guy who invented the concept “how does this work” and have them explain it to you multiple times until they find a way to compress it down into a 7 chunks, optimized for your current level of understanding.
A slightly more accurate anticipation of the CNH is:
people need to spend time learning a thing in order to coordinate around it. At the very least, the more time you need to spend getting people up to speed on a model, the less time they have to actually act on that model
people have idiosyncratic learning styles, and are going to misinterpret some bits of your plan, and you won’t know in advance which ones. Dealing with this requires individual attention, noticing their mistakes and correcting them. Middle managers (and middle “educators” can help to alleviate this, but every link in the chain reduces your control over what message gets distributed. If you need 10,000 people to all understand and act on the same plan/model, it needs to be simple or robust enough to survive 10,000 people misinterpreting it in slightly different ways
This gets even worse if you need to change your plan over time in response to new information, since now people are getting it confused with the old plan, or they don’t agree with the new plan because they signed up for the old plan, and then you have to Do Politics to get them on board with the new plan.
At the very least, if you’ve coordinated perfectly, each time you change your plan you need to shift from “focusing on execution” to “focusing on getting people up to speed on the new model.”
The way that I’d actually do this, and plan to do this (in line with Benquo’s reply to you), is to repackage the concept into something that I understand more deeply and which I expect to unpack more easily in the future.
Part of this requires me to do some work for myself (no amount of good authorship can replace putting at least some work into truly understanding something)
Part of this has to do with me having my own framework (rooted in Robust Agency among other things) which is different from Benquo’s framework, and Ben’s personal experience playing werewolf.
But a lot of my criticism of the current frame is that it naturally suggest compacting the model in the wrong way. (to be clear, I think this is fine for a post that represents a low-friction strategy to post your thoughts and conversations as they form, without stressing too much about optimizing pedagogy. I’m glad Ben posted the Villager/Werewolf post. But I think the presentation makes it harder to learn than it needs to be, and is particularly ripe for being misinterpreted in a way that benefits rather than harms werewolves, and if it’s going to be coming up in conversation a lot I think it’d be worth investing time in optimizing it better)
That seems like the sort of hack that lets you pass a test, not the sort of thing that makes knowledge truly a part of you. To achieve the latter, you have to bump it up against your anticipations, and constantly check to see not only whether the argument makes sense to you, but whether you understand it well enough to generate it in novel cases that don’t look like the one you’re currently concerned with.
I think it’s possible to use in a “mindful” way even if most people are doing it wrong? The system reminding you what you read n days ago gives you a chance to connect it to the real world today when you otherwise would have forgotten.
Holden Karnofsky explicitly disclaimed the “independence via multiple funders” consideration as not one that motivated the partial funding recommendation.
While most things have at least some motives to control your behavior, I do think GiveWell outlines a pretty reasonable motivation here that they explained in detail in the exact blogpost that you linked (and I know that you critiqued that reasoning on your blog, though I haven’t found the arguments there particularly compelling). Even if their reasoning is wrong, they might still genuinely believe that their reasoning is right, which I do think is very important to distinguish from “marketing copy designed to control your behavior”.
I am often wrong and still try to explain to others why I am right. Sometimes this is the cause of bad external incentives, but sometimes it’s also just a genuine mistake. Humans are not perfect reasoners and they make mistakes for reasons other than to take advantage of other people (sometimes they are tired, or sometimes they haven’t invented physics yet and try to build planes anyway, or sometimes they haven’t figured out what good game theory actually looks like and try their best anyways).
For clarity, the claim Givewell-at-the-time made was:
With the two claims I’ve heard about why the 50% split being:
1. There’s still more than $8 billion dollars worth of good to do, and they expect their last dollar to be worth more than current dollars.
(I agree that this is at least somewhat sketchy, esp. when you think about Gates Foundation and others, although I think the case is less strong than Benquo is presenting here)
2. Having a charity have most/all of their money come from a single donor creates some distorting effects, where the charity feels more beholden to that donor. Whereas if their donations are diversified the charity feels more free to make their own strategic choices. (I more recently heard someone from [OpenPhil or Givewell, can’t remember which], saying that they sometimes made offhand comments to an org like “hmm, would it make sense for you to do X?” and the org treated that like “OMG we need to do X in order to get GoodVentures money” and then ran off to implement X, when the OpenPhil researcher had meant that more as an offhand hypothesis.
This second point seems pretty plausible to me, and was the thing that ultimately updated me away from the “OpenPhil should just fund everything” hypothesis.
Benquo, I can’t remember if you have a post concretely addressing that – if so can you link to it?
Here’s the part of the old series that dealt with this consideration: http://benjaminrosshoffman.com/givewell-case-study-effective-altruism-4/
The problem already exists on multiple levels, and the decision GiveWell made doesn’t really alleviate it much. We should expect that GiveWell / Open Philanthropy Project is already distorting its judgment to match its idea of what Good Ventures wants, and the programs it’s funding are already distorting their behavior to match their idea of what GiveWell / Open Philanthropy Project wants (since many of the “other” donors aren’t actually uncorrelated with GiveWell’s recommendations either!).
This line of thinking also seems like pretty much the opposite of the one that suggests that making a large grant to OpenAI in order to influence it would be a good idea, as I pointed out here. The whole arrangement is very much not what someone who was trying to avoid this kind of problem would build, so I don’t buy it as an ad hoc justification for this particular decision.
I find this general pattern (providing reasons for things, that if taken seriously would actually recommend a quite different course of action than the one being considered) pretty unfortunate, and I wish I saw a feasible way to insist on better behavior. What’s your model for how GiveWell should behave if they seriously wanted to avoid that sort of distortion? Why do you think it matches their revealed preferences?
This ended up taking awhile (and renewed some of my sympathy for the “I tried to discuss this all clearly and dispassionately and basically nobody listened” issue).
First, to save future people some effort, here is my abridged summary of what you said relating to “independence.” (Also: here is a link directly to the relevant part of the blogpost)
Relying on a single donor does come with issues.
There are separate issues for:
Givewell’s Independence (from Good Ventures)
Top Charity Independence (from Givewell)
Top Charity Independence
This section mostly summarized the bits I and Benquo covered in this thread, with Ben’s takeaways being:
I’m not sure I understand these suggestions yet, but they seem worth mulling over.
GiveWell independence
This section was fairly long (much longer than the previous one). I’m tempted to say “the thing I really cared about was the answer to the first problem”. But I’ve tried to build a habit where, when I ask a question and someone responds in a different frame, I try to grok why their frame is different since that’s often more illuminating (and at least seems like good form, building good will so that when I’m confident my frame makes more sense I can cash in and get others to try to understand mine)
Summarizing the section will take awhile and for now I think I just recommend people read the whole thing.
My off-the-cuff, high level response to the Givewell independence section + final conslusions (without having fully digested them) is:
Ben seems to be arguing that Givewell should either become much more independent from Good Ventures and OpenPhil (and probably move to a separate office), so that it can actually present the average donor will unbiased, relevant information (rather than information entangled with Good Venture’s goals/models)
or
I can see both of these as valid options to explore, and that going to either extreme would probably maximize particular values.
But it’s not obvious either of those maximize area-under-the-curve-of-total-values.
There’s value to people with deep models being able to share those models. Bell Labs worked by having people being able to bounce ideas off each other, casually run into each other, and explain things to each other iteratively. My current sense is that I wish there was more opportunity for people in the EA landscape to share models more deeply with each other on a casual, day-to-day basis, rather than less (while still sharing as much as possible with the general public so people in the general public can also get engaged)
This does come with tradeoffs of neither maximizing independent judgment nor maximizing output nor most easily avoiding particular epistemic and integrity pitfalls, but it’s where I expect the most total value to lie.
Trying to build something kind of like Bell Labs would be great! I don’t see how it’s relevant to the current discussion, though.
Right now, we (maybe? I’m not sure) have something like a few different mini-Bell-labs, that each have their own paradigm (and specialists within that paradigm).
The world where Givewell, Good Ventures and OpenPhil share an office is more Bell Labs like than one where they all have different offices. (FHI and UK CEA is a similar situation, as is CFAR/MIRI/LW). One of your suggestions in the blogpost was specifically that they split up into different, fully separate entities.
I’m proposing that Bell Labs exists on a spectrum, that sharing office space is a mechanism to be more Bell Labs like, and that generally being more Bell Labs like is better (at least in a vacuum)
(My shoulder Benquo now says something like “but if you’re models are closely entangled with those of your funders, don’t pretend like you are offering neutral services.” Or maybe “it’s good to share office space with people thinking about physics, because that’s object level. It’s bad to share office space with the people funding you.” Which seems plausible but not overwhelmingly obvious given the other tradeoffs at play)
People working at Bell Labs were trying to solve technical problems, not marketing or political problems. Sharing ideas across different technical disciplines is potentially a good thing, and I can see how FHI and MIRI in particular are a little bit like this, though writing white papers is a very different even within a technical field from figuring out how to make a thing work. But it doesn’t seem like any of the other orgs substantially resemble Bell Labs at all, and the benefits of collocation for nontechnical projects are very different from the benefits for technical projects—they have more to do with narrative alignment (checking whether you’re selling the same story), and less to do with opportunities to learn things of value outside the context of a shared story.
Collocation of groups representing (others’) conflicting interests represents increased opportunity for corruption, not for generative collaboration.
Okay. I’m not sure whether I agree precisely but agree that that’s the valid hypothesis, which I hadn’t considered before in quite these terms, and updates my model a bit.
The version of this that I’d more obviously endorse goes:
Collocation of groups representing conflicting interests represents increased opportunity for corruption.
Collocation of people who are building models represents increased opportunity for generative collaboration.
Collocation of people who are strategizing together represents increased opportunity for working on complex goals that require shared complex models, and/or shared complex plans. (Again, as said elsethread, I agree that plans are models are different, but I think they are subject to a lot of the same forces, with plans being subject to some additional forces as well)
These are all true, and indeed in tension.
I also think “sharing a narrative” and “building technical social models” are different, although easily confused (both from the outside and inside – I’m not actually sure which confusion is easier). But you do actually need social models if you’re tackling social domains, which do actually benefit from interpersonal generativity.
I think these are a much stronger objection jointly than separately. If Cari Tuna wants to run her own foundation, then it’s probably good for her to collocate with the staff of that foundation.
(I do want to note that this is a domain where I’m quite confused about the right answer. I think I stand by the individual comments I made last night but somewhat regret posting them as quickly as I did without thinking about it more and it seems moderately likely that some pieces of my current take on the situation are incoherent)
Thanks. Will re-read the original post and think a bit more.
Some further thoughts on that: I agree social-reality-distortions are a big problem, although I don’t think the werewolf/villager-distinction is the best frame. (In answer to Wei_Dai’s comment elsethread, “am I a werewolf” isn’t a very useful question. You almost certainly are at least slightly cognitively-distorted due to social reality, at least some of the time. You almost certainly sometimes employ obfuscatory techniques in order to give yourself room to maneuver, at least sort of, at least some times.)
But I think thinking in terms of villagers and werewolves leads you to ask the question ‘who is a werewolf’ moreso than ‘how do we systematically disincentivize obfuscatory or manipulative behavior’, which seems a more useful question.
I bring this all up in this particular subthread because I think it’s important that one thing that incentivizes obfuscatory behavior is giving away billions of dollars.
My sense (not backed up by much legible argument) is that a major source of inefficiencies of the Gates Foundation (and OpenPhil to a lesser degree) is that they’ve created an entire ecosystem, which both attracts people motivated by power/money/prestige (simply to staff the organization), as well as incentives for charities to goodhart themselves to become legibly-valuable to the Gates Foundation’s values.
Meanwhile, my experiencing reading OpenPhil articles is that they usually take pretty serious pains to say “don’t take our estimates literally, these are very rough, please actually look at the spreadsheets that generated them and plug in your own values.” AFAICT they’re making a pretty good faith effort actually just be able to talk about object-level stuff without their statements being enactive language, and it’s just really hard to get people to treat them that way.
(There are certainly older Givewell posts that seem to be concretely making the “drowning children everywhere” mistake, but AFAICT the current AMF page doesn’t even give a concrete final estimate at all, instead the various object level costs and then linking to a spreadsheet and a whole other blogpost about how they do cost estimates)
I do see patterns within the broader EA community that push towards taking the low-cost-per-lives-saved estimates literally, where there are lots of movement-building forces that really want to translate things into a simple, spreadable message. Some of this seems like it was caused or exacerbated by specific people at specific times, but it also seems like the movement-building-forces almost exist as a force in their own right that’s hard to stop.
It seems like there’s this general pattern, that occurs over and over, where people follow a path going:
1. Woah. Drowning child argument!
2. Woah. Lives are cheap!
3. Woah, obviously this is important to take action on and scale up now. Mass media! Get the message out!
4. Oh. This is more complicated.
5. Oh, I see, it’s even more complicated. (where complication can include moving from global poverty to x-risk as a major focus, as well as realizing that global poverty isn’t as simple to solve)
6. Person has transitioned into a more nuanced and careful thinker, and now is one of the people in charge of some kind of org or at least a local community somewhere. (for one example, see CEA’s article on shifting from mass media to higher fidelity methods of transition)
But, the mass media (and generally simplified types of thinking independent of strategy) are more memetically virulent than the more careful thinking, and new people keep getting excited about them in waves that are self-sustaining and hard to clarify (esp. since the original EA infrastructure was created by people at the earlier stages of thinking). So it keeps on being something that a newcomer will bump into most often in EA spaces.
CEA continues to actively make the kinds of claims implied by taking GiveWell’s cost per life saved numbers literally, as I pointed out in the post. Exact quote from the page I linked:
Either CEA isn’t run by people in stage 6, or … it is, but keeps making claims like this anyway.
I want to upvote this in particular.
Clearly, the second question is also useful, but there is little hope of understanding, much less effectively counteracting, obfuscatory behavior, unless at least some people can see it as it happens, i.e. detect who is (locally) acting like a werewolf. (Note that the same person can act more/less obfuscatory at different times, in different contexts, about different things, etc)
Sure, I just think the right frame here is “detect and counteract obfuscatory behavior” rather than “detect werewolves.” I think the “detect werewolves”, or even “detect werewolf behavior” frame is more likely to collapse into tribal and unhelpful behavior at scale [edit: and possibly before then]
(This is for very similar reasons to why EA arguments often collapse into “donate all your money to help people”. It’s not that the nuances position isn’t there, it just gets outcompeted by simpler versions of itself)
In your previous comment you’re talking to Wei Dai, though. Do you think Wei Dai is going to misinterpret the werewolf concept in this manner? If so, why not link to the original post to counteract the possible misinterpretation, instead of implying that the werewolf frame itself is wrong?
(meta note: I’m worried here about the general pattern of people optimizing discourse for “the public” who is nonspecific and assumed to be highly uninformed / willfully misinterpreting / etc, in a way that makes it impossible for specific, informed people (such as you and Wei Dai) to communicate in a nuanced, high-information fashion)
[EDIT: also note that the frame you objected to (the villagers vs werewolf frame) contains important epistemic content that the “let’s incentivize non-obfuscatory behavior” frame doesn’t, as you agreed in your subsequent comment after I pointed it out. Which means I’m going to even more object to saying “the villagers/werewolf frame is bad” with the defense being that “people might misinterpret this”, without offering a frame that contains the useful epistemic content of the misinterpretable frame]
I do agree that this is a pattern to watch out for. I don’t think it applies here, but could be wrong. I think it’s very important that people be able to hold themselves to higher standards than what they can easily explain to the public, and it seems like a good reflex to notice when people might be trying to do that and point it out.
But I’m worried here about well-informed people caching ideas wrongly, not about the general public. More to say about this, but first want to note:
Huh—this just feels like a misinterpretation or reading odd things into what I said.
It had seemed obvious to me that to disincentivize obfuscatory behavior, you need people to be aware of what obfuscatory behavior looks like and what to do about it, and it felt weird that you saw that as something different.
It is fair that I may not have communicated that well, but that’s part of my point – communication is quite hard. Similarly, I don’t think the original werewolf post really communicates the thing it was meant to.
“Am I a werewolf” is not a particularly useful question to ask, and neither is “is so and so a werewolf?” because the answer is almost always “yes, kinda.” (and what exactly you mean by “kinda” is doing most of the work). But, nonetheless, this is the sort of question that the werewolf frame prompts people to ask.
I’m worried about this, concretely, because after reading Effective Altruism is Self Recommending a while, despite the fact that I thought lots about it, and wrote up detailed responses to it (some of which I posted and some of which I just thought about privately), and I ran a meetup somewhat inspired by taking it seriously...
...despite all that, a year ago when I tried to remember what it was about, all I could remember was “givewell == ponzi scheme == bad”, without any context of why the ponzi scheme metaphor mattered or how the principle was supposed to generalize. I’m similarly worried that a year from now, “werewolves == bad, hunt werewolves”, is going to be the thing I remember about this.
The five-word-limit isn’t just for the uninformed public, it’s for serious people trying to coordinate. The public can only coordinate around 5-word things. Serious people trying to be informed still have to ingest lots of information and form detailed models but those models are still going to have major bits that are compressed, out of pieces that end up being about five words. And this is a major part of why many people are confused about Effective Altruism and how to do it right in the first place.
If that’s your outlook, it seems pointless to write anything longer than five words on any topic other than how to fix this problem.
I agree with the general urgency of the problem, although I think the frame of your comment is somewhat off. This problem seems… very information-theoretically-entrenched. I have some sense that you think of it as solvable in a way that it’s fundamentally not actually solvable, just improvable, like you’re trying to build a perpetual motion machine instead of a more efficient engine. There is only so much information people can process.
(This is based entirely off of reading between the lines of comments you’ve made, and I’m not confident what your outlook actually is here, and apologies for the armchair psychologizing).
I think you can make progress on it, which would look something like:
0) make sure people are aware of the problem
1) building better infrastructure (social or technological), probably could be grouped into a few goals:
nudge readers towards certain behavior
nudge writers towards certain behavior
provide tools that amplify readers capabilities
provide tools that amplify writer’s capabilities
2) meanwhile, as a writer, make sure that the concepts you create for the public discourse are optimized for the right kind of compression. Some ideas compress better than others. (I have thought about the details of this
This *is* my outlook, and that yes this that both I, as well as you and Jessica, should probably be taking some kind of action that takes this outlook strategically seriously if you aren’t already.
Distillation Technology
A major goal I have for LessWrong, which the team has talked about a lot, is improving distillation technology. It’s not what we’re currently working on because, well, there are *multiple* top priorities that all seem pretty urgent (and all seem like pieces of the same puzzle). But I think Distillation Tech is the sort of thing most likely to meaningfully improve the situation.
Right now the default mode people interact with LessWrong and many other blogging platforms is “write up a thing, post it, maybe change a few things in response to feedback.” But for ideas that are actually going to become building blocks of the intellectual commons, you need to continuously invest in improving them.
Arbital tried to do this, and it failed because the problem is hard in weird ways, many of them somewhat hard to anticipate.
http://distill.pub tackles a piece of this but not in a way that seems especially scalable.
Scott Alexander’s short story Ars Longa Vita Brevis is a fictional account of what seems necessary to me.
I do hope that by the end of this year the LW team will have made some concrete progress on this. I think it is plausibly a mistake that we haven’t focused on it already – we discussed switching gears towards it at our last retreat but it seemed to make more sense to finish Open Questions.
Trying to nudge others seems like an attempt to route around the problem rather than solve it. It seems like you tried pretty hard to integrate the substantive points in my “Effective Altruism is self-recommending” post, and even with pretty extensive active engagement, your estimate is that you only retained a very superficial summary. I don’t see how any compression tech for communication at scale can compete with what an engaged reader like you should be able to do for themselves while taking that kind of initiative.
We know this problem has been solved in the past in some domains—you can’t do a thing like the Apollo project or build working hospitals where cardiovascular surgery is regularly successful based on a series of atomic five-word commands; some sort of recursive general grammar is required, and at least some of the participants need to share detailed models.
One way this could be compatible with your observation is that people have somewhat recently gotten worse at this sort of skill; another is that credit-assignment is an unusually difficult domain to do this in. My recent blog posts have argued that at least the latter is true.
In the former case (lost literacy), we should be able to reconstruct older modes of coordination. In the latter (politics has always been hard to think clearly about), we should at least internally be able to learn from each other by learning to apply cognitive architectures we use in domains where we find this sort of thing comparatively easy.
I think I may have communicatedly somewhat poorly by phrasing this in terms of 5 words, rather than 5 chunks, and will try to write a new post sometime that presents a more formal theory of what’s going on.
I mentioned in the comments of the previous post:
And:
I do in fact expect that the Apollo project worked via finding ways to cache things into manageable chunks, even for the people who kept the whole project in their head.
Chunks can be nested, and chunks can include subtle neural-network-weights that are part of your background experience and aren’t quite explicit knowledge. It can be very hard to communicate subtle nuances as part of the chunks if you don’t have excess to high volume and preferably in-person communication.
I’d be interested in figuring out how to operationalize this as a bet and check how the project actually worked. What I have heard (epistemic status: heard it from some guy on the internet) is that actually, most people on the project did not have all the pieces in their head, and the only people who did were the pilots.
My guess is that the pilots had a model of how to *use* and *repair* all the pieces of the ship, but couldn’t have built it themselves.
My guess it that “the people who actually designed and assembled the thing” had a model of how all the pieces fit together, but not as a deep a model of how and when to use it, and may have only understood the inputs and outputs of each piece.
And meanwhile, while I’m not quite sure how to operationalize the bet, I would bet maybe $50 that (conditional on us finding a good operationalization), that the number of people who had the full model or anything like it was quite small. (“You Have About Five Words” doesn’t claim you can’t have more than 5 words of nuance, it claims that you can’t coordinate large groups of people that depend on more than 5 words of nuance. I bet there were less than 100 people and probably closer to 10 who had anything like a full model of everything going on)
I think I’m unclear on how this constrains anticipations, and in particular it seems like there’s substantial ambiguity as to what claim you’re making, such that it could be any of these:
You can’t communicate recursive structures or models with more than five total chunks via mass media such as writing.
You can’t get humans to act (or in particular to take initiative) based on such models, so you’re limited to direct commands when coordinating actions.
There exist such people, but they’re very few and stretched between very different projects and there’s nothing we can do about that.
??? Something else ???
I think there are two different anticipation-constraining-claims, similar but not quite what you said there:
Working Memory Learning Hypothesis – people can learn complex or recursive concepts, but each chunk that they learn cannot be composed of more than 7 other chunks. You can learn a 49 chunk concept but first must distill it into seven 7-chunk-concepts, learn each one, and then combine them together.
Coordination Nuance Hypothesis – there are limits to how nuanced a model you can coordinate around, at various scales of coordination. I’m not sure precisely what the limits are, but it seems quite clear that the more people you are coordinating the harder it is to get them to share a nuanced model or strategy. It’s easier to have a nuanced strategy with 10 people than 100, 1000, or 10,000.
I’m less confident of the Working Memory hypothesis (it’s an armchair inside view based on my understanding of how working memory works)
I’m fairly confident in the Coordination Nuance Hypothesis, which is based on observations about how people actually seem to coordinate at various scales and how much nuance they seem to preserve.
In both cases, there are tools available to improve your ability to learn (as an individual), disseminate information (as a communicator), and keep people organized (as a leader). But none of the tools changed the fundamental equation, just the terms.
Anticipation Constraints:
The anticipation-constraint of the WMLH is “if you try to learn a concept that requires more than 7 chunks, you will fail. If a concept requires 12 chunks, you will not successfully learn it (or will learn a simplified bastardization of it) until you find a way to compress the 12 chunks into 7. If you have to do this yourself, it will take longer than if an educator has optimized it for you in advance.”
The anticipation constraint of the CNH is that if you try to coordinate with 100 people of a given level of intelligence, the shared complexity of the plan that you are enacting will be lower than the complexity of the plan you could enact with 10 people. If you try to implement a more complex plan or orient around a more complex model, your organization will make mistakes due to distorted simplifications of the plan. And this gets worse as your organizations scales.
CNH is still ambiguous between “nuanced plan” and “nuanced model” here, and those seem extremely different to me.
I agree they are different but think it is the case that with a larger group you have a harder time with either of them, for roughly the same reasons at roughly the same rate of increased difficulty.
The Working Memory Hypothesis says the Bell Labs is useful, in part, because whenever you need to combine multiple interdisciplinary concepts that are each complicated to invent a new concept…
instead of having to read a textbook that explains it one-particular-way (and, if it’s not your field, you’d need to get up to speed on the entire field in order to have any context at all) you can just walk down the hall and ask the guy who invented the concept “how does this work” and have them explain it to you multiple times until they find a way to compress it down into a 7 chunks, optimized for your current level of understanding.
A slightly more accurate anticipation of the CNH is:
people need to spend time learning a thing in order to coordinate around it. At the very least, the more time you need to spend getting people up to speed on a model, the less time they have to actually act on that model
people have idiosyncratic learning styles, and are going to misinterpret some bits of your plan, and you won’t know in advance which ones. Dealing with this requires individual attention, noticing their mistakes and correcting them. Middle managers (and middle “educators” can help to alleviate this, but every link in the chain reduces your control over what message gets distributed. If you need 10,000 people to all understand and act on the same plan/model, it needs to be simple or robust enough to survive 10,000 people misinterpreting it in slightly different ways
This gets even worse if you need to change your plan over time in response to new information, since now people are getting it confused with the old plan, or they don’t agree with the new plan because they signed up for the old plan, and then you have to Do Politics to get them on board with the new plan.
At the very least, if you’ve coordinated perfectly, each time you change your plan you need to shift from “focusing on execution” to “focusing on getting people up to speed on the new model.”
Make spaced repetition cards?
The way that I’d actually do this, and plan to do this (in line with Benquo’s reply to you), is to repackage the concept into something that I understand more deeply and which I expect to unpack more easily in the future.
Part of this requires me to do some work for myself (no amount of good authorship can replace putting at least some work into truly understanding something)
Part of this has to do with me having my own framework (rooted in Robust Agency among other things) which is different from Benquo’s framework, and Ben’s personal experience playing werewolf.
But a lot of my criticism of the current frame is that it naturally suggest compacting the model in the wrong way. (to be clear, I think this is fine for a post that represents a low-friction strategy to post your thoughts and conversations as they form, without stressing too much about optimizing pedagogy. I’m glad Ben posted the Villager/Werewolf post. But I think the presentation makes it harder to learn than it needs to be, and is particularly ripe for being misinterpreted in a way that benefits rather than harms werewolves, and if it’s going to be coming up in conversation a lot I think it’d be worth investing time in optimizing it better)
That seems like the sort of hack that lets you pass a test, not the sort of thing that makes knowledge truly a part of you. To achieve the latter, you have to bump it up against your anticipations, and constantly check to see not only whether the argument makes sense to you, but whether you understand it well enough to generate it in novel cases that don’t look like the one you’re currently concerned with.
I think it’s possible to use in a “mindful” way even if most people are doing it wrong? The system reminding you what you read n days ago gives you a chance to connect it to the real world today when you otherwise would have forgotten.
Holden Karnofsky explicitly disclaimed the “independence via multiple funders” consideration as not one that motivated the partial funding recommendation.