Evaluating GiveWell as a startup idea based on Paul Graham's philosophy

Effective altruism is a growing movement, and a number of organizations (mostly foundations and nonprofits) have been started in the domain. One of the very first of these organizations, and arguably the most successful and influential, has been charity evaluator GiveWell. In this blog post, I examine the early history of GiveWell and see what factors in this early history helped foster its success.

My main information source is GiveWell’s original business plan (PDF, 86 pages). I’ll simply refer to this as the “GiveWell business plan” later in the post and will not link to the source each time. If you’re interested in what the GiveWell website looked like at the time, you can browse the website as of early May 2007 here.

To provide more context to GiveWell’s business plan, I will look at it in light of Paul Graham’s pathbreaking article How to Get Startup Ideas. The advice here is targeted at early stage startups. GiveWell doesn’t quite fit the “for-profit startup” mold, but GiveWell in its early stages was a nonprofit startup of sorts. Thus, it would be illustrative to see just how closely GiveWell’s choices were in line with Paul Graham’s advice.

There’s one obvious way that this analysis is flawed and inconclusive: I do not systematically compare GiveWell with other organizations. There is no “control group” and no possibility of isolating individual aspects that predicted success. I intend to write additional posts later on the origins of other effective altruist organizations, after which a more fruitful comparison can be attempted. I think it’s still useful to start with one organization and understand it thoroughly. But keep this limitation in mind before drawing any firm conclusions, or believing that I have drawn firm conclusions.

The idea: working on a real problem that one faces at a personal level, is acutely familiar with, is of deep interest to a (small) set of people right now, and could eventually be of interest to many people

Graham writes (emphasis mine):

The very best startup ideas tend to have three things in common: they’re something the founders themselves want, that they themselves can build, and that few others realize are worth doing. Microsoft, Apple, Yahoo, Google, and Facebook all began this way.

Why is it so important to work on a problem you have? Among other things, it ensures the problem really exists. It sounds obvious to say you should only work on problems that exist. And yet by far the most common mistake startups make is to solve problems no one has.

[...]

When a startup launches, there have to be at least some users who really need what they’re making—not just people who could see themselves using it one day, but who want it urgently. Usually this initial group of users is small, for the simple reason that if there were something that large numbers of people urgently needed and that could be built with the amount of effort a startup usually puts into a version one, it would probably already exist. Which means you have to compromise on one dimension: you can either build something a large number of people want a small amount, or something a small number of people want a large amount. Choose the latter. Not all ideas of that type are good startup ideas, but nearly all good startup ideas are of that type.

Imagine a graph whose x axis represents all the people who might want what you’re making and whose y axis represents how much they want it. If you invert the scale on the y axis, you can envision companies as holes. Google is an immense crater: hundreds of millions of people use it, and they need it a lot. A startup just starting out can’t expect to excavate that much volume. So you have two choices about the shape of hole you start with. You can either dig a hole that’s broad but shallow, or one that’s narrow and deep, like a well.

Made-up startup ideas are usually of the first type. Lots of people are mildly interested in a social network for pet owners.

Nearly all good startup ideas are of the second type. Microsoft was a well when they made Altair Basic. There were only a couple thousand Altair owners, but without this software they were programming in machine language. Thirty years later Facebook had the same shape. Their first site was exclusively for Harvard students, of which there are only a few thousand, but those few thousand users wanted it a lot.

When you have an idea for a startup, ask yourself: who wants this right now? Who wants this so much that they’ll use it even when it’s a crappy version one made by a two-person startup they’ve never heard of? If you can’t answer that, the idea is probably bad. [3]

You don’t need the narrowness of the well per se. It’s depth you need; you get narrowness as a byproduct of optimizing for depth (and speed). But you almost always do get it. In practice the link between depth and narrowness is so strong that it’s a good sign when you know that an idea will appeal strongly to a specific group or type of user.

But while demand shaped like a well is almost a necessary condition for a good startup idea, it’s not a sufficient one. If Mark Zuckerberg had built something that could only ever have appealed to Harvard students, it would not have been a good startup idea. Facebook was a good idea because it started with a small market there was a fast path out of. Colleges are similar enough that if you build a facebook that works at Harvard, it will work at any college. So you spread rapidly through all the colleges. Once you have all the college students, you get everyone else simply by letting them in.

GiveWell in its early history seems like a perfect example of this:

Real problem experienced personally: The problem of figuring out how and where to donate money was a personal problem that the founders experienced firsthand as customers, so they knew there was a demand for something like GiveWell.
Of deep interest to some people: The people who started GiveWell had a few friends who were in a similar situation: they wanted to know where best to donate money, but did not have enough resources to do a full-fledged investigation. The number of such people may have been small, but since these people were intending to donate money in the thousands of dollars, there were enough of them who had deep interest in GiveWell’s offerings.
Could eventually be of interest to many people: Norms around evidence and effectiveness could change gradually as more people started identifying as effective altruists. So, there was a plausible story for how GiveWell might eventually influence a large number of donors across the range from small donors to billionaires.

Quoting from the GiveWell business plan (pp. 3-7, footnotes removed; bold face in original):

GiveWell started with a simple question: where should I donate?

We wanted to give. We could afford to give. And we had no prior commitments to any particular charity; we were just looking for the channel through which our donations could help people (reduce suffering; increase opportunity) as much as possible.

The first step was to survey our options. We found that we had more than we could reasonably explore comprehensively. There are 2,625 public charities in the U.S. with annual budgets over $100 million, 88,812 with annual budgets over $1 million. Restricting ourselves to the areas of health, education (excluding universities), and human services, there are 480 with annual budgets over $100 million, 50,505 with annual budgets over $1 million.

We couldn’t explore them all, but we wanted to find as many as possible that fit our broad goal of helping people, and ask two simple questions: what they do with donors’ money, and what evidence exists that their activities help people?

Existing online donor resources, such as Charity Navigator, give only basic financial data and short, broad mission statements (provided by the charities and unedited). To the extent they provide metrics, they are generally based on extremely simplified, problematic assumptions, most notably the assumption that the less a charity spends on administrative expenses, the better. These resources could not begin to help us with our questions, and they weren’t even very useful in narrowing the field (for example, even if we assumed Charity Navigator’s metrics to be viable, there are 1,277 total charities with the highest possible rating, 562 in the areas of health, education and human services).

We scoured the Internet, but couldn’t find the answers to our questions either through charities’ own websites or through the foundations that fund them. It became clear to us that answering these questions was going to be a lot of work. We formed GiveWell as a formal commitment to doing this work, and to putting everything we found on a public website so other donors wouldn’t have to repeat what we did. Each of the eight of us chose a problem of interest (malaria, microfinance, diarrheal disease, etc.) – this was necessary in order to narrow our scope – and started to evaluate charities that addressed the problem.

[...]

We immediately found that there are enormous opportunities to help people, but no consensus whatsoever on how to do it best. [...]

Realizing that we were trying to make complex decisions, we called charities and questioned them thoroughly. We wanted to see what our money was literally being spent on, and for charities with multiple programs and regions of focus we wanted to know how much of their budget was devoted to each. We wanted to see statistics – or failing that, stories – about people
who’d benefited from these programs, so we could begin to figure out what charities were pursuing the best strategies. But when we pushed for these things, charities could not provide them.

They responded with surprise (telling us they rarely get questions as detailed as ours, even from multi-million dollar donors) and even suspicion (one executive from a large organization accused Holden of running a scam, though he wouldn’t explain what sort of scam can be run using information about a charity’s budget and activities). See Appendix A for details of these exchanges. What we saw led us to conclude that charities were neither accustomed to nor capable of answering our basic questions: what do you do, and what is the evidence that it works?

This is why we are starting the Clear Fund, the world’s first completely transparent charitable grantmaker. It’s not because we were looking for a venture to start; everyone involved with this project likes his/her current job. Rather, the Clear Fund comes simply from a need for a resource that doesn’t exist: an information source to help donors direct their money to where it will accomplish the most good.

We feel that the questions necessary to decide between charities aren’t being answered or, largely, asked. Foundations often focus on new projects and innovations, as opposed to scaling up proven ways of helping people; and even when they do evaluate the latter, they do not make what they find available to foster dialogue or help other donors (see Appendix D for more on this). Meanwhile, charities compete for individual contributions in many ways, from marketing campaigns to personal connections, but not through comparison of their answers to our two basic questions. Public scrutiny, transparency, and competition of charities’ actual abilities to improve the world is thus practically nonexistent. That makes us worry about the quality of their operations – as we would for any set of businesses that doesn’t compete on quality – and without good operations, a charity is just throwing money at a problem.

[...]

With money and persistence, we believe we can get the answers to our questions – or at least establish the extent to which different charities are capable of answering them. If we succeed, the tremendous amount of money available for solving the world’s problems will become better spent, and the world will reap enormous benefits. We believe our project will accomplish the following:
1. Help individual donors find the best charities to give to. [...]

2. Foster competition to find the best ways of improving the world. [...]

3. Foster global dialogue between everyone interested – both amateur and professional –
in the best tactics for improving the world. [...]

4. Increase engagement and participation in charitable causes. [...]

All of the benefits above fall under the same general principle. The Clear Fund will put a new focus on the strategies – as opposed to the funds – being used to attack the world’s problems.

How do you know if the idea is scalable? You just gotta be the right person

We already quoted above GiveWell’s reasons for believing that their idea could eventually influence a large volume of donations. But how could we know at the time whether their beliefs were reasonable? Graham writes (emphasis mine):

How do you tell whether there’s a path out of an idea? How do you tell whether something is the germ of a giant company, or just a niche product? Often you can’t. The founders of Airbnb didn’t realize at first how big a market they were tapping. Initially they had a much narrower idea. They were going to let hosts rent out space on their floors during conventions. They didn’t foresee the expansion of this idea; it forced itself upon them gradually. All they knew at first is that they were onto something. That’s probably as much as Bill Gates or Mark Zuckerberg knew at first.

Occasionally it’s obvious from the beginning when there’s a path out of the initial niche. And sometimes I can see a path that’s not immediately obvious; that’s one of our specialties at YC. But there are limits to how well this can be done, no matter how much experience you have. The most important thing to understand about paths out of the initial idea is the meta-fact that these are hard to see.

So if you can’t predict whether there’s a path out of an idea, how do you choose between ideas? The truth is disappointing but interesting: if you’re the right sort of person, you have the right sort of hunches. If you’re at the leading edge of a field that’s changing fast, when you have a hunch that something is worth doing, you’re more likely to be right.

How well does GiveWell fare in terms of the potential of the people involved? Were the people who founded GiveWell (specifically Holden Karnofsky and Elie Hassenfeld) the “right sort of person” to found GiveWell? It’s hard to give an honest answer that’s not clouded by information available in hindsight. But let’s try. On the one hand, neither of the co-founders had direct experience working with nonprofits. However, they had both worked in finance and the analytical skills they employed in the financial industry may have been helpful when they switched to analyzing evidence and organizations in the nonprofit sector (see the “Our qualifications” section of the GiveWell business plan). Arguably, this was more relevant to what they wanted to do with GiveWell than direct experience with the nonprofit world. Overall, it’s hard to say (without the benefits of hindsight or inside information about the founders) that the founders were uniquely positioned, but the outside view indicators seem generally favorable.

Post facto, there seems to be some evidence that GiveWell’s founders exhibited good aesthetic discernment. But this is based on GiveWell’s success, so invoking that as a reason is a circular argument.

Schlep blindness?

In a different essay titled Schlep Blindness, Graham writes:

There are great startup ideas lying around unexploited right under our noses. One reason we don’t see them is a phenomenon I call schlep blindness. Schlep was originally a Yiddish word but has passed into general use in the US. It means a tedious, unpleasant task.

[...]

One of the many things we do at Y Combinator is teach hackers about the inevitability of schleps. No, you can’t start a startup by just writing code. I remember going through this realization myself. There was a point in 1995 when I was still trying to convince myself I could start a company by just writing code. But I soon learned from experience that schleps are not merely inevitable, but pretty much what business consists of. A company is defined by the schleps it will undertake. And schleps should be dealt with the same way you’d deal with a cold swimming pool: just jump in. Which is not to say you should seek out unpleasant work per se, but that you should never shrink from it if it’s on the path to something great.

[...]

How do you overcome schlep blindness? Frankly, the most valuable antidote to schlep blindness is probably ignorance. Most successful founders would probably say that if they’d known when they were starting their company about the obstacles they’d have to overcome, they might never have started it. Maybe that’s one reason the most successful startups of all so often have young founders.

In practice the founders grow with the problems. But no one seems able to foresee that, not even older, more experienced founders. So the reason younger founders have an advantage is that they make two mistakes that cancel each other out. They don’t know how much they can grow, but they also don’t know how much they’ll need to. Older founders only make the first mistake.

It could be argued that schlep blindness was the reason nobody else had started GiveWell before GiveWell. Most people weren’t even thinking of doing something like this because the idea seemed like so much work that nobody went near it. Why then did GiveWell’s founders select the idea? There’s no evidence to suggest that Graham’s “ignorance” remedy was the reason. Rather, the GiveWell business plan explicitly embraces complexity. In fact, one of their early section titles is Big Problems with Complex Solutions. It seems like the GiveWell founders found challenge more exciting than deterring. Lack of intimate knowledge with the nonprofit sector might have been a factor, but it probably wasn’t a driving one.

Competition

Graham writes:

Because a good idea should seem obvious, when you have one you’ll tend to feel that you’re late. Don’t let that deter you. Worrying that you’re late is one of the signs of a good idea. Ten minutes of searching the web will usually settle the question. Even if you find someone else working on the same thing, you’re probably not too late. It’s exceptionally rare for startups to be killed by competitors—so rare that you can almost discount the possibility. So unless you discover a competitor with the sort of lock-in that would prevent users from choosing you, don’t discard the idea.

If you’re uncertain, ask users. The question of whether you’re too late is subsumed by the question of whether anyone urgently needs what you plan to make. If you have something that no competitor does and that some subset of users urgently need, you have a beachhead.

[...]

You don’t need to worry about entering a “crowded market” so long as you have a thesis about what everyone else in it is overlooking. In fact that’s a very promising starting point. Google was that type of idea. Your thesis has to be more precise than “we’re going to make an x that doesn’t suck” though. You have to be able to phrase it in terms of something the incumbents are overlooking. Best of all is when you can say that they didn’t have the courage of their convictions, and that your plan is what they’d have done if they’d followed through on their own insights. Google was that type of idea too. The search engines that preceded them shied away from the most radical implications of what they were doing—particularly that the better a job they did, the faster users would leave.

A crowded market is actually a good sign, because it means both that there’s demand and that none of the existing solutions are good enough. A startup can’t hope to enter a market that’s obviously big and yet in which they have no competitors. So any startup that succeeds is either going to be entering a market with existing competitors, but armed with some secret weapon that will get them all the users (like Google), or entering a market that looks small but which will turn out to be big (like Microsoft).

Did GiveWell enter a crowded market? As Graham suggests above, it depends heavily on how you define the market. Charity Navigator existed at the time, and GiveWell and Charity Navigator compete to serve certain donor needs. But they are also sufficiently different. Here’s what GiveWell said about Charity Navigator in the GiveWell business plan:

Existing online donor resources, such as Charity Navigator, give only basic financial data and short, broad mission statements (provided by the charities and unedited). To the extent they provide metrics, they are generally based on extremely simplified, problematic assumptions, most notably the assumption that the less a charity spends on administrative expenses, the better. These resources could not begin to help us with our questions, and they weren’t even very useful in narrowing the field (for example, even if we assumed Charity Navigator’s metrics to be viable, there are 1,277 total charities with the highest possible rating, 562 in the areas of health, education and human services)

In other words, GiveWell did enter a market with existing players, indicating that there was a need for things in the broad domain that GiveWell was offering. At the same time, what GiveWell offered was sufficiently different that it was not bogged down by the competition.

Incidentally, in recent times, people from Charity Navigator have been critical of GiveWell and other “effective altruism” proponents. Their critique has itself come for some criticism, and some people have argued that this may be a response to GiveWell’s growth leading to it moving the same order of magnitude of money as Charity Navigator (see the discussion here for more). Indeed, in 2013, GiveWell surpassed Charity Navigator in money moved through the website, though we don’t have clear evidence of whether GiveWell is cutting into Charity Navigator’s growth.

Other precursors (of sorts) to GiveWell, mentioned by William MacAskill in a Facebook comment, are the Poverty Action Lab, Copenhagen Consensus.

How prescient was GiveWell?

With the benefit of hindsight, how impressive do we find GiveWell’s early plans in predicting its later trajectory? Note that prescience in predicting the later trajectory could also be interpreted as rigidity of plan and unwillingness to change. But since GiveWell appears to have been quite a success, there is a prior in favor of prescience being good (what I mean is that if GiveWell had failed, the fact that they predicted all the things they’d do would be the opposite of impressive, but given their success, the fact that they predicted things in advance also indicates that they chose good strategy from the outset).

Note that I’m certainly not claiming that a startup’s failure to predict the future should be a big strike against it. As long as the organization can adapt to and learn from new information, it’s fine. But of course, getting more things right from the start is better to the extent it’s feasible.

By and large, both the vision and the specific goals outlined in the plan were quite prescient. I noted the following differences between the plan then and the reality as it transpired:

In the plan, GiveWell said it would try to identify top charities in a few select areas (they listed seven areas) and refrain from comparing very different domains. Over the years, they have moved more in the direction of directly comparing different domains and offering a few top charities culled across all domains. Even though they seem to have been off in their plan, they were directionally correct compared to what existed. They were already consolidating different causes within the same broad category. For instance, they write (GiveWell business plan, p. 21):

A charity that focuses on fighting malaria and a charity that focuses on fighting tuberculosis are largely aiming for the same end goal – preventing death – and if one were clearly better at preventing death than the other, it would be reasonable to declare it a better use of funds. By contrast, a charity that focuses on creating economic opportunity has a fundamentally different end goal. It may be theoretically possible to put jobs created and lives saved in the same terms (and there have been some attempts to create metrics that do so), but ultimately different donors are going to have very different perspectives on whether it’s more worthwhile to create a certain number of jobs or prevent a certain number of deaths.
GiveWell doesn’t predict clearly enough that it will evolve into a more “foundation”-like entity. Note that at the time of the business plan, they were envisionining themselves as deriving their negotiating power with nonprofits through their role as grantmakers. They then transformed into deriving their power largely from their role as recommenders of top charities. Then, around 2012, following the collaboration with Good Ventures, they switched back to grantmaker mode, but in a far grander way than they’d originally envisaged.
At the time of the GiveWell business plan, they see their main source of money moved being small donors. In recent years, as they moved to more “foundation”-like behavior, they seem to have started shifting attention to influencing the giving decisions of larger donors. This might be purely due to the unpredictable fact that they joined hands with the Good Ventures foundation, rather than due to any systemic or predictable reasons. It remains to be seen whether they influence more donations by very large donors in the future. Another aspect of this is that GiveWell’s original business plan was more ambitious about influencing the large number of small donors out there than (I think) GiveWell is now.
GiveWell seems to have moved away from a focus on examining individual charities to understanding the landscape sufficiently well to directly identify the best opportunities, and then to comparing broad causes. The GiveWell business plan, on the other hand, repeatedly talked about “pitting charities against each other” (p. 11) as their main focal activity. In recent years, however, GiveWell has started stepping back and concentrating more on using their big picture understanding of the realm to more efficiently identify the very best opportunities rather than evaluating all relevant charities and causes. This is reflected in their conversation notes as well as the GiveWell Labs initiative. After creating GiveWell Labs, they have shifted more in the direction of thinking at the level of causes rather than individual interventions.

The role of other factors in GiveWell’s success

Was GiveWell destined to succeed, or did it get lucky? I believe a mix of both: GiveWell was bound to succeed in some measure, but a number of chance factors played a role in its achieving success to its current level. A recent blog post by GiveWell titled Our work on outreach contains some relevant evidence. The one single person who may have been key to GiveWell’s success is the ethicist and philosopher Peter Singer. Singer is a passionate advocate of the idea that people are morally obligated to donate money to help the world’s poorest people. Singer played a major role in GiveWell’s success in the following ways:

Singer both encouraged people to give and directed people interested in giving to GiveWell’s website when they asked him where they should give.
Singer was an inspiration for many effective giving organizations. He is credited as an inspiration by Oxford ethicist Toby Ord and his wife physician Bernadette Young, who together started Giving What We Can, a society promoting effective giving. Giving What We Can used GiveWell’s research for its own recommendations and pointed people to the website. In addition, Singer’s book The Life You Can Save also inspired the creation of the eponymous organization. Giving What We Can was a starting point for related organizations in the nascent effective altruism movement, including 80000 Hours, the umbrella group The Centre for Effective Altruism, and many other resources.
Cari Tuna and her husband (and Facebook co-founder) Dustin Moskovitz read about GiveWell in The Life You Can Save by Peter Singer around the same time they met Holden through a mutual friend. Good Ventures, the foundation set up by Tuna and Moskovitz has donated several million dollars to GiveWell’s recommended charities (over 9 million USD in 2013) and the organizations have collaborated somewhat. More in this blog post by Cari Tuna.

The connection of GiveWell to the LessWrong community might also have been important, though less so than Peter Singer. It could have been due to the efforts of a few people interested in GiveWell who discussed it on LessWrong. Jonah Sinick’s LessWrong posts about GiveWell (mentioned in GiveWell’s post about their work on outreach) are an example (full disclosure: Jonah Sinick is collaborating with me on Cognito Mentoring). Note that although only about 3% of donations made through GiveWell are explicitly attributable to LessWrong, GiveWell has received a lot of intellectual engagement from the LessWrong community and other organizations and individuals connected with the community.

How should the above considerations modify our view of GiveWell’s success? I think the key thing GiveWell did correctly was become a canonical go-to reference for where to direct donors on making good giving decisions. By staking out that space early on, they were able to capitalize on Peter Singer. Also, it’s not just GiveWell that benefited from Peter Singer — we can also argue that Singer’s arguments were made more effective by the existence of GiveWell. The first line of counterargument to Singer’s claim is that most charities aren’t cost-effective. Singer’s being able to point to a resource to help identify good charities make people take his argument more seriously.

I think that GiveWell’s success at making itself the canonical source was more important than the specifics of their research. But the specifics may have been important in convincing a sufficiently large critical mass of influential people to recommend GiveWell as a canonical source, so the factors are hard to disentangle.

Would something like GiveWell have existed if GiveWell hadn’t existed? How would the effective altruism movement be different?

These questions are difficult to explore, and discussing them would take us too far afield. This post on the Effective Altruists Facebook thread offers an interesting discussion. The upshot is that, although Giving What We Can was started two years after GiveWell, people involved with its early history say that the core ideas of looking at cost-effectiveness and recommending the very best places to donate money was mooted before its formal inception, some time around 2006 (when GiveWell had not been formally created). At the time, the people involved were unaware of GiveWell. William MacAskill says that GWWC may have done more work on the cost-effectiveness side if GiveWell wasn’t already doing it.

I ran this post by Jonah Sinick and also emailed a draft to the GiveWell staff. I implemented some of their suggestions, and am grateful to them for taking the time to comment on my draft. Any responsibility for errors, omissions, and misrepresentations is solely mine.

Evaluating GiveWell as a startup idea based on Paul Graham’s philosophy