Why CFAR? The view from 2015

PeteMichaud23 Dec 2015 22:46 UTC
73 points
63 comments17 min readLW linkArchive

Follow-up to: 2013 and 2014.

In this post, we:

We are in the middle of our matching fundraiser; so if you’ve been considering donating to CFAR this year, now is an unusually good time.

CFAR’s mission, and why that mission matters today

CFAR’s mission is to help people develop the abilities that let them meaningfully assist with the world’s most important problems, by improving their ability to arrive at accurate beliefs, act effectively in the real world, and sustainably care about that world.

We know this is an audacious thing to try—especially the “ability to form accurate beliefs” part—but it seems to us that such attempts work sometimes anyhow. Eliezer’s Sequences seem to offer principled improvements to some aspects of some peoples’ world-modeling skill (via synthesizing much recent cognitive science, probability theory, etc.); this seems to us to be a useful point from which to build.

The fact remains that we do not yet have the talent necessary to win—to see the world’s problems clearly, plot strategies that have a shot at working, update when those strategies don’t work, and plan effectively around unknowns. To avoid any great filters that may be lurking, solve global and even astronomical challenges, and create a flourishing world for all.

Arguably, people of the caliber we’re shooting for don’t exist yet, but even if they do, it seems clear that we don’t have enough of them to have enough of a guarantee of actually succeeding.

So, audacious or not, this is a task that needs to be done, and CFAR is our attempt to do it. If we can widen the bottleneck on thinking better and doing more, we’re increasing the odds of a better future regardless of what the important problems turn out to be.

Our progress to date

By the end of 2014, CFAR had created workshops that participants liked a lot and which evidence suggests had concrete benefits for them. However, our mission remains to impact the world. The question became whether we could adapt our workshops into something that had the potential for large impact.

Our central goal for 2015 was therefore to create what we called a “minimum strategic product”—a product that, as we put it last year, would “more directly justify CFAR’s claim to be an effective altruist project” by demonstrating that we could sometimes improve peoples’ thinking skill, competence, and/​or do-gooding to the point where they were able to engage in direct work on a key talent-limited task.

Running the MIRI Summer Fellows program gave the opportunity we’d sought to try our hand at creating such direct impact. Our plan was to test and develop our curriculum and training methods through running a training program that would not only improve people’s ability to think about some of the big questions, but also do so in a fashion that could lead to immediate progress.

How did we do? Here’s what Nate Soares, MIRI’s Executive Director, had to say:

“MSFP was a resounding success: many participants gained new skills relevant to alignment research, and the program led directly to multiple MIRI hires. The world needs more talented people focusing on big important problems, and CFAR has figured out how to develop those sorts of talents in practice.”

While working to help create AI alignment researchers, we also found that this focus on how to become a better scientist led us into more fruitful territory for improving our understanding of the art. (If you’re curious, you can see a highly incoherent version of some of the skills we tried to get across in this working document. Read below for more details about art creation, and our plans to expand on more targeted training programs.)

Last year’s “goals for 2015”

We hit some of our concrete goals for 2015 and got distracted from others (partly, perils of unanticipated opportunities :-/​).

We created a provisional metric for participants’ before-and-after strategic usefulness, hitting the first goal; we started tracking that metric, hitting the second goal. Then we found that the metric was too unwieldy and too interpersonally tricky to regularly use on participants, making this “hitting” of our “goals” somewhat less useful than we had hoped. (On the upside, we learned something about how not to build metrics. :-/​)

We then got the opportunity to run MIRI Summer Fellows, as noted above… and mostly dropped our previously declared goals to pull off the program, partly because our goals had been meant as a concretization of “can we train people who matter for the world”, and the Summer Fellows program seemed like a better concretization of the same. (The program required a lot of new curriculum beyond what we had before, and a lot of skill development on the part of our teaching staff; and even so, and despite Nate’s calling it a “resounding success”, we had a feeling of leaving a lot of opportunity on the table; opportunity we intend to pick up in our second MIRI Summer Fellows program this coming summer).

From the original “concrete goals” list: goal three was a bit wishy-washy but was probably done. Goals four and five we did not even measure to see if we hit. We should and will measure this, and will let you know when we do; it seems good that we opportunistically put our all into the summer fellows program (and okay to de-emphasize old goals in pursuit of that), but good also to then follow it up for the sake of feedback loops and honesty.

Organizational capital

2015 was the year in which we finally managed to stop wearing all the hats thanks to a huge increase in organizational capital. At the start of 2015 workshops were stressful for staff. Between workshops, our workdays were cluttered with a disproportionate amount of attention spent on logistics, alumni followups, and tasks like accounting.

This stress and clutter was part of what was preventing us from seeing what we were doing, and figuring out how to actually contribute to the world; smoothing out the wrinkles in our day-to-day workflow was (we think) a major stepping stone toward discovering our minimum strategic product.

That’s why we spent a lot of time and effort this year on streamlining operations and increasing specialization so that we could both free the capacity to focus on developing the art and create the capacity to scale our workshops. We systematized tasks like accounting and venue searches, and began using alumni volunteers as follow up mentors to supplement our newly-created post-workshop email exercises and online hangouts. These efforts culminated in two new hires—Pete Michaud and Duncan Sabien—and a reorganization of CFAR into two subteams, Core (focused on operations) and Labs (focused on research).

For a complete overview of what we intend to accomplish in 2016, see Ambitions for 2016 below.

Some snapshots from our rationality development

There is the process by which we improve a workshop, and there is the process by which we improve our understanding of how rationality works at its core. The two processes don’t always help one another, but this year they did.

How we got there:

  • As it turns out, attempting to create AI risk scientists (as opposed to boosting the scientist-nature of everyday people) put a subtle but very different spin on the teaching of Sequences-style epistemic rationality. It helped that the researchers were themselves trying to model mind-like processes and that they stubbornly insisted on building related models of what the heck we were trying to convey.

  • MIRI Summer Fellows was also a project we could just actually see mattered, and there’s nothing quite like actual stakes when it comes to creating a sense of drive and purpose, and being willing to update.

  • Improving organizational capital created a positive feedback loop. Working to make our workshops “crisp”—to clean up the methods and metaphors that weren’t pulling their weight—helped make more of what we knew more visible.

Here are some brief highlights of the new Art of Rationality that we’re currently seeing:

  • One pillar, not three. CFAR has long talked about wanting to boost three distinct things in our participants (competence, epistemic rationality, and do-gooding). But we’ve had the strong sense that there were ways to strengthen all three through the practice of a single, unified art of “applied rationality” (for instance, a deep understanding of reductionism seems to help with all three). Recently, we’ve gotten better at articulating how this link works. For example:

  • Double Crux is a structured format for collaboratively finding the truth in cases where two people disagree. Instead of non-interactively offering pieces of their respective platforms, people jointly seek the actual question at the crux of the disagreement—the root uncertainty that has the potential to affect both of their beliefs. We introduced this as an epistemic rationality technique, and used in in this way at e.g. EA Global, where people argued about cause prioritization; it then made its way also into our material on competence and on how to sustainably care deeply about the world. (See the next two bullet points.)

  • Competence as “deep/​internal epistemic rationality.” If I am frequently late to appointments and “don’t want to be,” one can frame this as stemming from an inaccurate anticipation somewhere in my mind—perhaps I mis-anticipate whether my actions will make me late, or perhaps I disagree with myself as to whether lateness in fact harms my goals. Either way, it can be helpful (in our experience) to “internally double crux” the apparent disagreement (i.e., to play the double crux game between two different models within my own head, working until I have both a better model and a better actual outcome). More generally, we are increasingly making headway on “competence” or “instrumental rationality” problems via techniques aimed at integrating accurate beliefs into all parts of one’s psyche.

  • Do-gooding and epistemic rationality. “Do-gooding” would seem to be a goal that some have and others don’t, and it would seem odd to try to shift goals by learning epistemic rationality. But it seems to many of us (informally, anecdotally) that there is a kind of “deep epistemic rationality” that doesn’t change one’s goals, but does help one make actual contact with what is at stake in the world, and with the parts of one’s psyche that already care about those stakes… and this can sometimes help in practice to build deep, sustainable caring. The idea is again to e.g. notice a part of you that thinks the world matters, and a part of you that is afraid to look in that direction, and help these parts trade model-pieces and update back and forth (double crux, again). For an early attempt to articulate pieces of this “art of connecting to deep caring”, see Val’s recent post on grieving.

  • Teaching the synthesis. Our pre-2015 workshops were made of techniques, which was like sounding out words a letter at a time (C-A-T…C…Ca…Cat!). After years of trying to use these techniques to point at the deeper skill (Cat! Hat! Antidisestablishmentarianism!), we’ve finally found framings and explanations (like this one) that actually bridge the gap. Those framings, plus an explicit emphasis on synthesis and the addition of peer-to-peer tutoring, have successfully transformed the techniques into stepping stones toward the actual art. (The techniques are now stuffed into the first two days; the synthesis, and the rhythms of using applied rationality in practice, now occupy the second half of the workshop and give people a better sense of the lived feeling of the art. We think.)

This is the beginning of work that we’re poised to expand and improve in the coming year via our new Labs group.

Financial Retrospective for 2015

General overview

Our net cashflow for the year is about $14k positive so far, though without any further revenue we expect to be around $30k negative by the end of December 2015, as most of our large expenses (rent, payroll, etc.) occur at the end of the month. Note that this includes donation revenue from last year’s winter fundraiser.

Our basic monthly operating costs for 2015 have averaged $40k, although the average after September went up to $44k due to changing and slightly expanding our team. This is the number we use to determine burn rate.

$30k of this was payroll in the last quarter, and the rest was split amongst rent and utilities, parking, office supplies, meals, and miscellaneous. Many of these resources are used for in-office events like test sessions, Less Wrong meetups, and rationality training sessions; each staff member has a different and often changing split of percentage time working on operations, curriculum design, teaching, data analysis, etc. That’s why giving a good number for monthly overhead is tricky and unreliable. But to give it a go, it looks like roughly a third of monthly expenses is for organization maintenance.

A bit over half of the revenue covering this came from donations. The rest came from net revenue from our standard introductory workshops plus MIRI’s payment for our running MSFP. (More details below.)

Main workshops

Our standard introductory workshops serve several important purposes for us. One of them is that we hope to develop useful products that simultaneously support our mission and also make CFAR less fiscally dependent on donations.

We ran four of these workshops (three in the Bay Area and one in Boston). They varied widely in both cost and revenue due to travel, testing out new venues, changing the number of participants per workshop, and several other factors. All told, ignoring costs of staff time (as that’s factored into the above burn rate), CFAR main workshops took in a total of ~$123k net revenue (i.e., revenue exceeding cost), or an average of ~$31k net revenue per workshop. Compared to last year, this is down ~$107k total, but up ~$6k per workshop. This is due to us choosing to run less than half as many workshops so as to focus on:

  1. Making the workshops more efficient

  2. Running other programs equally well

  3. Setting up better systems both for workshops and for research

In addition, we’ve continued a trend from last year: we’ve decreased the per-workshop cost in staff time, partly through streamlined curriculum and improved systems and partly through training volunteers to conduct follow-ups, freeing up our core staff to build new programs and spend more time developing advanced rationality theory and instruction. (The volunteer training also does double-duty: the original impetus for doing it was wanting to help alumni benefit from the “learn by teaching” phenomenon, so we are both freeing up staff time and also using this to help deepen alums’ skill with rationality.)

Alumni events

CFAR typically goes into alumni events (workshops and the annual reunion) with the assumption that we’re taking on a cost. We view these as opportunities to explore potentially new areas of rationality and also as ways of encouraging and supporting the CFAR alumni community in their development as rationalists and as a community. It has generally been our policy that we don’t charge for alumni events, but instead we let our alumni know what the per capita cost comes to and ask them to consider donating to compensate.

We track the donations that are in support of these events separately from our standard general donations. As a result, we can pretty clearly see how much each event cost us on beyond the associated donations. That is, we can see net cost. In that spirit, here is what we “paid” on net for each of our alumni programs, ignoring staff time:

  • For net zero cost (participants covered meals), we ran a one-day workshop out of the CFAR office on applying Sequences-style thinking to one’s daily life and to hard problems like exrisk, as part of our preparation for MSFP.

  • For net zero cost (again, participants covered meals), we ran a 2-day workshop out of the CFAR office on applying Sequences-style thinking to AI risk analysis, also as part of our preparation for MSFP.

  • For net zero cost (participants donated enough to cover venue and meals), we ran a “Hamming” workshop in Boston, to explore what techniques are needed to identify and dive into the most important problems one is currently facing (at work, in one’s personal life, as an altruist, or in whatever other domain).

  • For ~$2k, we ran a mentoring workshop out of Tiburon, to train volunteers to help us run large-scale workshops and also to do follow-up conversations with participants to help them benefit from the workshop in the weeks & months afterwards.

  • For ~$15k, we ran our annual alumni reunion. This year we had ~130 participants, with presentations and exercises on some angles on rationality that we think are promising. These also seem to be a lot of fun and help to energize the alumni community and keep us in touch with fresh ideas from the community that haven’t yet been put in writing.

  • For net zero cost, we have continued to run a weekly “rationality dojo” out of the CFAR office, where alumni work to deepen their skills with rationality and experiment with possible refinements or additions to the art.

Special programs

This year we ran two main summer programs:

  • SPARC ran for its fourth year in a row. Cisco and MIRI covered the costs of this program, so the non-time cost to CFAR was nil.

  • MIRI hired CFAR to run a three-week intensive Summer Fellows Program (MSFP), aimed at identifying and developing promising math research talent potentially related to AI safety research. MIRI covered the costs of running MSFP and paid CFAR $85k to cover both curriculum development time and time running the program itself.

In addition, an unnamed company hired CFAR to run a small training for them. The net financial effect on CFAR was zero: we charged enough to cover costs, viewing this workshop as an opportunity to continue exploring how CFAR might tailor its material for particular workplaces or specific needs.

Financial Summary

Our financial focus this last year was less on making money now and more on establishing internal infrastructure and strategies for developing solid income going forward.

We’re now in an excellent position to make CFAR much less dependent on donations going forward while simultaneously putting more focused effort on development, testing, and sharing of rationality tools than we’ve been able to do in the past.

This has made 2016 look very promising — but it has also put us in a difficult position right now.

We’re farther behind right now than we were this time last year, and we need some capital to implement the plans we have in mind. Predicting markets is always hard, but we think that with one more financial push this winter, we can both improve our contribution to the development of rationality and also make CFAR largely or maybe even entirely financially self-sustaining in 2016.

Ambitions for 2016

Hitting Scale

CFAR’s mission cashes out when people we equipped to think better and do more are actually in positions where they are changing the future of our world for the better.

With our external brand and our positioning within the community, we are perhaps uniquely well positioned to attract bright people, orient them to the values of systematically truer beliefs and world scale impact, and then make sure they get into the highest leverage positions they can fill.

We’ve spent the last three years leveling up our own ability to transmit a skillset and culture that we believe will move the needle in the right direction, and now is the time to execute at scale.

Core and Labs

To make scaling possible and still be able to competently tackle the pedagogical challenges we face, CFAR has arranged itself into two divisions: CFAR Core and CFAR Labs.

Pete Michaud (that’s me!) was hired to manage Core operations, including workshop and curriculum production and logistics. Anna Salamon will take the helm of CFAR Labs, which will be principally responsible for answering the questions:

  • What are the highest impact skillsets?

  • How can we detect them?

  • How can we train them?

  • Is our training actually affecting the important dimensions at the high end?

The Plan

Broadly, in order to attract more people, level them up reliably, and make sure they land in the highest impact positions they can, our plan is to:

  1. Substantially increase workshop volume

  2. Expand our community and continued training opportunities

  3. Directly address talent gaps by working with other organizations

  4. Continue increasing the quality of our instruction

Increase Workshop Volume

We intend to substantially increase the number of intake workshops we run and the number of participants we can serve per workshop.

“Intake workshops” here means workshops for people who haven’t necessarily been exposed to our material or community; said another way, these are workshops that will bring new people into our alumni network.

We are actively seeking a direct sales manager who can not only generate leads but close workshop sales. An alternative is to hire a two person marketing and sales team who together can generate leads and place prospects into workshops.

With the help of that new outreach team, we hope to add on the order of 1,000 new alumni in 2016, increasing our total throughput by nearly an order of magnitude.

Handling that new volume of alumni will require increasing attention to streamlining operations, which CFAR Core is handling partially by adding new team members and clarifying roles. In addition to me as the new Managing Director, we’ve already hired Duncan Sabien, an experienced educator and robustly capable operations generalist. Aside from the outreach team already mentioned, we also intend to hire a community manager (see below for details) and office assistant to fill in the inevitable gaps of an organization moving as fast as we intend to.

Community and Continued Training Opportunities

Bringing more talented people into the alumni network is only half the battle. Once participants have gone from “Zero to One,” only a community of practice can help ensure continued growth for most people.

We believe that one of the primary benefits of CFAR training is ongoing participation in the alumni community, both local to the Bay Area and throughout the world in local meetups and online. That’s why we’re going to invest in making the community stronger, with even more alumni events, experimental workshops, and deep-dive classes into specific aspects of our curriculum.

Perhaps the crown jewel of our community program is our Mentorship Training Program (MTP), which began its life as our TA Workshop. We intend to develop that seed into a robust pipeline capable of transforming workshop participants into trained rationality instructors.

One major benefit of the MTP will be that we’ll have more mentors and instructors to handle the increased load of all these workshops, classes, and other events.

But the MTP is a major growth opportunity even for people who aren’t necessarily interested in spreading the art of rationality themselves; we believe from our experience over the past three years that the best way to fully grok the art is to be immersed in a field of peers striving for the same, and ultimately to be able to teach it yourself.

This is what we intend to create with the MTP and new focus on community events.

To plan and manage all these alumni events, we’re looking for a capable community manager.

Directly Addressing Talent Gaps

In addition to our classic workshops and general education alumni programs, we’ll also be attempting to ramp up our targeted workshops meant to fill talent gaps for specific organizations.

For example, we’ll run our second MIRI Summer Fellows Program, as well as a grant funded by the Future of Life Institute to help promising upcoming AI researchers think about AI safety. We’re in conversation with other organizations, and it’s our intention to have an increasing number of these workshops that focus on thinking skills needed for particular tasks in order to help fill critical gaps in important organizations on very small time horizons.

If funding permits and our experiments in this area go well, we intend to make these types of workshops more frequent, and perhaps expand on past success with programs like a European SPARC, and possible “summer camp” style events where we try to identify particularly talented high school students for training and recruitment into existential risk research.

Labs: Informal experimentation toward a better “Applied Rationality”

The split between Core and Labs doesn’t only allow focus on operations—it also allows our Lab folk to invest in the informal experiments, arguments, data-gathering, etc. that seems, over time, to conduce to a better applied rationality.

(This process is messy. Rationality today is not at the level of Newton. It isn’t even at the level of Ptolemy, who, despite the mockability of the nested-epicycles method, could predict the motions of the planets with great precision. Rationality is more at the level of a toddler running around, putting everything in its mouth, and ending up thereby with a more integrated informal world-model by having examined many example-objects through several senses each. Our aim this year in Labs is basically to put many many things in our mouths rapidly, and to argue about models in between, and to especially expose ourselves to people who are working on issues that matter in already-very-competent ways who we can nevertheless try to make better, and to try in this way to get a better sense of the higher-end parts of “rationality”.)

Toward this end, Labs is currently:

  • Offering one-on-one coaching to quite a few individuals who seem to be contributing to the world in a high-end way; and trying to figure out how they’re doing what they’re doing, and what pieces may help them contribute more;

  • Working toward more robust and explicit models of the underlying mechanisms that create drive, scientific and epistemic skill, and relevant real-world competence (and how to intervene upon them);

  • Creating new written rationality sequences meant to expand upon, augment, and improve the original sequences that brought so many people into the culture of being “less wrong,” and oriented them around audacious goals that actually make a difference;

  • Planning experimental workshops of varied sorts, aiming to boost people further toward “actually useful skill-levels in applied rationality”.

We are very excited, and expect that art development will be much easier now that we have a subteam who is free to just actually focus on it. (Last year, we were all doing workshop admissions, logistics, accounting, …)

Limitations and Updates

The primary limiting factor in these plans is our ability to attract a truly excellent sales person or team. With a sufficient workshop participation, cashflow bottlenecks are broken and we‘ll achieve economies of scale that will fundamentally transform our operations.

Failing that recruitment, the next best alternative is to grow organically through the MTP and other community programs. That is a much slower process, but pushes us in the same fundamental direction.

And as always, our plans coming into contact with the reality of 2016 will correctly cause us to update, iterate, and potentially pivot given new evidence and insight.

The path forward, and how you can help

CFAR’s mission is to gather together people with the potential for real and meaningful impact, and to cause them to come closer to meeting that potential. It doesn’t much matter whether you think we’re under a ticking clock of existential risk, or you’re concerned about a million humans dying every week, or you’re simply grumpy that we haven’t gotten a human past low earth orbit since 1972—our individual and collective thinking skill is a key bottleneck on our future.

Applied rationality, more than almost anything else, has a shot at being a truly all-purpose tool in humanity’s toolkit, and the bigger the problems on the horizon, the more vital that tool becomes.

2016 will be a particularly critical year in CFAR’s history. We’re restructuring our team in pretty major ways, and finding the right team members (or not) will determine our ability to get the right character and culture from this new beginning; and we’ve had at least three good people in the last eight months who we wanted to hire, and who wanted to work for us, but who required salaries we couldn’t afford. Beginnings are far easier times in which to make change, and this is the closest we’ve come to a fresh beginning—and the time we’ve most expected differential impact from marginal donation—since our inaugural fundraiser of late 2012.

The world of AI risk is changing rapidly, and decisions made over the coming months will shape the future of the field—it would be well to get relevant training programs going now, and not to wait for some later additional hard-won new beginning for CFAR in 2018 or something. The strategic competence we will have going into the spring is likely to be the difference between a CFAR that actually matters, and one that sounds good but is ultimately irrelevant.

There are at least four major ways to help:

  1. Donate directly to our winter fundraising drive. This is the most straightforward way to help, and makes a categorical difference in our ability to execute the mission. (A large majority of our funding comes from small donors.)

  2. If you’re interested in rationality, or in the larger questions of humanity’s future and existential risk, consider reading the Sequences, or otherwise working to improve your thinking and world-modeling skill. (Strong community epistemology is extremely helpful.)

  3. We’re always looking for new alumni, particularly those who care about both rationality and the world. If you haven’t been, consider applying to a CFAR workshop; and if you have been, consider mentioning it to people who fit said description.

  4. If you’re interested in joining us for the long haul, we’re currently looking to hire a sales manager, a community manager, and an office assistant (funding permitting). We’ve identified these three roles as the highest-impact additions to the CFAR staff, and are eager to hear from enthusiastic and qualified candidates.

This is the mission; these are the steps. CFAR has made substantial progress on building a talent pipeline for clear thinkers and world changers, in large part thanks to generous contributions of time, money, energy, and insight from people like you. We’d like to see a world where this goal has been achieved, and your support is what gets us there. Thanks for reading; do send us any thoughts; and do please consider donating now.

PeteMichaud23 Dec 2015 22:46 UTC
73 points
63 comments17 min readLW linkArchive