I’ve heard a lot of people say things along the lines that CFAR “no longer does original research into human rationality.” Does that seem like an accurate characterization? If so, why is it the case that you’ve moved away from rationality research?
Hello, I am a CFAR contractor who considers nearly all of their job to be “original research into human rationality”. I don’t do the kind of research many people imagine when they hear the word “research” (RCT-style verifiable social science, and such). But I certainly do systematic inquiry and investigation into a subject in order to discover or revise beliefs, theories, applications, etc. Which is, you know, literally the dictionary.com definition of research.
I’m not very good at telling stories about myself, but I’ll attempt to describe what I do during my ordinary working hours anyway.
All of the time, I keep an eye out for things that seem to be missing or off in what I take to be the current art of rationality. Often I look to what I see in the people close to me, who are disproportionately members of rationality-and-EA-related organizations, watching how they solve problems and think through tricky stuff and live their lives. I also look to my colleagues at CFAR, who spend many many hours in dialogue with people who are studying rationality themselves, for the first time or on a continuing basis. But since my eyes are in my own head, I look most for what is absent in my own personal art of rationality.
For example, when I first read the Sequences in 2012 or 2013, I gained a lot, but I also felt a gaping hole in the shape of something like “recognizing those key moments in real-life experience when the rationality stuff you’ve thought so much about comes whizzing by your head at top speed, looking nothing at all like the abstractions you’ve so far considered”. That’s when I started doing stuff like snapping my fingers every time I saw a stop sign, so I could get a handle on what “noticing” even is, and begin to fill in the hole. I came up with a method of hooking intellectual awareness up to immediate experience, then I spent a whole year throwing the method at a whole bunch of real life situations, keeping track of what I observed, revising the method, talking with people about it as they worked with the same problem themselves, and generally trying to figure out the shape of the world around phenomenology and trigger-action planning.
I was an occasional guest instructor with CFAR at the time, and I think that over the course of my investigations, CFAR went from spending very little time on the phenomenological details of key experiences to working that sort of thing into nearly every class. I think it’s now the case that rationality as it currently exists contains an “art of noticing”.
My way of investigating always pushes into what I can’t yet see or grasp or articulate. Thus, it has the unfortunate property of being quite difficult to communicate about directly until the research program is mostly complete. So I can say a lot about my earlier work on noticing, but talking coherently about what exactly CFAR’s been paying me for lately is much harder. It’s all been the same style of research, though, and if I had to give names to my recent research foci, I’d say I’ve been looking into original seeing, some things related to creativity and unconstrained thought, something about learning and what it means to own your education, and experiences related to community and cooperation.
It’s my impression that CFAR has always had several people doing this kind of thing, and that several current CFAR staff members consider it a crucial part of their jobs as well. When I was hired, Tim described research as “the beating heart” of our organization. Nevertheless, I personally would like more of it in future CFAR, and I’d like it to be done with a bit more deliberate institutional support.
That’s why it was my primary focus when working with Eli to design our 2019 instructor training program. The program consisted partially of several weekend workshops, but in my opinion the most important part happened while everyone was at home.
My main goal, especially for the first weekend, was to help the trainees choose a particular area of study. It was to be something in their own rationality that really mattered to them and that they had not yet mastered. When they left the workshop, they were to set off on their own personal quest to figure out that part of the world and advance the art.
This attitude, which we’ve been calling “questing” of late, is the one with which I hope CFAR instructors will approach any class they intend to teach, whether it’s something like “goal factoring” that many people have taught in the past, or something completely new that nobody’s even tried to name yet. When you really get the hang of the questing mentality, you never stop doing original rationality research. So to whatever degree I achieved my goal with instructor training (which everyone seems to think is a surprisingly large degree), CFAR is moving in the direction of more original rationality research, not less.
How do CFAR’s research interests/priorities compare with LW’s Open Problems in Human Rationality? Based on Brienne and Anna’s replies here, I suspect the answer is “they’re pretty different”, but I’d like to hear what accounts for this divergence.
I quite like the open questions that Wei Dai wrote there, and I expect I’d find progress on those problems to be helpful for what I’m trying to do with CFAR. If I had to outline the problem we’re solving from scratch, though, I might say:
Figure out how to:
use reason (and stay focused on the important problems, and remember “virtue of the void” and “lens that sees its own flaws, and be quick where you can) without
going nutso, or losing humane values, and while:
being able to coordinate well in teams.
Wei Dai’s open problems feel pretty relevant to this!
I think in practice this goal leaves me with subproblems such as:
How do we un-bottleneck “original seeing” / hypothesis-generation;
What is the “it all adds up to normality” skill based in; how do we teach it;
Where does “mental energy” come from in practice, and how can people have good relationships to this;
What’s up with people sometimes seeming self-conscious/self-absorbed (in an unfortunate, slightly untethered way) and sometimes seeming connected to “something to protect” outside themselves?
It seems to me that “something to protect” makes people more robustly mentally healthy. Is that true? If so why? Also how do we teach it?
Why is it useful to follow “spinning plates” (objects that catch your interest for their own sake) as well as “hamming questions”? What’s the relationship between those two? (I sort of feel like they’re two halves of the same coin somehow? But I don’t have a model.)
As well as more immediately practical questions such as: How can a person do “rest days” well. What ‘check sums’ are useful for noticing when something breaks as you’re mucking with your head. Etc.
My model is that CFAR is doing the same activity it was always doing, which one may or may not want to call “research”.
I’ll describe that activity here. I think it is via this core activity (plus accidental drift, or accidental hill-climbing in response to local feedbacks) that we have generated both our explicit curriculum, and a lot of the culture around here.
Components of this core activity (in no particular order):
We try to teach specific skills to specific people, when we think those skills can help them. (E.g. goal-factoring; murphyjitsu; calibration training on occasion; etc.)
We keep our eyes open while we do #1. We try to notice whether the skill does/doesn’t match the student’s needs. (E.g., is this so-called “skill” actually making them worse at something that we or they can see? Is there a feeling of non-fit suggesting something like that? What’s actually happening as the “skill” gets “learned”?)
We call this noticing activity “seeking PCK” and spend a bunch of time developing it in our mentors and instructors.
We try to stay in touch with some of our alumni after the workshop, and to notice what the long-term impacts seem to be (are they actually practicing our so-called “skills”? Does it help when they do? More broadly, what changes do we just-happen-by-coincidence to see in multiple alumni again and again, and are these positive or negative changes, and what might be causing them?
In part, we do this via the four follow-up calls that participants receive after they attend the mainline workshop; in part we do it through the alumni reunions, the kind of contact that comes naturally from being in the same communities, etc.
We often describe some of what we think we’re seeing, and speculate about where to go given that, in CFAR’s internal colloqium.
We pay particular attention to alumni who are grappling with existential risk or EA, partly because it seems to pose distinct difficulties that it would be nice if someone found solutions to.
Spend a bunch of time with people who are succeeding at technical AI safety work, trying to understand what skills go into that. Spend a bunch of time with people who are training to do technical AI safety work (often at the same time that people who can actually do such work are there), working to help transfer useful mindset (while trying also to pay attention to what’s happening.
Right now we do this mostly at the AIRCS and MSFP workshops.
Spend a bunch of time engaging smart new people to see what skills/mindsets they would add to the curriculum, so we don’t get too stuck in a local optimum.
What this looks like recently:
The instructor training workshops are helping us with this. Many of us found those workshops pretty generative, and are excited about the technique-seeds and cultural content that the new instructor candidates have been bringing.
The AIRCS program has also been bringing in highly skilled computer scientists, often from outside the rationality and EA community. My own thinking has changed a good bit in contact with the AIRCS experience. (They are less explicitly articulate about curriculum than the instructor candidates; but they ask good questions, buy some pieces of our content, get wigged out by other pieces of our content in a non-random manner, answer follow-up questions in contact with that that sometimes reveal implicit causal models of how to think that seem correct to me, etc. And so they are a major force for AIRCS curriculum generation in that way.)
Gaps in 5:
I do wish we had better contact with more and varied highly productive thinkers/markers of different sorts, as a feed-in to our curriculum. We unfortunately have no specific plans to fix this gap in 2020 (and I don’t think it could fit without displacing some even-more-important planned shift—we have limited total attention); but it would be good to do sometime over the next five years. I keep dreaming of a “writers’ workshop” and an “artist’s workshop” and so on, aimed at seeing how our rationality stuff mutates when it hits people with different kinds of visibly-non-made-up productive skill.
We sometimes realize that huge swaths of our curriculum are having unwanted effects and try to change it. We sometimes realize that our model of “the style of thinking we teach” is out of touch with our best guesses about what’s good, and try to change it.
We try to study any functional cultures that we see (e.g., particular functional computer science communities; particular communities found in history books), to figure out what magic was there. We discuss this informally and with friends and with e.g. the instructor candidates.
We try to figure out how thinking ever correlates with the world, and when different techniques make this better and worse in different context. And we read the Sequences to remember that this is what we’re doing.
We could stand to do this one more; increasing this is a core planned shift for 2020. But we’ve always done it some, including over the last few years.
The “core activity” exemplified in the above list is, of course, not RCT-style verifiable track records-y social science (which is one common meaning of “research”). There is a lot of merit to that verifiable social science, but also a lot of slowness to it, and I cannot imagine using it to design the details of a curriculum, although I can imagine using it to see whether a curriculum has particular high-level effects.
We also still do some (but not as much as we wish we could do) actual data-tracking, and have plans to do modestly more of it over the coming year. I expect this planned modest increase will be useful for our broader orientation but not much of a direct feed-in into curriculum, although it might help us tweak certain knobs upward or downward a little.
Also worth noting that there are a few different claims of the sort OP mentions that people make, I think. One thing people sometimes mean by this is “CFAR no longer does the sort of curriculum development which would be necessary to create an ‘Elon Musk factory.’”
CFAR never had the goal of hugely amplifying the general effectiveness of large numbers of people (which I’m happy about, since I’m not sure achieving that goal would be good). One should not donate to CFAR in order to increase the chances of an Elon Musk factory.
That is, we were always focused on high-intensity interventions for small numbers of people—especially the people who are the very easiest to impact (have free time; smart and reflective; lucky in their educational background and starting position). We did not expect things to generalize to larger sets.
(Mostly. We did wonder about books and things for maybe impacting the epistemics (not effectiveness) of some larger number of people a small amount. And I do personally think that if there ways to help with the general epistemic, wisdom, or sanity of larger sets of people, even if by a small amount, that would be worth meaningful tradeoffs to create. But we are not presently aiming for this (except in the broadest possible “keep our eyes open and see if we someday notice some avenue that is actually worth taking here” sense), and with the exception of helping to support Julia Galef’s upcoming rationality book back when she was working here, we haven’t ever attempted concrete actions aimed at figuring out how to impact larger sets of people).
I agree, though, that one should not donate to CFAR in order to increase the chances of an Elon Musk factory.
But then again, so is Y-combinator, and every other incubator, as well as pretty much every leadership retreat (ok maybe not the leadership retreats, because Elon Musk is a terrible leader, but they’re trying to do something like create a factory for what people imagine Elon Musk to be like). It seems like a very competitive space to create an Elon Musk factory, because its’ so economically valuable.
This is a drive-by, but I don’t believe this statement, based on the fact that Elon has successfully accomplished several hard things via the use of people organized in hierarchies (companies). I’m sure he has foibles, and it might not be fun to work for him, but he does get shit done.
I think that there are many rare and positive qualities of Musk that I try to emulate, and some rare qualities that are damaging and that I shouldn’t emulate. Importantly, from many broad perspectives (like thinking that economic growth is a robust good) it’s pretty weird to think that Elon Musk is bad. I presume you think Musk is pretty unilateralist and think that he probably did net damage with the building of OpenAI?
I think Musk is impressive in many ways. I didn’t really intend to express skepticism of him in particular, so much as of what might happen if one created loads more people as agenty as him. For example, I can easily imagine this accelerating capabilities progress relative to safety progress, which strikes me as bad.
I’ve heard a lot of people say things along the lines that CFAR “no longer does original research into human rationality.” Does that seem like an accurate characterization? If so, why is it the case that you’ve moved away from rationality research?
Hello, I am a CFAR contractor who considers nearly all of their job to be “original research into human rationality”. I don’t do the kind of research many people imagine when they hear the word “research” (RCT-style verifiable social science, and such). But I certainly do systematic inquiry and investigation into a subject in order to discover or revise beliefs, theories, applications, etc. Which is, you know, literally the dictionary.com definition of research.
I’m not very good at telling stories about myself, but I’ll attempt to describe what I do during my ordinary working hours anyway.
All of the time, I keep an eye out for things that seem to be missing or off in what I take to be the current art of rationality. Often I look to what I see in the people close to me, who are disproportionately members of rationality-and-EA-related organizations, watching how they solve problems and think through tricky stuff and live their lives. I also look to my colleagues at CFAR, who spend many many hours in dialogue with people who are studying rationality themselves, for the first time or on a continuing basis. But since my eyes are in my own head, I look most for what is absent in my own personal art of rationality.
For example, when I first read the Sequences in 2012 or 2013, I gained a lot, but I also felt a gaping hole in the shape of something like “recognizing those key moments in real-life experience when the rationality stuff you’ve thought so much about comes whizzing by your head at top speed, looking nothing at all like the abstractions you’ve so far considered”. That’s when I started doing stuff like snapping my fingers every time I saw a stop sign, so I could get a handle on what “noticing” even is, and begin to fill in the hole. I came up with a method of hooking intellectual awareness up to immediate experience, then I spent a whole year throwing the method at a whole bunch of real life situations, keeping track of what I observed, revising the method, talking with people about it as they worked with the same problem themselves, and generally trying to figure out the shape of the world around phenomenology and trigger-action planning.
I was an occasional guest instructor with CFAR at the time, and I think that over the course of my investigations, CFAR went from spending very little time on the phenomenological details of key experiences to working that sort of thing into nearly every class. I think it’s now the case that rationality as it currently exists contains an “art of noticing”.
My way of investigating always pushes into what I can’t yet see or grasp or articulate. Thus, it has the unfortunate property of being quite difficult to communicate about directly until the research program is mostly complete. So I can say a lot about my earlier work on noticing, but talking coherently about what exactly CFAR’s been paying me for lately is much harder. It’s all been the same style of research, though, and if I had to give names to my recent research foci, I’d say I’ve been looking into original seeing, some things related to creativity and unconstrained thought, something about learning and what it means to own your education, and experiences related to community and cooperation.
It’s my impression that CFAR has always had several people doing this kind of thing, and that several current CFAR staff members consider it a crucial part of their jobs as well. When I was hired, Tim described research as “the beating heart” of our organization. Nevertheless, I personally would like more of it in future CFAR, and I’d like it to be done with a bit more deliberate institutional support.
That’s why it was my primary focus when working with Eli to design our 2019 instructor training program. The program consisted partially of several weekend workshops, but in my opinion the most important part happened while everyone was at home.
My main goal, especially for the first weekend, was to help the trainees choose a particular area of study. It was to be something in their own rationality that really mattered to them and that they had not yet mastered. When they left the workshop, they were to set off on their own personal quest to figure out that part of the world and advance the art.
This attitude, which we’ve been calling “questing” of late, is the one with which I hope CFAR instructors will approach any class they intend to teach, whether it’s something like “goal factoring” that many people have taught in the past, or something completely new that nobody’s even tried to name yet. When you really get the hang of the questing mentality, you never stop doing original rationality research. So to whatever degree I achieved my goal with instructor training (which everyone seems to think is a surprisingly large degree), CFAR is moving in the direction of more original rationality research, not less.
How do CFAR’s research interests/priorities compare with LW’s Open Problems in Human Rationality? Based on Brienne and Anna’s replies here, I suspect the answer is “they’re pretty different”, but I’d like to hear what accounts for this divergence.
I quite like the open questions that Wei Dai wrote there, and I expect I’d find progress on those problems to be helpful for what I’m trying to do with CFAR. If I had to outline the problem we’re solving from scratch, though, I might say:
Figure out how to:
use reason (and stay focused on the important problems, and remember “virtue of the void” and “lens that sees its own flaws, and be quick where you can) without
going nutso, or losing humane values, and while:
being able to coordinate well in teams.
Wei Dai’s open problems feel pretty relevant to this!
I think in practice this goal leaves me with subproblems such as:
How do we un-bottleneck “original seeing” / hypothesis-generation;
What is the “it all adds up to normality” skill based in; how do we teach it;
Where does “mental energy” come from in practice, and how can people have good relationships to this;
What’s up with people sometimes seeming self-conscious/self-absorbed (in an unfortunate, slightly untethered way) and sometimes seeming connected to “something to protect” outside themselves?
It seems to me that “something to protect” makes people more robustly mentally healthy. Is that true? If so why? Also how do we teach it?
Why is it useful to follow “spinning plates” (objects that catch your interest for their own sake) as well as “hamming questions”? What’s the relationship between those two? (I sort of feel like they’re two halves of the same coin somehow? But I don’t have a model.)
As well as more immediately practical questions such as: How can a person do “rest days” well. What ‘check sums’ are useful for noticing when something breaks as you’re mucking with your head. Etc.
I’m not sure I understand what you mean by “something to protect.” Can you give an example?
[Answered by habryka]
Presumable it’s a reference to: https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect
Thanks! forgot about that post.
If Brienne wanted to give their own answer to that post, even if it was incomplete, I’d be very excited about that.
Done.
Hurrah! :D
My model is that CFAR is doing the same activity it was always doing, which one may or may not want to call “research”.
I’ll describe that activity here. I think it is via this core activity (plus accidental drift, or accidental hill-climbing in response to local feedbacks) that we have generated both our explicit curriculum, and a lot of the culture around here.
Components of this core activity (in no particular order):
We try to teach specific skills to specific people, when we think those skills can help them. (E.g. goal-factoring; murphyjitsu; calibration training on occasion; etc.)
We keep our eyes open while we do #1. We try to notice whether the skill does/doesn’t match the student’s needs. (E.g., is this so-called “skill” actually making them worse at something that we or they can see? Is there a feeling of non-fit suggesting something like that? What’s actually happening as the “skill” gets “learned”?)
We call this noticing activity “seeking PCK” and spend a bunch of time developing it in our mentors and instructors.
We try to stay in touch with some of our alumni after the workshop, and to notice what the long-term impacts seem to be (are they actually practicing our so-called “skills”? Does it help when they do? More broadly, what changes do we just-happen-by-coincidence to see in multiple alumni again and again, and are these positive or negative changes, and what might be causing them?
In part, we do this via the four follow-up calls that participants receive after they attend the mainline workshop; in part we do it through the alumni reunions, the kind of contact that comes naturally from being in the same communities, etc.
We often describe some of what we think we’re seeing, and speculate about where to go given that, in CFAR’s internal colloqium.
We pay particular attention to alumni who are grappling with existential risk or EA, partly because it seems to pose distinct difficulties that it would be nice if someone found solutions to.
Spend a bunch of time with people who are succeeding at technical AI safety work, trying to understand what skills go into that. Spend a bunch of time with people who are training to do technical AI safety work (often at the same time that people who can actually do such work are there), working to help transfer useful mindset (while trying also to pay attention to what’s happening.
Right now we do this mostly at the AIRCS and MSFP workshops.
Spend a bunch of time engaging smart new people to see what skills/mindsets they would add to the curriculum, so we don’t get too stuck in a local optimum.
What this looks like recently:
The instructor training workshops are helping us with this. Many of us found those workshops pretty generative, and are excited about the technique-seeds and cultural content that the new instructor candidates have been bringing.
The AIRCS program has also been bringing in highly skilled computer scientists, often from outside the rationality and EA community. My own thinking has changed a good bit in contact with the AIRCS experience. (They are less explicitly articulate about curriculum than the instructor candidates; but they ask good questions, buy some pieces of our content, get wigged out by other pieces of our content in a non-random manner, answer follow-up questions in contact with that that sometimes reveal implicit causal models of how to think that seem correct to me, etc. And so they are a major force for AIRCS curriculum generation in that way.)
Gaps in 5:
I do wish we had better contact with more and varied highly productive thinkers/markers of different sorts, as a feed-in to our curriculum. We unfortunately have no specific plans to fix this gap in 2020 (and I don’t think it could fit without displacing some even-more-important planned shift—we have limited total attention); but it would be good to do sometime over the next five years. I keep dreaming of a “writers’ workshop” and an “artist’s workshop” and so on, aimed at seeing how our rationality stuff mutates when it hits people with different kinds of visibly-non-made-up productive skill.
We sometimes realize that huge swaths of our curriculum are having unwanted effects and try to change it. We sometimes realize that our model of “the style of thinking we teach” is out of touch with our best guesses about what’s good, and try to change it.
We try to study any functional cultures that we see (e.g., particular functional computer science communities; particular communities found in history books), to figure out what magic was there. We discuss this informally and with friends and with e.g. the instructor candidates.
We try to figure out how thinking ever correlates with the world, and when different techniques make this better and worse in different context. And we read the Sequences to remember that this is what we’re doing.
We could stand to do this one more; increasing this is a core planned shift for 2020. But we’ve always done it some, including over the last few years.
The “core activity” exemplified in the above list is, of course, not RCT-style verifiable track records-y social science (which is one common meaning of “research”). There is a lot of merit to that verifiable social science, but also a lot of slowness to it, and I cannot imagine using it to design the details of a curriculum, although I can imagine using it to see whether a curriculum has particular high-level effects.
We also still do some (but not as much as we wish we could do) actual data-tracking, and have plans to do modestly more of it over the coming year. I expect this planned modest increase will be useful for our broader orientation but not much of a direct feed-in into curriculum, although it might help us tweak certain knobs upward or downward a little.
Also worth noting that there are a few different claims of the sort OP mentions that people make, I think. One thing people sometimes mean by this is “CFAR no longer does the sort of curriculum development which would be necessary to create an ‘Elon Musk factory.’”
CFAR never had the goal of hugely amplifying the general effectiveness of large numbers of people (which I’m happy about, since I’m not sure achieving that goal would be good). One should not donate to CFAR in order to increase the chances of an Elon Musk factory.
That is, we were always focused on high-intensity interventions for small numbers of people—especially the people who are the very easiest to impact (have free time; smart and reflective; lucky in their educational background and starting position). We did not expect things to generalize to larger sets.
(Mostly. We did wonder about books and things for maybe impacting the epistemics (not effectiveness) of some larger number of people a small amount. And I do personally think that if there ways to help with the general epistemic, wisdom, or sanity of larger sets of people, even if by a small amount, that would be worth meaningful tradeoffs to create. But we are not presently aiming for this (except in the broadest possible “keep our eyes open and see if we someday notice some avenue that is actually worth taking here” sense), and with the exception of helping to support Julia Galef’s upcoming rationality book back when she was working here, we haven’t ever attempted concrete actions aimed at figuring out how to impact larger sets of people).
I agree, though, that one should not donate to CFAR in order to increase the chances of an Elon Musk factory.
Do you have any advice on who to donate to in order to increase the chances of an Elon Musk factory?
It seems like paradigm academy is trying to do something like create an Elon Musk Factory:
http://paradigmacademy.co/
But then again, so is Y-combinator, and every other incubator, as well as pretty much every leadership retreat (ok maybe not the leadership retreats, because Elon Musk is a terrible leader, but they’re trying to do something like create a factory for what people imagine Elon Musk to be like). It seems like a very competitive space to create an Elon Musk factory, because its’ so economically valuable.
This is a drive-by, but I don’t believe this statement, based on the fact that Elon has successfully accomplished several hard things via the use of people organized in hierarchies (companies). I’m sure he has foibles, and it might not be fun to work for him, but he does get shit done.
I claim that Elon has done this despite his leadership abilities.
I think that it’s possible to be a bad leader but an effective CEO.
It seems to me unclear what exactly do you mean with the terms. What do you mean with leadership as compared to being a CEO?
Leadership (as for instance leadership retreats are trying to teach it) is the intersection between management and strategy.
Another way to put it, its’ the discipline of getting people to do what’s best for your organization.
Do you think that Elon doesn’t get his employees to do what’s best for his companies?
I think he’s bad at this.
You can see this in some aspects of his companies.
High micromanagement. High turnover. Disgruntled former employees.
I’m not aware of existing organizations that seem likely to me to create such a factory.
I think that there are many rare and positive qualities of Musk that I try to emulate, and some rare qualities that are damaging and that I shouldn’t emulate. Importantly, from many broad perspectives (like thinking that economic growth is a robust good) it’s pretty weird to think that Elon Musk is bad. I presume you think Musk is pretty unilateralist and think that he probably did net damage with the building of OpenAI?
I think Musk is impressive in many ways. I didn’t really intend to express skepticism of him in particular, so much as of what might happen if one created loads more people as agenty as him. For example, I can easily imagine this accelerating capabilities progress relative to safety progress, which strikes me as bad.