Refine: An Incubator for Conceptual Alignment Research Bets

adamShimi15 Apr 2022 8:57 UTC
LW: 144 AF: 45
13 comments4 min readLW link

I’m opening an incubator called Refine for conceptual alignment research in London, which will be hosted by Conjecture. The program is a three-month fully-paid fellowship for helping aspiring independent researchers find, formulate, and get funding for new conceptual alignment research bets, ideas that are promising enough to try out for a few months to see if they have more potential.

If this sounds like something you’d be interested in, you can apply here!

Why?

I see a gaping hole in the alignment training ecosystem: there are no programs dedicated specifically to creating new independent conceptual researchers and helping them build original research agendas.

The programs that do exist (AI Safety Camp, SERI MATS) tend to focus on an apprenticeship (or “accelerated PhD”) model in which participants work under researchers on already-established research directions. And while there are avenues for independent alignment researchers to get started on their own, it is fraught with many risks, slowing down progress considerably.

So I feel the need for a program geared specifically towards conceptual alignment researchers that are interested in doing their own research and making their own research bets.

Who?

This program is for self-motivated and curious people who want to become independent conceptual alignment researchers and expand the portfolio of alignment bets and research ideas available.

When I look at great conceptual researchers like John Wentworth, Paul Christiano, Evan Hubinger, Steve Byrnes, Vanessa Kosoy, and others, as well as at the good (famous and not) researchers I know from my PhD, they all have the same thing in common: they ask a question and keep looking for the answer. They tolerate confusion, not in the sense that they accept it, but in that they are able to work with it and not hide away behind premature formalization. They don’t give up on the problem; they search for different angles and approaches until it yields. Paul Graham calls this being relentlessly resourceful.

(Relentlessly Resourceful, Paul Graham, 2009)

I was writing a talk for investors, and I had to explain what to look for in founders. What would someone who was the opposite of hapless be like? They’d be relentlessly resourceful. Not merely relentless. That’s not enough to make things go your way except in a few mostly uninteresting domains. In any interesting domain, the difficulties will be novel. Which means you can’t simply plow through them, because you don’t know initially how hard they are; you don’t know whether you’re about to plow through a block of foam or granite. So you have to be resourceful. You have to keep trying new things.

This is one of the main traits I’m looking for in an applicant — someone who will lead a new research agenda and morph it proactively, as needed.

Another point that matters is being curious about different topics and ideas than the ones traditionally discussed in alignment. As I wrote in a recent post and plan to discuss more in an upcoming sequence, I think we need to be more pluralist in our approach to alignment, and explore far more directions, from novel ideas to old approaches that may have been discarded too soon. And new ideas often come from unexpected places.

As one example, here is what Jesse Schell writes about his experience speaking to a professional juggler who performed tricks no one else could do:

(The Art of Game Design, Jesse Schell, 2008)

“The secret is: don’t look to other jugglers for inspiration—look everywhere else.” He proceeded to do a beautiful looping pattern, where his arms kind of spiraled, and he turned occasional pirouettes. “I learned that one watching a ballet in New York. and this one...” he did a move that involved the balls popping up and down as his hands fluttered delicately back and forth. “I learned that from a flock of geese I saw take off from a lake up in Maine. And this,” he did a weird mechanical looking movement where the balls almost appeared to move at right angles. “I learned that from a paper punch machine on Long Island.” He laughed a little and stopped juggling for a minute. “People try to copy these moves, but they can’t. They always try… yeah, look at that fella, over there!” He pointed to a juggler with a long ponytail across the gym who was doing the “ballet” move, but it just looked dumb. Something was missing, but I couldn’t say what.

“See, these guys can copy my moves, but they can’t copy my inspiration.”

As for previous experience with alignment research, it can both be a blessing and a curse. While familiarity with alignment concepts can help bootstrap the learning and idea generation process, it also risks clogging the babble process by constraining “what makes sense”. For those it would be helpful for, the program includes some initial teaching on core alignment ideas (according to me) and the mental moves necessary for good alignment research.

Some concrete details

We plan to invite the first cohort of 4-5 fellows from July/​August through September/​October (wiggle room depending on some ops details), though exact dates will be determined by their availability. We anticipate that other cohorts will follow, so if you miss the first round but are still interested, please apply.

This is a full-time position in London where fellows will work out of Conjecture’s offices. The program includes:

  • Travel and Housing: Round-trip plane/​train tickets to and from London, housing for the duration of the program, as well as public transportation within London.

  • Stipend: A stipend of ~$3,000/​month (after tax) to cover meals and discretionary expenses.

  • Office Infrastructure: A desk in the Conjecture office (and tech setup when needed) and access to Conjecture’s conference rooms and other amenities.

  • Collaboration: Formal opportunities to discuss research directions with other conceptual and applied alignment researchers and engineers at Conjecture, and opportunities to meet and share ideas with other London-based alignment researchers.

  • Funding Assistance: Help in finding funding opportunities and in writing grant proposals for continuing to study research bets after the incubator.

During the first month of the program, participants will spend their time discussing abstract models of alignment, what the problem is about, and the different research approaches that have been pursued. The focus will be on understanding the assumptions and constraints behind the different takes and research programs, to get a high-level map of the field.

The next ~two months of the program will focus on helping fellows babble new research bets on alignment, refine them, test them, and either throw them away or change them. By the end, the goal is for fellows to narrow in on a research bet that could be further investigated in the following 6 months, and is promising enough to warrant funding.

It’s worth noting that while the incubator is being housed by Conjecture, fellows do not have any constraints imposed by the company. Fellows will not have to work on Conjecture’s research agendas or be obligated to collaborate after the program is over. Similarly, I’m not looking for people to work on my own research ideas, but for new exciting research bets I wouldn’t have thought about.

How can I apply?

We will review applications on a rolling-basis, with a usual delay of 1 week before response and a month before a decision (with a work task in the middle). The application is open now!

adamShimi15 Apr 2022 8:57 UTC
LW: 144 AF: 45
13 comments4 min readLW link