These are all fantastic questions! I’ll try to answer some of the ones I can. (Unfortunately a lot of the people who could answer the rest are pretty busy right now with EAGxBerkeley, getting set up for REMIX, etc., but I’m guessing that they’ll start having a chance to answer some of these in the coming days.)
Regarding the research program, I’m guessing there’s around 6-10 research projects ongoing, with between 1 and 3 students working on each; I’m guessing almost none of the participants have previous research experience. (Kuhan would have the actual numbers here.) This program just got started in late October, so certainly no published results yet.
I’m guessing the mentors are not all on the same page about how much of the value comes from doing object-level useful research vs. upskilling. My feeling is that it’s mostly upskilling, with the exception of a few projects where the mentor was basically taking on a RA for a project they were already working on full-time. In fact, when pitching projects, I explicitly disclaimed for some of them that I thought they were likely not useful for alignment (but would be useful for learning research skills and ML upskilling).
It sounds like in your situation, there’s a lack of experienced mentors. (Though I’ll note that a mentor spending ~1 hour per week meeting with a group sounds like plenty to me.) If that’s right, then I think I’d recommend focusing on ML upskilling programming instead of starting a research program. My thoughts here are: (1) I doubt participants will get much mileage out of working on projects that they came up with themselves, especially without mentors to help them shape their work; (2) poorly mentored research projects can be frustrating for the mentees, and might sour them on further engaging with your programming or AI safety as a whole; (3) ML upskilling programming seems almost as valuable to me and much easier to do well.
Regarding general member programming: for our weekly reading group, we pick readings week-by-week, usually based on someone messaging a group chat saying “I’d really love to read X this week.” (X was often something that had come out in the last week or so.) I don’t think this wasn’t an especially good way to do things, but we got lucky and it mostly worked out.
That said, I think most of the value here was from getting a bunch of aligned people in a room reading something and discussing with each other. If you don’t already have a lot of people sold on AI x-risk and with a background similar to having completed AGISF, I think it’d be better to run a more structured reading group rather than doing something like this.
Like we mentioned in the post, we think that we actually underinvested in developing programming for our members to participate in (instead putting slightly too much work into making the intro fellowship go well). Most of our full members were too busy for the research program, and the bar for facilitating for our intro fellowship was relatively high (other than Xander, all of our facilitators were PhD students or people who worked full-time on AIS). So the only real thing we had for full members were the weekly general member meetings and the retreats at the end of the semester.
For the typical person who finished your AGISF intro group and has good technical skills, which activities would you most want them to focus on? (My guess would be research > outreach and facilitation > participant in reading groups > social events.)
I think my ordering would be
research > further ML upskilling > reading groups > outreach
with social events not really mattering much to me, and facilitating not being an option for most of them, thanks to our wealth of over-qualified facilitators. I’m not sure how this should translate to your situation, sorry.
Regarding the intro fellowship, we hadn’t really considered MLSS at all, and probably we should have. I think we were approaching things from a frame separating our programming into things that require coding (ML upskilling) and things that don’t (AGISF), but this was potentially a mistake. The MLSS curriculum looks good, I agree that it seems better at getting people research-ready, and I’ll think about whether it makes sense to incorporate some of this stuff for next semester—thanks for this suggestion!
One dynamic to keep in mind is that when you advertise for an AI educational program, you’ll get a whole bunch of people who are excited about AI and don’t care much about the safety angle (it seems like lots of the people we attracted to our research program were like this). To some extent this is okay—it gives a chance to persuade people who would have otherwise gone into AI capabilities work! -- but I think it’s also worth trying not to spend resources teaching ML to people who will just go off and work in capabilities. One nice thing about AGISF is that it starts off with multiple weeks on safety, allowing people who aren’t interested in safety to self-select out before the technical material. (And the technical content is mostly stuff that I’m not worried is could advance capabilities anyway.) So if you’ve noticed that you have a lot of people sticking around to the end of your curriculum without really engaging with the safety angle, I might recommend front-loading some AGISF-style safety content.
Anyway, above-and-beyond anything I say above, I think my top piece of advice is to have a 1-1 call with Xander (or more if you’ve spoken with him already). I think Xander is really good at this stuff and consistently made really good judgement calls in the process of building HAIST and MAIA, and I expect he’d be really helpful in helping you think through the same issues in your context at USC.
These are all fantastic questions! I’ll try to answer some of the ones I can. (Unfortunately a lot of the people who could answer the rest are pretty busy right now with EAGxBerkeley, getting set up for REMIX, etc., but I’m guessing that they’ll start having a chance to answer some of these in the coming days.)
Regarding the research program, I’m guessing there’s around 6-10 research projects ongoing, with between 1 and 3 students working on each; I’m guessing almost none of the participants have previous research experience. (Kuhan would have the actual numbers here.) This program just got started in late October, so certainly no published results yet.
I’m guessing the mentors are not all on the same page about how much of the value comes from doing object-level useful research vs. upskilling. My feeling is that it’s mostly upskilling, with the exception of a few projects where the mentor was basically taking on a RA for a project they were already working on full-time. In fact, when pitching projects, I explicitly disclaimed for some of them that I thought they were likely not useful for alignment (but would be useful for learning research skills and ML upskilling).
It sounds like in your situation, there’s a lack of experienced mentors. (Though I’ll note that a mentor spending ~1 hour per week meeting with a group sounds like plenty to me.) If that’s right, then I think I’d recommend focusing on ML upskilling programming instead of starting a research program. My thoughts here are: (1) I doubt participants will get much mileage out of working on projects that they came up with themselves, especially without mentors to help them shape their work; (2) poorly mentored research projects can be frustrating for the mentees, and might sour them on further engaging with your programming or AI safety as a whole; (3) ML upskilling programming seems almost as valuable to me and much easier to do well.
Regarding general member programming: for our weekly reading group, we pick readings week-by-week, usually based on someone messaging a group chat saying “I’d really love to read X this week.” (X was often something that had come out in the last week or so.) I don’t think this wasn’t an especially good way to do things, but we got lucky and it mostly worked out.
That said, I think most of the value here was from getting a bunch of aligned people in a room reading something and discussing with each other. If you don’t already have a lot of people sold on AI x-risk and with a background similar to having completed AGISF, I think it’d be better to run a more structured reading group rather than doing something like this.
Like we mentioned in the post, we think that we actually underinvested in developing programming for our members to participate in (instead putting slightly too much work into making the intro fellowship go well). Most of our full members were too busy for the research program, and the bar for facilitating for our intro fellowship was relatively high (other than Xander, all of our facilitators were PhD students or people who worked full-time on AIS). So the only real thing we had for full members were the weekly general member meetings and the retreats at the end of the semester.
I think my ordering would be
with social events not really mattering much to me, and facilitating not being an option for most of them, thanks to our wealth of over-qualified facilitators. I’m not sure how this should translate to your situation, sorry.
Regarding the intro fellowship, we hadn’t really considered MLSS at all, and probably we should have. I think we were approaching things from a frame separating our programming into things that require coding (ML upskilling) and things that don’t (AGISF), but this was potentially a mistake. The MLSS curriculum looks good, I agree that it seems better at getting people research-ready, and I’ll think about whether it makes sense to incorporate some of this stuff for next semester—thanks for this suggestion!
One dynamic to keep in mind is that when you advertise for an AI educational program, you’ll get a whole bunch of people who are excited about AI and don’t care much about the safety angle (it seems like lots of the people we attracted to our research program were like this). To some extent this is okay—it gives a chance to persuade people who would have otherwise gone into AI capabilities work! -- but I think it’s also worth trying not to spend resources teaching ML to people who will just go off and work in capabilities. One nice thing about AGISF is that it starts off with multiple weeks on safety, allowing people who aren’t interested in safety to self-select out before the technical material. (And the technical content is mostly stuff that I’m not worried is could advance capabilities anyway.) So if you’ve noticed that you have a lot of people sticking around to the end of your curriculum without really engaging with the safety angle, I might recommend front-loading some AGISF-style safety content.
Anyway, above-and-beyond anything I say above, I think my top piece of advice is to have a 1-1 call with Xander (or more if you’ve spoken with him already). I think Xander is really good at this stuff and consistently made really good judgement calls in the process of building HAIST and MAIA, and I expect he’d be really helpful in helping you think through the same issues in your context at USC.