these are the two main ways I would expect MATS to have impact: research output during the program and future research output/career trajectories of scholars.
We expect to achieve more impact through (2) than (1); per the theory of change section above, MATS’ mission is to expand the talent pipeline. Of course, we hope that scholars produce great research through the program, but often we are excited about scholars doing research (1) as a means for scholars tobecome better researchers (2). Other times, these goals are in tension. For example, some mentors explicitly de-emphasize research outputs, encouraging scholars to focus instead on pivoting frequently to develop better taste.
If you have your own reason to think (1) is comparably important to (2), I wonder if you also think providing mentees to researchers whose agendas you support is a similarly important source of MATS’ impact? From the Theory of Change section:
By mentoring MATS scholars, these senior researchers also benefit from research assistance
I would also suspect that #2 (finding/generating good researchers) is more valuable than #1 (generating or accelerating good research during the MATS program itself).
One problem with #2 is that it’s usually harder to evaluate and takes longer to evaluate. #2 requires projections, often over the course of years. #1 is still difficult to evaluate (what is “good alignment research” anyways?) but seems easier in comparison.
Also, I would expect correlations between #1 and #2. Like, one way to evaluate “how good are we doing at training researchers//who are the best researchers” is to ask “how good is the research they are producing//who produced the best research in this 3-month period?”
This process is (of course) imperfect. For example, someone might have great output because their mentor handed them a bunch of ready-to-go-projects, but the scholar didn’t actually have to learn the important skills of “forming novel ideas” or “figuring out how to prioritize between many different directions.”
But in general, I think it’s a pretty decent way to evaluate things. If someone has produced high-quality and original research during the MATS program, that sure does seem like a strong signal for their future potential. Likewise, in the opposite extreme, if during the entire summer cohort there were 0 instances of useful original work, that doesn’t necessarily mean something is wrong, but it would make me go “hmmm, maybe we should brainstorm possible changes to the program that could make it more likely that we see high-quality original output next time, and then we see how much those proposed changes trade-off against other desireada.”
(It seems quite likely to me that the MATS team has already considered all of this; just responding on the off-chance that something here is useful!)
We expect to achieve more impact through (2) than (1); per the theory of change section above, MATS’ mission is to expand the talent pipeline. Of course, we hope that scholars produce great research through the program, but often we are excited about scholars doing research (1) as a means for scholars to become better researchers (2). Other times, these goals are in tension. For example, some mentors explicitly de-emphasize research outputs, encouraging scholars to focus instead on pivoting frequently to develop better taste.
If you have your own reason to think (1) is comparably important to (2), I wonder if you also think providing mentees to researchers whose agendas you support is a similarly important source of MATS’ impact? From the Theory of Change section:
I would also suspect that #2 (finding/generating good researchers) is more valuable than #1 (generating or accelerating good research during the MATS program itself).
One problem with #2 is that it’s usually harder to evaluate and takes longer to evaluate. #2 requires projections, often over the course of years. #1 is still difficult to evaluate (what is “good alignment research” anyways?) but seems easier in comparison.
Also, I would expect correlations between #1 and #2. Like, one way to evaluate “how good are we doing at training researchers//who are the best researchers” is to ask “how good is the research they are producing//who produced the best research in this 3-month period?”
This process is (of course) imperfect. For example, someone might have great output because their mentor handed them a bunch of ready-to-go-projects, but the scholar didn’t actually have to learn the important skills of “forming novel ideas” or “figuring out how to prioritize between many different directions.”
But in general, I think it’s a pretty decent way to evaluate things. If someone has produced high-quality and original research during the MATS program, that sure does seem like a strong signal for their future potential. Likewise, in the opposite extreme, if during the entire summer cohort there were 0 instances of useful original work, that doesn’t necessarily mean something is wrong, but it would make me go “hmmm, maybe we should brainstorm possible changes to the program that could make it more likely that we see high-quality original output next time, and then we see how much those proposed changes trade-off against other desireada.”
(It seems quite likely to me that the MATS team has already considered all of this; just responding on the off-chance that something here is useful!)