Furthermore, I don’t think independent AI safety funding is that important anymore; models are smart enough now that most of the work to do in AI safety is directly working with them, most of that is happening at labs,
It might be the case that most of the quality weighted safety research involving working with large models is happening at labs, but I’m pretty skeptical that having this mostly happen at labs is the best approach and it seems like OpenPhil should be actively interested in building up a robust safety research ecosystem outside of labs.
(Better model access seems substantially overrated in its importance and large fractions of research can and should happen with just prompting or on smaller models. Additionally, at the moment, open weight models are pretty close to the best models.)
(This argument is also locally invalid at a more basic level. Just because this research seems to be mostly happening at large AI companies (which I’m also more skeptical of I think) doesn’t imply that this is the way it should be and funding should try to push people to do better stuff rather than merely reacting to the current allocation.)
Yeah, I think that’s a pretty fair criticism, but afaict that is the main thing that OpenPhil is still funding in AI safety? E.g. all the RFPs that they’ve been doing, I think they funded Jacob Steinhardt, etc. Though I don’t know much here; I could be wrong.
Wasn’t the relevant part of your argument like, “AI safety research outside of the labs is not that good, so that’s a contributing factor among many to it not being bad to lose the ability to do safety funding for governance work”? If so, I think that “most of OpenPhil’s actual safety funding has gone to building a robust safety research ecosystem outside of the labs” is not a good rejoinder to “isn’t there a large benefit to building a robust safety research ecosystem outside of the labs?”, because the rejoinder is focusing on relative allocations within “(technical) safety research”, and the complaint was about the allocation between “(technical) safety research” vs “other AI x-risk stuff”.
It might be the case that most of the quality weighted safety research involving working with large models is happening at labs, but I’m pretty skeptical that having this mostly happen at labs is the best approach and it seems like OpenPhil should be actively interested in building up a robust safety research ecosystem outside of labs.
(Better model access seems substantially overrated in its importance and large fractions of research can and should happen with just prompting or on smaller models. Additionally, at the moment, open weight models are pretty close to the best models.)
(This argument is also locally invalid at a more basic level. Just because this research seems to be mostly happening at large AI companies (which I’m also more skeptical of I think) doesn’t imply that this is the way it should be and funding should try to push people to do better stuff rather than merely reacting to the current allocation.)
Yeah, I think that’s a pretty fair criticism, but afaict that is the main thing that OpenPhil is still funding in AI safety? E.g. all the RFPs that they’ve been doing, I think they funded Jacob Steinhardt, etc. Though I don’t know much here; I could be wrong.
Wasn’t the relevant part of your argument like, “AI safety research outside of the labs is not that good, so that’s a contributing factor among many to it not being bad to lose the ability to do safety funding for governance work”? If so, I think that “most of OpenPhil’s actual safety funding has gone to building a robust safety research ecosystem outside of the labs” is not a good rejoinder to “isn’t there a large benefit to building a robust safety research ecosystem outside of the labs?”, because the rejoinder is focusing on relative allocations within “(technical) safety research”, and the complaint was about the allocation between “(technical) safety research” vs “other AI x-risk stuff”.