Indirectly, because 90 papers seems like a tiny number, vs. what got published on arxiv during that same time interval. (Depending on how one counts) I wouldn’t be surprised if there were > 90 papers from outside the labs even looking only at the unlearning category.
Counting the number of papers isn’t going to be a good strategy.
I do think total research outside of labs looks competitive with research from labs and probably research done outside of labs has produced more differential safety progress in total.
I also think open weight models are probably good so far in terms of making AI more likely to go well (putting aside norms and precedents of releases), though I don’t think this is necessarily implied from “more research happens outside of labs”.
probably research done outside of labs has produced more differential safety progress in total
To be clear — this statement is consistent with companies producing way more safety research than non-companies, if companies also produce even way more capabilities progress than non-companies? (Which I would’ve thought is the case, though I’m not well-informed. Not sure if “total research outside of labs look competitive with research from labs” is meant to deny this possibility, or if you’re only talking about safety research there.)
I’m just talking about research intended to be safety/safety-adjacent. As in, of this research, what has the quality weighted differential safety progress been.
Probably the word “differential” was just a mistake.
My main takeaway would be that this seems like quite strong evidence towards the view expressed in https://www.beren.io/2023-11-05-Open-source-AI-has-been-vital-for-alignment/, that most safety research doesn’t come from the top labs.
How does this paper suggest “that most safety research doesn’t come from the top labs”?
Indirectly, because 90 papers seems like a tiny number, vs. what got published on arxiv during that same time interval. (Depending on how one counts) I wouldn’t be surprised if there were > 90 papers from outside the labs even looking only at the unlearning category.
Counting the number of papers isn’t going to be a good strategy.
I do think total research outside of labs looks competitive with research from labs and probably research done outside of labs has produced more
differentialsafety progress in total.I also think open weight models are probably good so far in terms of making AI more likely to go well (putting aside norms and precedents of releases), though I don’t think this is necessarily implied from “more research happens outside of labs”.
To be clear — this statement is consistent with companies producing way more safety research than non-companies, if companies also produce even way more capabilities progress than non-companies? (Which I would’ve thought is the case, though I’m not well-informed. Not sure if “total research outside of labs look competitive with research from labs” is meant to deny this possibility, or if you’re only talking about safety research there.)
I’m just talking about research intended to be safety/safety-adjacent. As in, of this research, what has the quality weighted differential safety progress been.
Probably the word “differential” was just a mistake.