Co-Executive Director at ML Alignment & Theory Scholars Program (2022-present)
Co-Founder & Board Member at London Initiative for Safe AI (2023-present)
Manifund Regrantor (2023-present) | RFPs here
Advisor, Catalyze Impact (2023-present) | ToC here
Advisor, AI Safety ANZ (2024-present)
Ph.D. in Physics at the University of Queensland (2017-2023)
Group organizer at Effective Altruism UQ (2018-2021)
Give me feedback! :)
I’m open to this argument, but I’m not sure it’s true under the Trump administration.
Technical AI alignment/control is still impactful; don’t go all-in on AI gov!
Liability incentivizes safeguards, even absent regulation;
Cheaper, more effective safeguards make it easier for labs to meet safety standards;
Concrete, achievable safeguards give regulation teeth.
Not sure this is interesting to anyone, but I compiled Zillow’s data on 2021-2025 Berkeley average rent prices recently, to help with rent negotiation. I did not adjust for inflation; these are the raw averages at each time.
I definitely think that people should not look at my estimates and say “here is a good 95% confidence interval upper bound of the number of employees in the AI safety ecosystem.” I think people should look at my estimates and say “here is a good 95% confidence interval lower bound of the number of employees in the AI safety ecosystem,” because you can just add up the names. I.e., even if there might be 10x the number of employees as I estimated, I’m at least 95% confident that there are more than my estimate obtained by just counting names (obviously excluding the 10% fudge factor).
So, conduct a sensitivity analysis on the definite integral with respect to choices of integration bounds? I’m not sure this level of analysis is merited given the incomplete data and unreliable estimation methodology for the number of independent researchers. Like, I’m not even confident that the underlying distribution is a power law (instead of, say, a composite of power law and lognormal distributions, or a truncated power law), and the value of seems very sensitive to data in the vicinity, so I wouldn’t want to rely on this estimate except as a very crude first pass. I would support an investigation into the number of independent researchers in the ecosystem, which I would find useful.
By “upper bound”, I meant “upper bound on the definite integral ”. I.e., for the kind of hacky thing I’m doing here, the integral is very sensitive to the choice of bounds . For example, the integral does not converge for . I think all my data here should be treated as incomplete and all my calculations crude estimates at best.
I edited the original comment to say ” might be a bad upper bound” for clarity.
It’s also worth noting that almost all of these roles are management, ML research, or software engineering; there are very few operations, communications, non-ML research, etc. roles listed, implying that these roles are paid significantly less.
Apparently the headcount for US corporations follows a power-law distribution, apart from mid-sized corporations, which fit a lognormal distribution better. I fit a power law distribution to the data (after truncating all datapoints with over 40 employees, which created a worse fit), which gave . This seems to imply that there are ~400 independent AI safety researchers (though note that is probability density function and this estimate might be way off); Claude estimates 400-600 for comparison. Integrating this distribution over gives ~1400 (2 s.f.) total employees working on AI safety or safety-adjacent work ( might be a bad upper bound, as the largest orgs have <100 employees).
I decided to exclude OpenAI’s nonprofit salaries as I didn’t think they counted as an “AI safety nonprofit” and their highest paid current employees are definitely employed by the LLC. I decided to include Open Philanthropy’s nonprofit employees, despite the fact that their most highly compensated employees are likely those under the Open Philanthropy LLC.
As part of MATS’ compensation reevaluation project, I scraped the publicly declared employee compensations from ProPublica’s Nonprofit Explorer for many AI safety and EA organizations (data here) in 2019-2023. US nonprofits are required to disclose compensation information for certain highly paid employees and contractors on their annual Form 990 tax return, which becomes publicly available. This includes compensation for officers, directors, trustees, key employees, and highest compensated employees earning over $100k annually. Therefore, my data does not include many individuals earning under $100k, but this doesn’t seem to affect the yearly medians much, as the data seems to follow a lognormal distribution, with mode ~$178k in 2023, for example.
I generally found that AI safety and EA organization employees are highly compensated, albeit inconsistently between similar-sized organizations within equivalent roles (e.g., Redwood and FAR AI). I speculate that this is primarily due to differences in organization funding, but inconsistent compensation policies may also play a role.
I’m sharing this data to promote healthy and fair compensation policies across the ecosystem. I believe that MATS salaries are quite fair and reasonably competitive after our recent salary reevaluation, where we also used Payfactors HR market data and advertised salary ranges of roles at Bay Area AI safety organizations for comparison. If anyone wants to do a more detailed study of the data, I highly encourage this!
Thanks! I wasn’t sure whether to include Simplex, or the entire Obelisk team at Astera (which Simplex is now part of), or just exclude these non-scaling lab hybrid orgs from the count (Astera does neuroscience too).
I counted total employees for most orgs. In the spreadsheet I linked, I didn’t include an estimate for total GDM headcount, just that of the AI Safety and Alignment Team.
Most of the staff at AE Studio are not working on alignment, so I don’t think it counts.
I was very inclusive. I looked at a range of org lists, including those maintained by 80,000 Hours and AISafety.com.
Assuming that I missed 10% of orgs, this gives a rough estimate for the total number of FTEs working on AI safety or adjacent work at ~1000, not including students or faculty members. This is likely an underestimate, as there are a lot of AI safety-adjacent orgs in areas like AI security and robustness.
I did a quick inventory on the employee headcount at AI safety and safety-adjacent organizations. The median AI safety org has 10 8 employees. I didn’t include UK AISI, US AISI, CAIS, and the safety teams at Anthropic, GDM, OpenAI, and probably more, as I couldn’t get accurate headcount estimates. I also didn’t include “research affiliates” or university students in the headcounts for academic labs. Data here. Let me know if I missed any orgs!
I expect mech interp to be particularly easy to automate at scale. If mech interp has capabilities externalities (e.g., uncovering useful learned algorithms or “retargeting the search”), this could facilitate rapid performance improvements.
It seems plausible to me that if AGI progress becomes strongly bottlenecked on architecture design or hyperparameter search, a more “genetic algorithm”-like approach will follow. Automated AI researchers could run and evaluate many small experiments in parallel, covering a vast hyperparameter space. If small experiments are generally predictive of larger experiments (and they seem to be, a la scaling laws) and model inference costs are cheap enough, this parallelized approach might be be 1) computationally affordable and 2) successful at overcoming the architecture bottleneck.
Apr 18, 11:59 pm PT :)
Makes you wonder if there’s some 4D chess going on here. Occam’s razor suggests otherwise, though. And if true, this seems wholly irresponsible, given that AI risk skeptics can just point to this situation as an example that “even if we do no safety testing/guardrails, it’s not that bad! It just offends a few people.” It seems hard to say which direction this will impact SB 53, for example.