Your third paragraph mentions “all AI company staff” and the last refers to “risk evaluators” (i.e. “everyone within these companies charged with sounding the alarm”). Are these groups roughly the same or is the latter subgroup significantly smaller?
I personally think it’s most important to have at least some technical employees who have the knowledge/expertise to evaluate the actual situation, and who also have the power to do something about it. I’d want this to include some people who’s primary job is more like “board member” and some people who’s primary job is more like “alignment and/or capabilities researcher.”
But there is a sense in which I’d feel way more comfortable if all technical AI employees (alignment and capabilities), and policy / strategy people, didn’t have profit equity, so they didn’t have inceptive to optimize against what was safe. So, there’s just a lot of eyes on the problem, and the overall egregore steering the company has one fewer cognitive distortion to manage.
This might be too expensive (OpenAI and Anthropic have a lot of money, but, like, that doesn’t mean they can just double everyone’s salary)
An idea that occurs to me is construct “windfall equity”, which only pays out if the company or world generates AI that is safely, massively improving the world.
I think the latter group is is much smaller. I’m not sure who exactly has most influence over risk evaluation, but the most obvious examples are company leadership and safety staff/red-teamers. From what I hear, even those currently receive equity (which seems corroborated by job listings, e.g. Anthropic, DeepMind, OpenAI).
Your third paragraph mentions “all AI company staff” and the last refers to “risk evaluators” (i.e. “everyone within these companies charged with sounding the alarm”). Are these groups roughly the same or is the latter subgroup significantly smaller?
I personally think it’s most important to have at least some technical employees who have the knowledge/expertise to evaluate the actual situation, and who also have the power to do something about it. I’d want this to include some people who’s primary job is more like “board member” and some people who’s primary job is more like “alignment and/or capabilities researcher.”
But there is a sense in which I’d feel way more comfortable if all technical AI employees (alignment and capabilities), and policy / strategy people, didn’t have profit equity, so they didn’t have inceptive to optimize against what was safe. So, there’s just a lot of eyes on the problem, and the overall egregore steering the company has one fewer cognitive distortion to manage.
This might be too expensive (OpenAI and Anthropic have a lot of money, but, like, that doesn’t mean they can just double everyone’s salary)
An idea that occurs to me is construct “windfall equity”, which only pays out if the company or world generates AI that is safely, massively improving the world.
I think the latter group is is much smaller. I’m not sure who exactly has most influence over risk evaluation, but the most obvious examples are company leadership and safety staff/red-teamers. From what I hear, even those currently receive equity (which seems corroborated by job listings, e.g. Anthropic, DeepMind, OpenAI).