There should be points for how the organizations act wrt to legislation. In the SB 1047 bill that CAIS co-sponsored, we’ve noticed some AI companies to be much more antagonistic than others. I think is is probably a larger differentiator for an organization’s goodness or badness.
@Dan H are you able to say more about which companies were most/least antagonistic?
It is pretty plausible to me that AI control is quite easy
I think it depends on how you’re defining an “AI control success”. If success is defined as “we have an early transformative system that does not instantly kill us– we are able to get some value out of it”, then I agree that this seems relatively easy under the assumptions you articulated.
If success is defined as “we have an early transformative that does not instantly kill us and we have enough time, caution, and organizational adequacy to use that system in ways that get us out of an acute risk period”, then this seems much harder.
The classic race dynamic threat model seems relevant here: Suppose Lab A implements good control techniques on GPT-8, and then it’s trying very hard to get good alignment techniques out of GPT-8 to align a successor GPT-9. However, Lab B was only ~2 months behind, so Lab A feels like it needs to figure all of this out within 2 months. Lab B– either because it’s less cautious or because it feels like it needs to cut corners to catch up– either doesn’t want to implement the control techniques or it’s fine implementing the control techniques but it plans to be less cautious around when we’re ready to scale up to GPT-9.
I think it’s fine to say “the control agenda is valuable even if it doesn’t solve the whole problem, and yes other things will be needed to address race dynamics otherwise you will only be able to control GPT-8 for a small window of time before you are forced to scale up prematurely or hope that your competitor doesn’t cause a catastrophe.” But this has a different vibe than “AI control is quite easy”, even if that statement is technically correct.
(Also, please do point out if there’s some way in which the control agenda “solves” or circumvents this threat model– apologies if you or Ryan has written/spoken about it somewhere that I missed.)
Right now, I think one of the most credible ways for a lab to show its committment to safety is through its engagement with governments.
I didn’t mean to imply that a lab should automatically be considered “bad” if its public advocacy and its private advocacy differ.
However, when assessing how “responsible” various actors are, I think investigating questions relating to their public comms, engagement with government, policy proposals, lobbying efforts, etc would be valuable.
If Lab A had slightly better internal governance but lab B had better effects on “government governance”, I would say that lab B is more “responsible” on net.
@Zach Stein-Perlman, great work on this. I would be interested in you brainstorming some questions that have to do with the lab’s stances toward (government) AI policy interventions.
After a quick 5 min brainstorm, here are some examples of things that seem relevant:
I remember hearing that OpenAI lobbied against the EU AI Act– what’s up with that?
I heard a rumor that Congresspeople and their teams reached out to Sam/OpenAI after his testimony. They allegedly asked for OpenAI’s help to craft legislation around licensing, and then OpenAI refused. Is that true?
Sam said we might need an IAEA for AI at some point– what did he mean by this? At what point would he see that as valuable?
In general, what do labs think the US government should be doing? What proposals would they actively support or even help bring about? (Flagging ofc that there are concerns about actual and perceived regulatory capture, but there are also major advantages to having industry players support & contribute to meaningful regulation).
Senator Cory Booker recently asked Jack Clark something along the lines of “what is your top policy priority right now//what would you do if you were a Senator.” Jack responded with something along the lines of “I would make sure the government can deploy AI successfully. We need a testing regime to better understand risks, but the main risk is that we don’t use AI enough, and we need to make sure we stay at the cutting edge.” What’s up with that?
Why haven’t Dario and Jack made public statements about specific government interventions? Do they believe that there are some circumstances under which a moratorium would need to be implemented, labs would need to be nationalized (or internationalized), or something else would need to occur to curb race dynamics? (This could be asked to any of the lab CEOs/policy team leads– I don’t mean to be picking on Anthropic, though I think Sam/OpenAI have had more public statements here, and I think the other labs are scoring more poorly across the board//don’t fully buy into the risks in the first place.)
Big tech is spending a lot of money on AI lobbying. How much is each lab spending (this is something you can estimate with publicly available data), and what are they actually lobbying for/against?
I imagine there’s a lot more in this general category of “labs and how they are interacting with governments and how they are contributing to broader AI policy efforts”, and I’d be excited to see AI Lab Watch (or just you) dive into this more.
I can imagine this growing into the default reference that people use when talking about whether labs are behaving responsibly.
I hope that this resource is used as a measure of relative responsibleness, and this doesn’t get mixed up with absolute responsibleness. My understanding is that the resource essentially says, “here’s some things that would be good– let’s see how the labs compare on each dimension.” The resource is not saying “if a lab gets a score above X% on each metric, then we are quite confident that the lab will not cause an existential catastrophe.”
Moreover, my understanding is that the resource is not taking a position on whether or not it is “responsible”– in some absolute sense– for a lab to be scaling toward AGI in our current world. I see the resource as saying “conditional on a lab scaling toward AGI, are they doing so in a way that is relatively more/less responsible compared to the others that are scaling toward AGI.”
This might be a pedantic point, but I think it’s an important one to emphasize– a lab can score in 1st place and still present a risk to humanity that reasonable people would still deem unacceptable & irresponsible (or put differently, a lab can score in 1st place and still produce a catastrophe).
Agree with lots of this– a few misc thoughts [hastily written]:
I think the Overton Window frame ends up getting people to focus too much on the dimension “how radical is my ask”– in practice, things are usually much more complicated than this. In my opinion, a preferable frame is something like “who is my target audience and what might they find helpful.” If you’re talking to someone who makes it clear that they will not support X, it’s silly to keep on talking about X. But I think the “target audience first” approach ends up helping people reason in a more sophisticated way about what kinds of ideas are worth bringing up. As an example, in my experience so far, many policymakers are curious to learn more about intelligence explosion scenarios and misalignment scenarios (the more “radical” and “speculative” threat models).
I don’t think it’s clear that the more effective actors in DC tend to be those who look for small wins. Outside of the AIS community, there sure do seem to be a lot of successful organizations that take hard-line positions and (presumably) get a lot of their power/influence from the ideological purity that they possess & communicate. Whether or not these organizations end up having more or less influence than the more “centrist” groups is, in my view, not a settled question & probably varies a lot by domain. In AI safety in particular, I think my main claim is something like “pretty much no group– whether radical or centrist– has had tangible wins. When I look at the small set of tangible wins, it seems like the groups involved were across the spectrum of “reasonableness.”
The more I interact with policymakers, the more I’m updating toward something like “poisoning the well doesn’t come from having radical beliefs– poisoning the well comes from lamer things like being dumb or uninformed, wasting peoples’ time, not understanding how the political process works, not having tangible things you want someone to do, explaining ideas poorly, being rude or disrespectful, etc.” I’ve asked ~20-40 policymakers (outside of the AIS bubble) things like “what sorts of things annoy you about meetings” or “what tends to make meetings feel like a waste of your time”, and no one ever says “people come in with ideas that are too radical.” The closest thing I’ve heard is people saying that they dislike it when groups fail to understand why things aren’t able to happen (like, someone comes in thinking their idea is great, but then they fail to understand that their idea needs approval from committee A and appropriations person B and then they’re upset about why things are moving slowly). It seems to me like many policy folks (especially staffers and exec branch subject experts) are genuinely interested in learning more about the beliefs and worldviews that have been prematurely labeled as “radical” or “unreasonable” (or perhaps such labels were appropriate before chatGPT but no longer are).
A reminder that those who are opposed to regulation have strong incentives to make it seem like basically-any-regulation is radical/unreasonable. An extremely common tactic is for industry and its allies to make common-sense regulation seem radical/crazy/authoritarian & argue that actually the people proposing strong policies are just making everyone look bad & argue that actually we should all rally behind [insert thing that isn’t a real policy.] (I admit this argument is a bit general, and indeed I’ve made it before, so I won’t harp on it here. Also I don’t think this is what Trevor is doing– it is indeed possible to raise serious discussions about “poisoning the well” even if one believes that the cultural and economic incentives disproportionately elevate such points).
In the context of AI safety, it seems to me like the most high-influence Overton Window moves have been positive– and in fact I would go as far as to say strongly positive. Examples that come to mind include the CAIS statement, FLI pause letter, Hinton leaving Google, Bengio’s writings/speeches about rogue AI & loss of control, Ian Hogarth’s piece about the race to god-like AI, and even Yudkowsky’s TIME article.
I think some of our judgments here depend on underlying threat models and an underlying sense of optimism vs. pessimism. If one things that labs making voluntary agreements/promises and NIST contributing to the development of voluntary standards are quite excellent ways to reduce AI risk, then the groups that have helped make this happen deserve a lot of credit. If one thinks that much more is needed to meaningfully reduce xrisk, then the groups that are raising awareness about the nature of the problem, making high-quality arguments about threat models, and advocating for stronger policies deserve a lot of credit.
I agree that more research on this could be useful. But I think it would be most valuable to focus less on “is X in the Overton Window” and more on “is X written/explained well and does it seem to have clear implications for the target stakeholders?”
I’m not sure who you’ve spoken to, but at least among the AI policy people who I talk to regularly (which admittedly is a subset of people who I think are doing the most thoughtful/serious work), I think nearly all of them have thought about ways in which regulation + regulatory capture could be net negative. At least to the point of being able to name the relatively “easy” ways (e.g., governments being worse at alignment than companies).
I continue to think people should be forming alliances with those who share similar policy objectives, rather than simply those who belong in the “I believe xrisk is a big deal” camp. I’ve seen many instances in which the “everyone who believes xrisk is a big deal belongs to the same camp” mentality has been used to dissuade people from communicating their beliefs, communicating with policymakers, brainstorming ideas that involve coordination with other groups in the world, disagreeing with the mainline views held by a few AIS leaders, etc.
The cultural pressures against policy advocacy have been so strong that it’s not surprising to see folks say things like “perhaps our groups are no longer natural allies” now that some of the xrisk-concerned people are beginning to say things like “perhaps the government should have more of a say in how AGI development goes than in status quo, where the government has played ~0 role and ~all decisions have been made by private companies.”
Perhaps there’s a multiverse out there in which the AGI community ended up attracting govt natsec folks instead of Bay Area libertarians, and the cultural pressures are flipped. Perhaps in that world, the default cultural incentives pushed people heavily brainstorming ways that markets and companies could contribute meaningfully to the AGI discourse, and the default position for the “AI risk is a big deal” camp was “well obviously the government should be able to decide what happens and it would be ridiculous to get companies involved– don’t be unilateralist by going and telling VCs about this stuff.”
I bring up this (admittedly kinda weird) hypothetical to point out just how skewed the status quo is. One might generally be wary of government overinvolvement in regulating emerging technologies yet still recognize that some degree of regulation is useful, and that position would likely still push them to be in the “we need more regulation than we currently have” camp.
As a final note, I’ll point out to readers less familiar with the AI policy world that serious people are proposing lots of regulation that is in between “status quo with virtually no regulation” and “full-on pause.” Some of my personal favorite examples include: emergency preparedness (akin to the OPPR), licensing (see Romney), reporting requirements, mandatory technical standards enforced via regulators, and public-private partnerships.
I’m interested in writing out somewhat detailed intelligence explosion scenarios. The goal would be to investigate what kinds of tools the US government would have to detect and intervene in the early stages of an intelligence explosion.
If you know anyone who has thought about these kinds of questions, whether from the AI community or from the US government perspective, please feel free to reach out via LessWrong.
To what extent would the organization be factoring in transformative AI timelines? It seems to me like the kinds of questions one would prioritize in a “normal period” look very different than the kinds of questions that one would prioritize if they place non-trivial probability on “AI may kill everyone in <10 years” or “AI may become better than humans on nearly all cognitive tasks in <10 years.”
I ask partly because I personally would be more excited of a version of this that wasn’t ignoring AGI timelines, but I think a version of this that’s not ignoring AGI timelines would probably be quite different from the intellectual spirit/tradition of FHI.
More generally, perhaps it would be good for you to describe some ways in which you expect this to be different than FHI. I think the calling it the FHI of the West, the explicit statement that it would have the intellectual tradition of FHI, and the announcement right when FHI dissolves might make it seem like “I want to copy FHI” as opposed to “OK obviously I don’t want to copy it entirely I just want to draw on some of its excellent intellectual/cultural components.” If your vision is the latter, I’d find it helpful to see a list of things that you expect to be similar/different.)
I would strongly suggest considering hires who would be based in DC (or who would hop between DC and Berkeley). In my experience, being in DC (or being familiar with DC & having a network in DC) is extremely valuable for being able to shape policy discussions, know what kinds of research questions matter, know what kinds of things policymakers are paying attention to, etc.
I would go as far as to say something like “in 6 months, if MIRI’s technical governance team has not achieved very much, one of my top 3 reasons for why MIRI failed would be that they did not engage enough with DC people//US policy people. As a result, they focused too much on questions that Bay Area people are interested in and too little on questions that Congressional offices and executive branch agencies are interested in. And relatedly, they didn’t get enough feedback from DC people. And relatedly, even the good ideas they had didn’t get communicated frequently enough or fast enough to relevant policymakers. And relatedly… etc etc.”
I do understand this trades off against everyone being in the same place, which is a significant factor, but I think the cost is worth it.
I do think evaporative cooling is a concern, especially if everyone (or a very significant amount) of people left. But I think on the margin more people should be leaving to work in govt.
I also suspect that a lot of systemic incentives will keep a greater-than-optimal proportion of safety-conscious people at labs as opposed to governments (labs pay more, labs are faster and have less bureaucracy, lab people are much more informed about AI, labs are more “cool/fun/fast-paced”, lots of govt jobs force you to move locations, etc.)
I also think it depends on the specific lab– EG in light of the recent OpenAI departures, I suspect there’s a stronger case for staying at OpenAI right now than for DeepMind or Anthropic.
I think now is a good time for people at labs to seriously consider quitting & getting involved in government/policy efforts.
I don’t think everyone should leave labs (obviously). But I would probably hit a button that does something like “everyone at a lab governance team and many technical researchers spend at least 2 hours thinking/writing about alternative options they have & very seriously consider leaving.”
My impression is that lab governance is much less tractable (lab folks have already thought a lot more about AGI) and less promising (competitive pressures are dominating) than government-focused work.
I think governments still remain unsure about what to do, and there’s a lot of potential for folks like Daniel K to have a meaningful role in shaping policy, helping natsec folks understand specific threat models, and raising awareness about the specific kinds of things governments need to do in order to mitigate risks.
There may be specific opportunities at labs that are very high-impact, but I think if someone at a lab is “not really sure if what they’re doing is making a big difference”, I would probably hit a button that allocates them toward government work or government-focused comms work.
I think now is a good time for people at labs to seriously consider quitting & getting involved in government/policy efforts.
I don’t think everyone should leave labs (obviously). But I would probably hit a button that does something like “everyone at a lab governance team and many technical researchers spend at least 2 hours thinking/writing about alternative options they have & very seriously consider leaving.”
My impression is that lab governance is much less tractable (lab folks have already thought a lot more about AGI) and less promising (competitive pressures are dominating) than government-focused work.
I think governments still remain unsure about what to do, and there’s a lot of potential for folks like Daniel K to have a meaningful role in shaping policy, helping natsec folks understand specific threat models, and raising awareness about the specific kinds of things governments need to do in order to mitigate risks.
There may be specific opportunities at labs that are very high-impact, but I think if someone at a lab is “not really sure if what they’re doing is making a big difference”, I would probably hit a button that allocates them toward government work or government-focused comms work.
Written on a Slack channel in response to discussions about some folks leaving OpenAI.
I think this should be broken down into two questions:
Before the EO, if we were asked to figure out where this kind of evals should happen, what institution would we pick & why?
After the EO, where does it make sense for evals-focused people to work?
I think the answer to #1 is quite unclear. I personally think that there was a strong case that a natsec-focused USAISI could have been given to DHS or DoE or some interagency thing. In addition to the point about technical expertise, it does seem relatively rare for Commerce/NIST to take on something that is so natsec-focused.
But I think the answer to #2 is pretty clear. The EO clearly tasks NIST with this role, and now I think our collective goal should be to try to make sure NIST can execute as effectively as possible. Perhaps there will be future opportunities to establish new places for evals work, alignment work, risk monitoring and forecasting work, emergency preparedness planning, etc etc. But for now, whether we think it was the best choice or not, NIST/USAISI are clearly the folks who are tasked with taking the lead on evals + standards.
I’m excited to see how Paul performs in the new role. He’s obviously very qualified on a technical level, and I suspect he’s one of the best people for the job of designing and conducting evals.
I’m more uncertain about the kind of influence he’ll have on various AI policy and AI national security discussions. And I mean uncertain in the genuine “this could go so many different ways” kind of way.
Like, it wouldn’t be particularly surprising to me if any of the following occurred:
Paul focuses nearly all of his efforts on technical evals and doesn’t get very involved in broader policy conversations
Paul is regularly asked to contribute to broader policy discussions, and he advocates for RSPs and other forms of voluntary commitments.
Paul is regularly asked to contribute to broader policy discussions, and he advocates for requirements that go beyond voluntary commitments and are much more ambitious than what he advocated for when he was at ARC.
Paul is regularly asked to contribute to broader policy discussions, and he’s not very good at communicating his beliefs in ways that are clear/concise/policymaker-friendly, so his influence on policy discussions is rather limited.
Paul [is/isn’t] able to work well with others who have very different worldviews and priorities.
Personally, I see this as a very exciting opportunity for Paul to form an identity as a leader in AI policy. I’m guessing the technical work will be his priority (and indeed, it’s what he’s being explicitly hired to do), but I hope he also finds ways to just generally improve the US government’s understanding of AI risk and the likelihood of implementing reasonable policies. On the flipside, I hope he doesn’t settle for voluntary commitments (especially as the Overton Window shifts) & I hope he’s clear/open about the limitations of RSPs.
More specifically, I hope he’s able to help policymakers reason about a critical question: what do we do after we’ve identified models with (certain kinds of) dangerous capabilities? I think the underlying logic behind RSPs could actually be somewhat meaningfully applied to USG policy. Like, I think we would be in a safer world if the USG had an internal understanding of ASL levels, took seriously the possibility of various dangerous capabilities thresholds being crossed, took seriously the idea that AGI/ASI could be developed soon, and had preparedness plans in place that allowed them to react quickly in the event of a sudden risk.
Anyways, a big congratulations to Paul, and definitely some evidence that the USAISI is capable of hiring some technical powerhouses.
I’ll admit I have only been loosely following the control stuff, but FWIW I would be excited about a potential @peterbarnett & @ryan_greenblatt dialogue in which you two to try to identify & analyze any potential disagreements. Example questions:
What is the most capable system that you think we are likely to be able to control?
What kind of value do you think we could get out of such a system?
To what extent do you expect that system to be able to produce insights that help us escape the acute risk period (i.e., get out of a scenario where someone else can come along and build a catastrophe-capable system without implementing control procedures or someone else comes along and scales to the point where the control procedures are no longer sufficient)
Here are three possible scenarios:
Scenario 1, Active Lying– Anthropic staff were actively spreading the idea that they would not push the frontier.
Scenario 2, Allowing misconceptions to go unchecked– Anthropic staff were aware that many folks in the AIS world thought that Anthropic had committed to not pushing the frontier, and they allowed this misconception to go unchecked, perhaps because they realized that it was a misconception that favored their commercial/competitive interests.
Scenario 3, Not being aware– Anthropic staff were not aware that many folks had this belief. Maybe they heard it once or twice but it never really seemed like a big deal.
Scenario 1 is clearly bad. Scenarios 2 and 3 are more interesting. To what extent does Anthropic have the responsibility to clarify misconceptions (avoid scenario 2) and even actively look for misconceptions (avoid scenario 3)?
I expect this could matter tangibly for discussions of RSPs. My opinion is that the Anthropic RSP is written in such a way that readers can come away with rather different expectations of what kinds of circumstances would cause Anthropic to pause/resume.
It wouldn’t be very surprising to me if we end up seeing a situation where many readers say “hey look, we’ve reached an ASL-3 system, so now you’re going to pause, right?” And then Anthropic says “no no, we have sufficient safeguards– we can keep going now.” And then some readers say “wait a second– what? I’m pretty sure you committed to pausing until your safeguards were better than that.” And then Anthropic says “no… we never said exactly what kinds of safeguards we would need, and our leadership’s opinion is that our safeguards are sufficient, and the RSP allows leadership to determine when it’s fine to proceed.”
In this (hypothetical) scenario, Anthropic never lied, but it benefitted from giving off a more cautious impression, and it didn’t take steps to correct this impression.
I think avoiding these kinds of scenarios requires some mix of:
Clear, specific falsifiable statements on behalf of labs.
Some degree of proactive attempts to identify and alleviate misconceptions.
One counterargument is something like “Anthropic is a company, and there are lots of things to do, and this is is demanding an unusually high amount of attention-to-detail and proactive communication that is not typically expected of companies.” To which my response is something like “yes, but I think it’s reasonable to hold companies to such standards if they wish to develop AGI. I think we ought to hold Anthropic and other labs to this standard, especially insofar as they want the benefits associated with being perceived as the kind of safety-conscious lab that refuses to push the frontier or commits to scaling policies that include tangible/concrete plans to pause.”
I like these examples. One thing I’ll note, however, is that I think the “warning shot discourse” on LW tends to focus on warning shots that would be convincing to a LW-style audience.
If the theory of the change behind the warning shot requires LW-types (for example, folks at OpenAI/DeepMind/Anthropic who are relatively familiar with AGI xrisk arguments) to become concerned, this makes sense.
But usually, when I think about winning worlds that involve warning shots, I think about government involvement as the main force driving coordination, an end to race dynamics, etc.
[Caveating that my models of the USG and natsec community are still forming, so epistemic status is quite uncertain for the rest of this message, but I figure some degree of speculation could be helpful anyways].
I expect the kinds of warning shots that would be concerning to governments/national security folks will look quite different than the kinds of warning shots that would be convincing to technical experts.
LW-style warning shots tend to be more– for a lack of a better term– rational. They tend to be rooted in actual threat models (e.g., we understand that if an AI can copy its weights, it can create tons of copies and avoid being easily turned off, and we also understand that its general capabilities are sufficiently strong that we may be close to an intelligence explosion or highly capable AGI).
In contrast, without this context, I don’t think that “we caught an AI model copying its weights” would necessarily be a warning shot for USG/natsec folks. It could just come across as “oh something weird happened but then the company caught it and fixed it.” Instead, I suspect the warning shots that are most relevant to natsec folks might be less “rational”, and by that I mean “less rooted in actual AGI threat models but more rooted in intuitive things that seem scary.”
Examples of warning shots that I expect USG/natsec people would be more concerned about:
An AI system can generate novel weapons of mass destruction
Someone uses an AI system to hack into critical infrastructure or develop new zero-days that impress folks in the intelligence community.
A sudden increase in the military capabilities of AI
These don’t relate as much to misalignment risk or misalignment-focused xrisk threat models. As a result, a disadvantage of these warning shots is that it may be harder to make a convincing case for interventions that focus specifically on misalignment. However, I think they are the kinds of things that might involve a sudden increase in the attention that the USG places on AI/AGI, the amount of resources it invests into understanding national security threats from AI, and its willingness to take major actions to intervene in the current AI race.
As such, in addition to asking questions like “what is the kind of warning shot that would convince me and my friends that we have something dangerous”, I think it’s worth separately asking “what is the kind of warning shot that would convince natsec folks that something dangerous or important is occurring, regardless of whether or not it connects neatly to AGI risk threat models.”
My impression is that the policy-relevant warning shots will be the most important ones to be prepared for, and the community may (for cultural/social/psychological reasons) be focusing too little effort on trying to prepare for these kinds of “irrational” warning shots.
I haven’t followed this in great detail, but I do remember hearing from many AI policy people (including people at the UKAISI) that such commitments had been made.
It’s plausible to me that this was an example of “miscommunication” rather than “explicit lying.” I hope someone who has followed this more closely provides details.
But note that I personally think that AGI labs have a responsibility to dispel widely-believed myths. It would shock me if OpenAI/Anthropic/Google DeepMind were not aware that people (including people in government) believed that they had made this commitment. If you know that a bunch of people think you committed to sending them your models, and your response is “well technically we never said that but let’s just leave it ambiguous and then if we defect later we can just say we never committed”, I still think it’s fair for people to be disappointed in the labs.
(I do think this form of disappointment should not be conflated with “you explicitly said X and went back on it”, though.)