A lot of AI governance folks primarily do research. They rarely engage with policymakers directly, and they spend much of their time reading and writing papers.
This was even more true before the release of GPT-4 and the recent wave of interest in AI policy. Before GPT-4, many people believed “you will look weird/crazy if you talk to policymakers about AI extinction risk.” It’s unclear to me how true this was (in a genuine “I am confused about this & don’t think I have good models of this” way). Regardless, there has been an update toward talking to policymakers about AI risk now that AI risk is a bit more mainstream.
My own opinion is that, even after this update toward policymaker engagement, the community as a whole is still probably overinvested in research and underinvested in policymaker engagement/outreach. (Of course, the two can be complimentary, and the best outreach will often be done by people who have good models of what needs to be done & can present high-quality answers to the questions that policymakers have).
Among the people who do outreach/policymaker engagement, my impression is that there has been more focus on the executive branch (and less on Congress/congressional staffers). The main advantage is that the executive branch can get things done more quickly than Congress. The main disadvantage is that Congress is often required (or highly desired) to make “big things” happen (e.g., setting up a new agency or a licensing regime).
the community as a whole is still probably overinvested in research and underinvested in policymaker engagement/outreach.
My prediction is that the AI safety community will overestimate the difficulty of policymaker engagement/outreach.
I think that the AI safety community has quickly and accurately taken social awkwardness and nerdiness into account, and factored that out of the equation. However, they will still overestimate the difficulty of policymaker outreach, on the basis that policymaker outreach requires substantially above-average sociability and personal charisma.
Even among the many non-nerd extroverts in the AI safety community, who have above average or well above average social skills (e.g. ~80th or 90th percentile), the ability to do well in policy requires an extreme combination of traits that produce intense charismatic competence, such the traits required for as a sense of humor near the level of a successful professional comedian (e.g. ~99th or 99.9th percentile). This is because the policy environment, like corporate executives, selects for charismatic extremity.
Because people who are introspective or think about science at all are very rarely far above the 90th percentile for charisma, even if only the obvious natural extroverts are taken into account, the AI safety community will overestimate the difficulty of policymaker outreach.
I’m not sure I understand the direction of reasoning here. Overestimating the difficulty would mean that it will actually be easier than they think, which would be true if they expected a requirement of high charisma but the requirement were actually absent, or would be true if the people who ended up doing it were of higher charisma than the ones making the estimate. Or did you mean underestimating the difficulty?
I should have made it more clear at the beginning.
AI governance successfully filters out the nerdy people
They see that they’re still having a hard time finding their way to the policymakers with influence (e.g. what Akash was doing, meeting people in order to meet more people through them).
They conclude that the odds of success are something like ~30% or any other number.
I think that they would be off by something like 10, so it would actually be ~40%, because factoring out the nerds still leaves you with the people at the 90th percentile of Charisma and you need people at the 99th percentile. They might be able to procure those people.
This is because I predict that people at the 99th percentile of Charisma are underrepresented in AI safety, even if you only look at the non-nerds.
Among the people who do outreach/policymaker engagement, my impression is that there has been more focus on the executive branch (and less on Congress/congressional staffers).
That makes sense and sounds sensible, at least pre-ChatGPT.
Modern congressional staffers are the product of Goodhart’s law; ~50-100 years ago, they were the ones that ran congress de-facto, so all the businessmen and voters wanted to talk to them, so the policymaking ended up moving elsewhere. Just like what happened with congressmen themselves ~100-150 years ago. Congressional staffers today primarily take constituent calls from voters, and make interest groups think they’re being listened to. Akash’s accomplishments came from wading through that bullshit, meeting people through people until he managed to find some gems.
Most policymaking today is called in from outside, with lobbyists having the domain-expertise needed to write the bills, and senior congressional staffers (like the legislative directors and legislative assistants here) overseeing the process, usually without getting very picky about the details.
It’s not like congressmembers have no power, but they’re just one part of what’s called an “Iron triangle”, the congressional lawmakers, the executive branch bureaucracies (e.g. FDA, CDC, DoD, NSA), and the private sector companies (e.g. Walmart, Lockheed, Microsoft, Comcast), with the lobbyists circulating around the three, negotiating and cutting deals between them. It’s incredibly corrupt and always has been, but not all-crushingly corrupt like African governments. It’s like the Military Industrial Complex, except that’s actually a bad example because congress is increasingly out of the loop de-facto on foreign policy (most structures are idiosyncratic, because the fundamental building block is people who are thinking of ways to negotiate backdoor deals).
People in the executive branch/bureaucracies like the DoD have more power on interesting things like foreign policy, Congress is more powerful for things that have been entrenched for decades like farming policy. Think tank people have no power but they’re much less stupid and have domain expertise and are often called up to help write bills instead of lobbyists.
I don’t know how AI policy is made in Congress, I jumped ship from domestic AI policy to foreign AI policy 3.5 years ago in order to focus more on the incentives from the US-China angle, Akash is the one to ask about where AI policymaking happens in congress, as he was the one actually there deep in the maze (maybe via DM because he didn’t describe it in this post).
I strongly recommend people talking to John Wentworth about AI policy, even if he doesn’t know much at first; after looking at Wentworth’s OpenAI dialog, he’s currently my top predicted candidate for “person who starts spending 2 hours a week thinking about AI policy instead of technical alignment, and thinks up galaxy brained solutions that break the stalemates that vexed the AI policy people for years”.
Most don’t do policy at all. Many do research. Since you’re incredulous, here are some examples of great AI governance research (which don’t synergize much with talking to policymakers):
I mean, those are all decent projects, but I would call zero of them “great”. Like, the whole appeal of governance as an approach to AI safety is that it’s (supposed to be) bottlenecked mainly on execution, not on research. None of the projects you list sound like they’re addressing an actual rate-limiting step to useful AI governance.
(I disagree. Indeed, until recently governance people had very few policy asks for government.)
Did that change because people finally finished doing enough basic strategy research to know what policies to ask for?
It didn’t seem like that to me. Instead, my impression was that it was largely triggered by ChatGPT and GPT4 making the topic more salient, and AI safety feeling more inside the Overton window. So there were suddenly a bunch of government people asking for concrete policy suggestions.
(I disagree. Indeed, until recently governance people had very few policy asks for government.)
Did that change because people finally finished doing enough basic strategy research to know what policies to ask for?
Yeah, that’s Luke Muehlhauser’s claim; see the first paragraph of the linked piece.
I mostly agree with him. I wasn’t doing AI governance years ago but my impression is they didn’t have many/good policy asks. I’d be interested in counterevidence — like pre-2022 (collections of) good policy asks.
Anecdotally, I think I know one AI safety person who was doing influence-seeking-in-government and was on a good track but quit (to do research) because they weren’t able to leverage their influence because the AI governance community didn’t really have asks for (the US federal) government.
My own model differs a bit from Zach’s. It seems to me like most of the publicly-available policy proposals have not gotten much more concrete. It feels a lot more like people were motivated to share existing thoughts, as opposed to people having new thoughts or having more concrete thoughts.
Luke’s list, for example, is more of a “list of high-level ideas” than a “list of concrete policy proposals.” It has things like “licensing” and “information security requirements”– it’s not an actual bill or set of requirements. (And to be clear, I still like Luke’s post and it’s clear that he wasn’t trying to be super concrete).
I’d be excited for people to take policy ideas and concretize them further.
Aside: When I say “concrete” in this context, I don’t quite mean “people on LW would think this is specific.” I mean “this is closer to bill text, text of a section of an executive order, text of an amendment to a bill, text of an international treaty, etc.”
I think there are a lot of reasons why we haven’t seen much “concrete policy stuff”. Here are a few:
This work is just very difficult– it’s much easier to hide behind vagueness when you’re writing an academic-style paper than when you’re writing a concrete policy proposal.
This work requires people to express themselves with more certainty/concreteness than academic-style research. In a paper, you can avoid giving concrete recommendations, or you can give a recommendation and then immediately mention 3-5 crucial considerations that could change the calculus. In bills, you basically just say “here is what’s going to happen” and do much less “and here are the assumptions that go into this and a bunch of ways this could be wrong.”
This work forces people to engage with questions that are less “intellectually interesting” to many people (e.g., which government agency should be tasked with X, how exactly are we going to operationalize Y?)
This work just has a different “vibe” to the more LW-style research and the more academic-style research. Insofar as LW readers are selected for (and reinforced for) liking a certain “kind” of thinking/writing, this “kind” of thinking/writing is different than the concrete policy vibe in a bunch of hard-to-articulate ways.
This work often has the potential to be more consequential than academic-style research. There are clear downsides of developing [and advocating for] concrete policies that are bad. Without any gatekeeping, you might have a bunch of newbies writing flawed bills. With excessive gatekeeping, you might create a culture that disincentivizes intelligent people from writing good bills. (And my own subjective impression is that the community erred too far on the latter side, but I think reasonable people could disagree here).
For people interested in developing the kinds of proposals I’m talking about, I’d be happy to chat. I’m aware of a couple of groups doing the kind of policy thinking that I would consider “concrete”, and it’s quite plausible that we’ll see more groups shift toward this over time.
??? WTF do people “in AI governance” do?
Quick answer:
A lot of AI governance folks primarily do research. They rarely engage with policymakers directly, and they spend much of their time reading and writing papers.
This was even more true before the release of GPT-4 and the recent wave of interest in AI policy. Before GPT-4, many people believed “you will look weird/crazy if you talk to policymakers about AI extinction risk.” It’s unclear to me how true this was (in a genuine “I am confused about this & don’t think I have good models of this” way). Regardless, there has been an update toward talking to policymakers about AI risk now that AI risk is a bit more mainstream.
My own opinion is that, even after this update toward policymaker engagement, the community as a whole is still probably overinvested in research and underinvested in policymaker engagement/outreach. (Of course, the two can be complimentary, and the best outreach will often be done by people who have good models of what needs to be done & can present high-quality answers to the questions that policymakers have).
Among the people who do outreach/policymaker engagement, my impression is that there has been more focus on the executive branch (and less on Congress/congressional staffers). The main advantage is that the executive branch can get things done more quickly than Congress. The main disadvantage is that Congress is often required (or highly desired) to make “big things” happen (e.g., setting up a new agency or a licensing regime).
My prediction is that the AI safety community will overestimate the difficulty of policymaker engagement/outreach.
I think that the AI safety community has quickly and accurately taken social awkwardness and nerdiness into account, and factored that out of the equation. However, they will still overestimate the difficulty of policymaker outreach, on the basis that policymaker outreach requires substantially above-average sociability and personal charisma.
Even among the many non-nerd extroverts in the AI safety community, who have above average or well above average social skills (e.g. ~80th or 90th percentile), the ability to do well in policy requires an extreme combination of traits that produce intense charismatic competence, such the traits required for as a sense of humor near the level of a successful professional comedian (e.g. ~99th or 99.9th percentile). This is because the policy environment, like corporate executives, selects for charismatic extremity.
Because people who are introspective or think about science at all are very rarely far above the 90th percentile for charisma, even if only the obvious natural extroverts are taken into account, the AI safety community will overestimate the difficulty of policymaker outreach.
I don’t think they will underestimate the value of policymaker outreach (in fact I predict they are overestimating the value, due to the American interests in using AI for information warfare pushing AI decisionmaking towards inaccessible and inflexible parts of natsec agencies). But I do anticipate underestimating the feasibility of policymaker outreach.
I’m not sure I understand the direction of reasoning here. Overestimating the difficulty would mean that it will actually be easier than they think, which would be true if they expected a requirement of high charisma but the requirement were actually absent, or would be true if the people who ended up doing it were of higher charisma than the ones making the estimate. Or did you mean underestimating the difficulty?
I should have made it more clear at the beginning.
AI governance successfully filters out the nerdy people
They see that they’re still having a hard time finding their way to the policymakers with influence (e.g. what Akash was doing, meeting people in order to meet more people through them).
They conclude that the odds of success are something like ~30% or any other number.
I think that they would be off by something like 10, so it would actually be ~40%, because factoring out the nerds still leaves you with the people at the 90th percentile of Charisma and you need people at the 99th percentile. They might be able to procure those people.
This is because I predict that people at the 99th percentile of Charisma are underrepresented in AI safety, even if you only look at the non-nerds.
That makes sense and sounds sensible, at least pre-ChatGPT.
Modern congressional staffers are the product of Goodhart’s law; ~50-100 years ago, they were the ones that ran congress de-facto, so all the businessmen and voters wanted to talk to them, so the policymaking ended up moving elsewhere. Just like what happened with congressmen themselves ~100-150 years ago. Congressional staffers today primarily take constituent calls from voters, and make interest groups think they’re being listened to. Akash’s accomplishments came from wading through that bullshit, meeting people through people until he managed to find some gems.
Most policymaking today is called in from outside, with lobbyists having the domain-expertise needed to write the bills, and senior congressional staffers (like the legislative directors and legislative assistants here) overseeing the process, usually without getting very picky about the details.
It’s not like congressmembers have no power, but they’re just one part of what’s called an “Iron triangle”, the congressional lawmakers, the executive branch bureaucracies (e.g. FDA, CDC, DoD, NSA), and the private sector companies (e.g. Walmart, Lockheed, Microsoft, Comcast), with the lobbyists circulating around the three, negotiating and cutting deals between them. It’s incredibly corrupt and always has been, but not all-crushingly corrupt like African governments. It’s like the Military Industrial Complex, except that’s actually a bad example because congress is increasingly out of the loop de-facto on foreign policy (most structures are idiosyncratic, because the fundamental building block is people who are thinking of ways to negotiate backdoor deals).
People in the executive branch/bureaucracies like the DoD have more power on interesting things like foreign policy, Congress is more powerful for things that have been entrenched for decades like farming policy. Think tank people have no power but they’re much less stupid and have domain expertise and are often called up to help write bills instead of lobbyists.
I don’t know how AI policy is made in Congress, I jumped ship from domestic AI policy to foreign AI policy 3.5 years ago in order to focus more on the incentives from the US-China angle, Akash is the one to ask about where AI policymaking happens in congress, as he was the one actually there deep in the maze (maybe via DM because he didn’t describe it in this post).
I strongly recommend people talking to John Wentworth about AI policy, even if he doesn’t know much at first; after looking at Wentworth’s OpenAI dialog, he’s currently my top predicted candidate for “person who starts spending 2 hours a week thinking about AI policy instead of technical alignment, and thinks up galaxy brained solutions that break the stalemates that vexed the AI policy people for years”.
Most don’t do policy at all. Many do research. Since you’re incredulous, here are some examples of great AI governance research (which don’t synergize much with talking to policymakers):
Towards best practices in AGI safety and governance
Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring
Survey on intermediate goals in AI governance
I mean, those are all decent projects, but I would call zero of them “great”. Like, the whole appeal of governance as an approach to AI safety is that it’s (supposed to be) bottlenecked mainly on execution, not on research. None of the projects you list sound like they’re addressing an actual rate-limiting step to useful AI governance.
(I disagree. Indeed, until recently governance people had very few policy asks for government.)
(Also note that lots of “governance” research is ultimately aimed at helping labs improve their own safety. Central example: Structured access.)
Did that change because people finally finished doing enough basic strategy research to know what policies to ask for?
It didn’t seem like that to me. Instead, my impression was that it was largely triggered by ChatGPT and GPT4 making the topic more salient, and AI safety feeling more inside the Overton window. So there were suddenly a bunch of government people asking for concrete policy suggestions.
Yeah, that’s Luke Muehlhauser’s claim; see the first paragraph of the linked piece.
I mostly agree with him. I wasn’t doing AI governance years ago but my impression is they didn’t have many/good policy asks. I’d be interested in counterevidence — like pre-2022 (collections of) good policy asks.
Anecdotally, I think I know one AI safety person who was doing influence-seeking-in-government and was on a good track but quit (to do research) because they weren’t able to leverage their influence because the AI governance community didn’t really have asks for (the US federal) government.
My own model differs a bit from Zach’s. It seems to me like most of the publicly-available policy proposals have not gotten much more concrete. It feels a lot more like people were motivated to share existing thoughts, as opposed to people having new thoughts or having more concrete thoughts.
Luke’s list, for example, is more of a “list of high-level ideas” than a “list of concrete policy proposals.” It has things like “licensing” and “information security requirements”– it’s not an actual bill or set of requirements. (And to be clear, I still like Luke’s post and it’s clear that he wasn’t trying to be super concrete).
I’d be excited for people to take policy ideas and concretize them further.
Aside: When I say “concrete” in this context, I don’t quite mean “people on LW would think this is specific.” I mean “this is closer to bill text, text of a section of an executive order, text of an amendment to a bill, text of an international treaty, etc.”
I think there are a lot of reasons why we haven’t seen much “concrete policy stuff”. Here are a few:
This work is just very difficult– it’s much easier to hide behind vagueness when you’re writing an academic-style paper than when you’re writing a concrete policy proposal.
This work requires people to express themselves with more certainty/concreteness than academic-style research. In a paper, you can avoid giving concrete recommendations, or you can give a recommendation and then immediately mention 3-5 crucial considerations that could change the calculus. In bills, you basically just say “here is what’s going to happen” and do much less “and here are the assumptions that go into this and a bunch of ways this could be wrong.”
This work forces people to engage with questions that are less “intellectually interesting” to many people (e.g., which government agency should be tasked with X, how exactly are we going to operationalize Y?)
This work just has a different “vibe” to the more LW-style research and the more academic-style research. Insofar as LW readers are selected for (and reinforced for) liking a certain “kind” of thinking/writing, this “kind” of thinking/writing is different than the concrete policy vibe in a bunch of hard-to-articulate ways.
This work often has the potential to be more consequential than academic-style research. There are clear downsides of developing [and advocating for] concrete policies that are bad. Without any gatekeeping, you might have a bunch of newbies writing flawed bills. With excessive gatekeeping, you might create a culture that disincentivizes intelligent people from writing good bills. (And my own subjective impression is that the community erred too far on the latter side, but I think reasonable people could disagree here).
For people interested in developing the kinds of proposals I’m talking about, I’d be happy to chat. I’m aware of a couple of groups doing the kind of policy thinking that I would consider “concrete”, and it’s quite plausible that we’ll see more groups shift toward this over time.