Man, I agree with almost all the content of this post, but dispute the framing. This seems like maybe an oportunity to write up some related thoughts about transparency in the x-risk ecosystem.
A few months ago, I had opportunity to talk with a number of EA-aligned or x-risk concerned folks working in policy or policy adjacent roles as part of a grant evaluation process. My views here are informed by those conversations, but I am overall quite far from the action of AI policy stuff. I try to carefully flag my epistemic state regarding the claims below.
Omission
I think a lot of people, especially in AI governance, are…
Saying things that they think are true
while leaving out other important things that they think are true, but are also so extreme or weird-sounding that they would lose credibility.
A central example is promoting regulations on frontier AI systems because powerful AI systems could develop bio-weapons that could be misused to wipe out large swaths of humanity.
I think that most of the people promoting that policy agenda with that argumentation, do in fact think that AI-developed bioweapons are a real risk of the next 15 years. And, I guess, many to most of them think that there is also a risk of an AI takeover (including one that results in human extinction), within a similar timeframe. They’re in fact more concerned about the AI takeover risk, but they’re focusing on the bio-weapons misuse case, because that’s more defensible, and (they think) easier to get others to take seriously.[1] So they’re more likely to succed in getting their agenda passed into law, if they focus on those more-plausible sounding risks.
This is not, according to me, a lie. They are not making up the danger of AI-designed bio-weapons. And it is normal, in politics, to not say many things that you think and believe. If a person was asked point-blank about the risk AI takeover, and they gave an answer that implied the risk was lower than they think it is, in private, I would consider that a lie. But failing to volunteer that info when you’re not being asked for it is something different.[2]
However, I do find this dynamic of saying defensible things in the overton window, and leaving out your more extreme beliefs, concerning.
It is on the table that we will have Superintelligence radically transforming planet earth by 2028. And government actors who might be able to take action on that now, are talking to advisors who do think that that kind of radical transformation is possible in that near a time frame. But those advisors hold back from telling the government actors that they think that, because they expect to loose the credibility they have.
This sure looks sus to me. It sure seems like a sad world where almost all of the people that were in a position to give a serious warning to the people in power, opted not to, and so the people in power didn’t take the threat seriously until it was too late.
But is is important to keep in mind that the people I criticizing are much closer to the relevant action than I am. They may just be straightforwardly correct that they will be discredited if they talk about Superintelligence in the near term.
I would be pretty surprised by that, given that eg OpenAI is talking about Superintelligence in the near team. And it overall becomes a lot less weird to talk about if 50 people from FHI, OpenPhil, the labs, etc. are openly saying that they think the risk of human extinction is >10%, instead of just that weird, longstanding kooky-guy, Eliezer Yudkowsky.
And it seems like if you lose credibility for soothsaying, and then you’re soothsaying looks like it’s coming true, you will earn your credibility back later? I don’t know if that’s actually how it works in politics.
But I’m not an expert here. I’ve heard at least one second hand anecdote of an EA in DC “coming out” as seriously concerned about Superintelligence and AI takeover risk, and “loosing points” for doing so.
And overall, I have 1000x less experience engaging with government than these people, who have specialized in this kind of thing. I suspect that they’re pretty calibrated about how different classes of people will react.
I am personally not sure how to balance advocating for a policy that seems more sensible and higher integrity to me, on my inside view, with taking into account the expertise of the people in these positions. For the time being, I’m trying to be transparent that my inside view wishes that EA policy people should be much more transparent about what they think, while also not punishing those people for following a different standard.
Belief-suppression
However, it gets worse than that. It’s not only that many policy folks are not expressing their full beliefs, I think they’re further exerting pressure on others not to express their full beliefs.
When I talk to EA people working in policy, about new people entering the advocacy space, they almost universally express some level of concern, due to “poisoning the well” dynamics.
To lay out an example of poisoning the well:
Let’s say that some young EAs are excited about the opportunities to influence AI policy. They show up in DC and manage to schedule meetings with staffers. The talk about AI and AI risk, and maybe advocate for some specific policy like a licensing regime.
But they’re amateurs. They don’t really know what they’re doing, and they commit a bunch of faux pas, revealing that they don’t know important facts about the relevant collations in congress, or which kinds of things are at all politically feasible. The staffers mark these people as unserious fools, who don’t know what they’re talking about, and who wasted their time. They disregard whatever proposal was put forward as un-serious. (The staffer doesn’t let on about this though. Standard practice is to act polite, and then laugh about the meeting with your peers over drinks.)
Then, 6 months later, a different, more established advocacy group or think tank comes forward with a very similar policy. But they’re now fighting an uphill battle, since people in government have already formed associations with that policy, and with the general worldview
As near as I can tell, I think this poisoning the well effect is real.
People in government are overwhelmed with ideas, policies, and decisions. They don’t have time to read the full reports, and often make relatively quick judgments.
And furthermore, they’re used to reasoning according to a coalition logic. to get legislation passed is not just a matter of whether it is a good idea, but largely depends on social context of the legislation. Who an idea is associated with is a strong determinant of whether to take it seriously. [3]
But this dynamic causes some established EA DC policy people to be wary of new people entering the space unless they already have a lot of policy experience, such that they can avoid making those kinds of faux pas. They would prefer that anyone entering the space have high levels of native social tact, and additionally to be familiar with DC etiquette.
I don’t know this to be the case, but I wouldn’t be surprised if, people’s sense of “DC etiquette” includes not talking about or not focusing too much on extreme, Sci-fi sounding scenarios.” I would guess that there’s one person working in the policy space can mess things up for everyone else in that space, and so that creates a kind of conformity pressure whereby everyone expresses the same sorts of thing.
To be clear, I know that that isn’t happening universally. There’s at least one person that I talked to, working at org X, who suggested the opposite approach—they wanted a new advocacy org to explicitly not try to the sync their messaging with org X. They thought it made more sense for different groups, especially if they had different beliefs about what’s necessary for a good future, to advocate for different policies.
But I my guess is that there’s a lot of this kind of thing, where there’s a social pressure, amongst EA policy people, toward revealing less of one’s private beliefs, lest one be seen as something of a loose cannon.
Even insofar as my inside view is mistaken about how productive it would be to say, straightforwardly, there’s an additional question of how well-coordinated this kind of policy should be. My guess, is that by trying to all stay within the overton window, the EA policy ecosystem as a whole is preventing the overton window from shifting, and it would be better if there were less social pressure towards conformity, to enable more cascading social updates.
I’m sure that some of those folks would deny that they’re more concerned about AI takeover risks. Some of them would claim something like agnosticism about which risks are biggest.
That said, my guess is that many of the people that I’m thinking of, in these policy positions, if they were asked, point blank, might lie in exactly that way. I have no specific evidence of that, but it does seem like the most likely way many of them would respond, given their overall policy about communicating their beliefs.
I think that kind of lying is very bad, both misleading the person or people who are seeking info from you and a defection on our collective discourse commons by making it harder for everyone who agrees with you to say what is true.
And there are common knowledge effects here. Maybe some bumbling fools present a policy to you. You happen to have the ability to assess that their policy proposal is actually a really good idea. But you know that the bumbling fools also presented to a number of your colleagues, who are now snickering at how dumb and non-savvy they were.
If a person was asked point-blank about the risk AI takeover, and they gave an answer that implied the risk was lower than they think it is, in private, I would consider that a lie
[...]
That said, my guess is that many of the people that I’m thinking of, in these policy positions, if they were asked, point blank, might lie in exactly that way. I have no specific evidence of that, but it does seem like the most likely way many of them would respond, given their overall policy about communicating their beliefs.
As a relevant piece of evidence here, Jason Matheny, when asked point-blank in a senate committee hearing about “how concerned should we be about catastrophic risks from AI?” responded with “I don’t know”, which seems like it qualifies as a lie by the standard you set here (which, to be clear, I don’t super agree with and my intention here is partially to poke holes in your definition of a lie, while also sharing object-level relevant information).
Quote (slightly paraphrased because transcription is hard):
Senator Peters: “The last question before we close. We’ve heard thoughts from various experts about the risk of human-like artificial intelligence or Artificial General Intelligence, including various catastrophic projections. So my final question is, what is the risk that Artificial General Intelligence poses, and how likely is that to matter in the near future?”
[...]
Matheny: “As is typically my last words: I don’t know. I think it’s a really difficult question. I think whether AGI is nearer or farther than thought, I think there are things we can do today in either case. Including regulatory frameworks that include standards with third party tests and audits, governance of supply chains so we can understand where large amounts of computing is going, and so that we can prevent large amounts of computing going to places with lower ethical standards that we and other democracies have”
Given my best model of Matheny’s beliefs, this sure does not seem like an answer that accurately summarizes his beliefs here, and represents a kind of response that I think causes people to be quite miscalibrated about the beliefs of experts in the field.
In my experience people raise the hypothetical of “but they would be honest when asked point blank” to argue that people working in the space are not being deceptive. However, I have now seen people being asked point blank, and I haven’t seen them be more honest than their original evasiveness implied, so I think this should substantially increase people’s priors on people doing something more deceptive here.
Jason Matheny is approximately the most powerful person in the AI policy space. I think he is setting a precedent here for making statements that meet at least the definition of lying you set out in your comment (I am still unsure whether to count that as lying, though it sure doesn’t feel honest), and if-anything, if I talk to people in the field, Matheny is generally known as being among the more open and honest people in the space.
If his beliefs are what I would have expected them to be (eg something like “agrees with the basic arguments laid out in Superintelligence, and was motivated to follow his current carrer trajectory by those arguments”), then this answer is at best, misleading and misrepresentation of his actual models.
Seeing this particular example, I’m on the fence about whether to call it a “lie”. He was asked about the state of the world, not about his personal estimates, and he answered in a way that was more about the state of knowable public knowledge rather than his personal estimate. But I agree that seems pretty hair-splitting.
As it is, I notice that I’m confused.
Why wouldn’t he say something to the effect of the following?
I don’t know; this kind of forecasting is very difficult, timelines forecasting is very difficult. I can’t speak with confidence one way or the other. However, my best guess from following the literature on this topic for many years is that the catastrophic concerns are credible. I don’t know how probable it is, but does not seem to me that it is merely outlandish sci fi scenario that AI will lead to human extinction, and is not out of the question that that will happen in the next 10 years.
That doesn’t just seem more transparent, and more cooperative with the questioner, it also seems...like an obvious strategic move?
Does he not, in fact, by the basic arguments in Superingelligence? Is there some etiquette that he feels that he shouldn’t say that?
I think your interpretation is fairly uncharitable. If you have further examples of this deceptive pattern from those sympathetic to AI risk I would change my perspective but the speculation in the post plus this example weren’t compelling:
I watched the video and firstly Senator Peters seems to trail off after the quoted part and ends his question by saying “What’s your assessment of how fast this is going and when do you think we may be faced with those more challenging issues?”. So straightforwardly his question is about timelines not about risk as you frame it. Indeed Matheny (after two minutes) literally responds “it’s a really difficult question. I think whether AGI is nearer or farther than thought …” (emphasis different to yours) so makes it likely to me Matheny is expressing uncertainty about timelines, not risk.
Overall I agree that this was an opportunity for Matheny to discuss AI x-risk and plausibly it wasn’t the best use of time to discuss the uncertainty of the situation. But saying this is dishonesty doesn’t seem well supported
No, the question was about whether there are apocalyptic risks and on what timeline we should be concerned about apocalyptic risks.
The questioner used the term ‘apocalyptic’ specifically. Three people answered the question, and the first two both also alluded to ‘apocalyptic’ risks and sort of said that they didn’t really think we need to think about that possibility. Them referring to apocalyptic risks goes to show that it was a key part of what the questioner wanted to understand — to what extent these risks are real and on what timeline we’ll need to react to them. My read is not that Matheny actively misled the speaker, but that he avoided answering, which is “hiding” rather than “lying” (I don’t agree with the OP that they’re identical).
I think the question was unclear so it was more acceptable to not directly address whether there is apocalyptic risk, but I think many people I know would have definitely said “Oh to be clear I totally disagree with the previous two people, there are definitely apocalyptic risks and we are not prepared for them and cannot deal with them after-the-fact (as you just mentioned being concerned about).”
Extra detail on what happened
Everyone who answered explicitly avoided making timeline predictions and instead talked about where they think the policy focus should be.
The first person roughly said “We have many problems with AI right now, let’s focus on addressing those.”
The middle person said the AI problems are all of the sort “people being sent to jail because of an errant ML system”.
Here’s the middle person in full, clearly responding to the question of whether there’s apocalyptic risks to be worried about:
People ask me what keeps me up at night. AGI does not keep me up at night. And the reason why it doesn’t, is because (as Ms Gibbons mentioned) the problems we are likely to face, with the apocalyptic visions of AGI, are the same problems we are already facing right now, with the systems that are already in play. I worry about people being sent to jail because of an errant ML system. Whether you use some fancy AGI to do the same thing, it’s the same problem… My bet is that the harms we’re going to see, as these more powerful systems come online — even with ChatGPT — are no different from the harms we’re seeing right now. So if we focus our efforts and our energies on governance and regulation and guardrails to address the harms we’re seeing right now, they will be able to adjust as the technology improves. I am not worried that what we put in place today will be out of date or out of sync with the new tech. The new tech is like the old tech, just supercharged.
Matheny didn’t disagree with them and didn’t address the question of whether it’s apocalyptic, just said he was uncertain, and then listed the policies he wanted to see: setting standards with 3rd party audits, and governance of hardware supply chain to track it and control that it doesn’t go to places that aren’t democracies.
To not state that you disagree with the last two positions signals that you agree with them, as the absence of your disagreement is evidence of the absence of disagreement. I don’t think Matheny outright said anything false but I think it is a bit misleading to not say “I totally disagree, I think the new tech will be akin to inventing a whole new superintelligent alien species that may kill us all and take over the universe” if something like that is what you believe.
My read is that he was really trying as hard as he could to not address whether there are apocalyptic risks and instead just focus on encouraging the sorts of policies he thought should be implemented.
My read is that he was really trying as hard as he could to not address whether there are apocalyptic risks and instead just focus on encouraging the sorts of policies he thought should be implemented.
Why, though?
Does he know something we don’t? Does he think that if he expresses that those risks are real he’ll lose political capital? People won’t put him or his friends in positions of power, because he’ll be branded as a kook?
Is he just in the habit of side-stepping the weird possibilities?
This looks to me, from the outside, like an unforced error. They were asking the question, about some core beliefs, pretty directly. It seems like it would help if, in every such instance, the EA people who think that the world might be destroyed by AGI in the next 20 years, say that they think that the world might be destroyed by AGI in the next 20 years.
As Ben said, this seems incongruent with the responses that the other two people gave, neither of which talked that much about timelines, but did seem to directly respond to the concern about catastrophic/apocalyptic risk from AGI.
I do agree that it’s plausible that Matheny somehow understood the question differently from the other two people, and interpreted it in a more timelines focused way, though he also heard the other two people talk, which makes that somewhat less likely. I do agree that the question wasn’t asked in the most cogent way.
Thanks for checking this! I mostly agree with all your original comment now (except the first part suggesting it was point blank, but we’re quibbling over definitions at this point), this does seem like a case of intentionally not discussing risk
the recent TIME article saying there’s no trade off between progress and safety
More generally, for having talked to many AI policy/safety members, I can say it’s a very common pattern. At the eve of the FLI open letter, one of the most senior persons in the AI governance & policy X risk community was explaining that it was stupid to write this letter and that it would make future policy efforts much more difficult etc.
I agree that it is important to be clear about the potential for catastrophic AI risk, and I am somewhat disappointed in the answer above (though I think calling “I don’t know” lying is a bit of a stretch). But on the whole, I think people have been pretty upfront about catastrophic risk, e.g. Dario has given an explicit P(doom) publicly, all the lab heads have signed the CAIS letter, etc.
Notably, though, that’s not what the original post is primarily asking for: it’s asking for people to clearly state that they agree that we should pause/stop AI development, not to clearly state that that they think AI poses a catastrophic risk. I agree that people should clearly state that they think there’s a catastrophic risk, but I disagree that people should clearly state that they think we should pause.
Primarily, that’s because I don’t actually think trying to get governments to enact some sort of a generic pause would make good policy. Analogizing to climate change, I think getting scientists to say publicly that they think climate change is a real risk helped the cause, but putting pressure on scientists to publicly say that environmentalism/degrowth/etc. would solve the problem has substantially hurt the cause (despite the fact that a magic button that halved consumption would probably solve climate change).
Man, I agree with almost all the content of this post, but dispute the framing. This seems like maybe an oportunity to write up some related thoughts about transparency in the x-risk ecosystem.
A few months ago, I had opportunity to talk with a number of EA-aligned or x-risk concerned folks working in policy or policy adjacent roles as part of a grant evaluation process. My views here are informed by those conversations, but I am overall quite far from the action of AI policy stuff. I try to carefully flag my epistemic state regarding the claims below.
Omission
I think a lot of people, especially in AI governance, are…
Saying things that they think are true
while leaving out other important things that they think are true, but are also so extreme or weird-sounding that they would lose credibility.
A central example is promoting regulations on frontier AI systems because powerful AI systems could develop bio-weapons that could be misused to wipe out large swaths of humanity.
I think that most of the people promoting that policy agenda with that argumentation, do in fact think that AI-developed bioweapons are a real risk of the next 15 years. And, I guess, many to most of them think that there is also a risk of an AI takeover (including one that results in human extinction), within a similar timeframe. They’re in fact more concerned about the AI takeover risk, but they’re focusing on the bio-weapons misuse case, because that’s more defensible, and (they think) easier to get others to take seriously.[1] So they’re more likely to succed in getting their agenda passed into law, if they focus on those more-plausible sounding risks.
This is not, according to me, a lie. They are not making up the danger of AI-designed bio-weapons. And it is normal, in politics, to not say many things that you think and believe. If a person was asked point-blank about the risk AI takeover, and they gave an answer that implied the risk was lower than they think it is, in private, I would consider that a lie. But failing to volunteer that info when you’re not being asked for it is something different.[2]
However, I do find this dynamic of saying defensible things in the overton window, and leaving out your more extreme beliefs, concerning.
It is on the table that we will have Superintelligence radically transforming planet earth by 2028. And government actors who might be able to take action on that now, are talking to advisors who do think that that kind of radical transformation is possible in that near a time frame. But those advisors hold back from telling the government actors that they think that, because they expect to loose the credibility they have.
This sure looks sus to me. It sure seems like a sad world where almost all of the people that were in a position to give a serious warning to the people in power, opted not to, and so the people in power didn’t take the threat seriously until it was too late.
But is is important to keep in mind that the people I criticizing are much closer to the relevant action than I am. They may just be straightforwardly correct that they will be discredited if they talk about Superintelligence in the near term.
I would be pretty surprised by that, given that eg OpenAI is talking about Superintelligence in the near team. And it overall becomes a lot less weird to talk about if 50 people from FHI, OpenPhil, the labs, etc. are openly saying that they think the risk of human extinction is >10%, instead of just that weird, longstanding kooky-guy, Eliezer Yudkowsky.
And it seems like if you lose credibility for soothsaying, and then you’re soothsaying looks like it’s coming true, you will earn your credibility back later? I don’t know if that’s actually how it works in politics.
But I’m not an expert here. I’ve heard at least one second hand anecdote of an EA in DC “coming out” as seriously concerned about Superintelligence and AI takeover risk, and “loosing points” for doing so.
And overall, I have 1000x less experience engaging with government than these people, who have specialized in this kind of thing. I suspect that they’re pretty calibrated about how different classes of people will react.
I am personally not sure how to balance advocating for a policy that seems more sensible and higher integrity to me, on my inside view, with taking into account the expertise of the people in these positions. For the time being, I’m trying to be transparent that my inside view wishes that EA policy people should be much more transparent about what they think, while also not punishing those people for following a different standard.
Belief-suppression
However, it gets worse than that. It’s not only that many policy folks are not expressing their full beliefs, I think they’re further exerting pressure on others not to express their full beliefs.
When I talk to EA people working in policy, about new people entering the advocacy space, they almost universally express some level of concern, due to “poisoning the well” dynamics.
As near as I can tell, I think this poisoning the well effect is real.
People in government are overwhelmed with ideas, policies, and decisions. They don’t have time to read the full reports, and often make relatively quick judgments.
And furthermore, they’re used to reasoning according to a coalition logic. to get legislation passed is not just a matter of whether it is a good idea, but largely depends on social context of the legislation. Who an idea is associated with is a strong determinant of whether to take it seriously. [3]
But this dynamic causes some established EA DC policy people to be wary of new people entering the space unless they already have a lot of policy experience, such that they can avoid making those kinds of faux pas. They would prefer that anyone entering the space have high levels of native social tact, and additionally to be familiar with DC etiquette.
I don’t know this to be the case, but I wouldn’t be surprised if, people’s sense of “DC etiquette” includes not talking about or not focusing too much on extreme, Sci-fi sounding scenarios.” I would guess that there’s one person working in the policy space can mess things up for everyone else in that space, and so that creates a kind of conformity pressure whereby everyone expresses the same sorts of thing.
To be clear, I know that that isn’t happening universally. There’s at least one person that I talked to, working at org X, who suggested the opposite approach—they wanted a new advocacy org to explicitly not try to the sync their messaging with org X. They thought it made more sense for different groups, especially if they had different beliefs about what’s necessary for a good future, to advocate for different policies.
But I my guess is that there’s a lot of this kind of thing, where there’s a social pressure, amongst EA policy people, toward revealing less of one’s private beliefs, lest one be seen as something of a loose cannon.
Even insofar as my inside view is mistaken about how productive it would be to say, straightforwardly, there’s an additional question of how well-coordinated this kind of policy should be. My guess, is that by trying to all stay within the overton window, the EA policy ecosystem as a whole is preventing the overton window from shifting, and it would be better if there were less social pressure towards conformity, to enable more cascading social updates.
I’m sure that some of those folks would deny that they’re more concerned about AI takeover risks. Some of them would claim something like agnosticism about which risks are biggest.
That said, my guess is that many of the people that I’m thinking of, in these policy positions, if they were asked, point blank, might lie in exactly that way. I have no specific evidence of that, but it does seem like the most likely way many of them would respond, given their overall policy about communicating their beliefs.
I think that kind of lying is very bad, both misleading the person or people who are seeking info from you and a defection on our collective discourse commons by making it harder for everyone who agrees with you to say what is true.
And anyone who might be tempted to lie in a situation like that should take some time in advance to think through how they could respond in a way that is both an honest representation of their actual beliefs and also not disruptive to their profesional and political commitments.
And there are common knowledge effects here. Maybe some bumbling fools present a policy to you. You happen to have the ability to assess that their policy proposal is actually a really good idea. But you know that the bumbling fools also presented to a number of your colleagues, who are now snickering at how dumb and non-savvy they were.
As a relevant piece of evidence here, Jason Matheny, when asked point-blank in a senate committee hearing about “how concerned should we be about catastrophic risks from AI?” responded with “I don’t know”, which seems like it qualifies as a lie by the standard you set here (which, to be clear, I don’t super agree with and my intention here is partially to poke holes in your definition of a lie, while also sharing object-level relevant information).
See this video 1:39:00 to 1:43:00: https://www.hsgac.senate.gov/hearings/artificial-intelligence-risks-and-opportunities/
Quote (slightly paraphrased because transcription is hard):
Given my best model of Matheny’s beliefs, this sure does not seem like an answer that accurately summarizes his beliefs here, and represents a kind of response that I think causes people to be quite miscalibrated about the beliefs of experts in the field.
In my experience people raise the hypothetical of “but they would be honest when asked point blank” to argue that people working in the space are not being deceptive. However, I have now seen people being asked point blank, and I haven’t seen them be more honest than their original evasiveness implied, so I think this should substantially increase people’s priors on people doing something more deceptive here.
Jason Matheny is approximately the most powerful person in the AI policy space. I think he is setting a precedent here for making statements that meet at least the definition of lying you set out in your comment (I am still unsure whether to count that as lying, though it sure doesn’t feel honest), and if-anything, if I talk to people in the field, Matheny is generally known as being among the more open and honest people in the space.
If his beliefs are what I would have expected them to be (eg something like “agrees with the basic arguments laid out in Superintelligence, and was motivated to follow his current carrer trajectory by those arguments”), then this answer is at best, misleading and misrepresentation of his actual models.
Seeing this particular example, I’m on the fence about whether to call it a “lie”. He was asked about the state of the world, not about his personal estimates, and he answered in a way that was more about the state of knowable public knowledge rather than his personal estimate. But I agree that seems pretty hair-splitting.
As it is, I notice that I’m confused.
Why wouldn’t he say something to the effect of the following?
That doesn’t just seem more transparent, and more cooperative with the questioner, it also seems...like an obvious strategic move?
Does he not, in fact, by the basic arguments in Superingelligence? Is there some etiquette that he feels that he shouldn’t say that?
What’s missing from my understanding here?
I think your interpretation is fairly uncharitable. If you have further examples of this deceptive pattern from those sympathetic to AI risk I would change my perspective but the speculation in the post plus this example weren’t compelling:
I watched the video and firstly Senator Peters seems to trail off after the quoted part and ends his question by saying “What’s your assessment of how fast this is going and when do you think we may be faced with those more challenging issues?”. So straightforwardly his question is about timelines not about risk as you frame it. Indeed Matheny (after two minutes) literally responds “it’s a really difficult question. I think whether AGI is nearer or farther than thought …” (emphasis different to yours) so makes it likely to me Matheny is expressing uncertainty about timelines, not risk.
Overall I agree that this was an opportunity for Matheny to discuss AI x-risk and plausibly it wasn’t the best use of time to discuss the uncertainty of the situation. But saying this is dishonesty doesn’t seem well supported
No, the question was about whether there are apocalyptic risks and on what timeline we should be concerned about apocalyptic risks.
The questioner used the term ‘apocalyptic’ specifically. Three people answered the question, and the first two both also alluded to ‘apocalyptic’ risks and sort of said that they didn’t really think we need to think about that possibility. Them referring to apocalyptic risks goes to show that it was a key part of what the questioner wanted to understand — to what extent these risks are real and on what timeline we’ll need to react to them. My read is not that Matheny actively misled the speaker, but that he avoided answering, which is “hiding” rather than “lying” (I don’t agree with the OP that they’re identical).
I think the question was unclear so it was more acceptable to not directly address whether there is apocalyptic risk, but I think many people I know would have definitely said “Oh to be clear I totally disagree with the previous two people, there are definitely apocalyptic risks and we are not prepared for them and cannot deal with them after-the-fact (as you just mentioned being concerned about).”
Extra detail on what happened
Everyone who answered explicitly avoided making timeline predictions and instead talked about where they think the policy focus should be.
The first person roughly said “We have many problems with AI right now, let’s focus on addressing those.”
The middle person said the AI problems are all of the sort “people being sent to jail because of an errant ML system”.
Here’s the middle person in full, clearly responding to the question of whether there’s apocalyptic risks to be worried about:
Matheny didn’t disagree with them and didn’t address the question of whether it’s apocalyptic, just said he was uncertain, and then listed the policies he wanted to see: setting standards with 3rd party audits, and governance of hardware supply chain to track it and control that it doesn’t go to places that aren’t democracies.
To not state that you disagree with the last two positions signals that you agree with them, as the absence of your disagreement is evidence of the absence of disagreement. I don’t think Matheny outright said anything false but I think it is a bit misleading to not say “I totally disagree, I think the new tech will be akin to inventing a whole new superintelligent alien species that may kill us all and take over the universe” if something like that is what you believe.
My read is that he was really trying as hard as he could to not address whether there are apocalyptic risks and instead just focus on encouraging the sorts of policies he thought should be implemented.
Why, though?
Does he know something we don’t? Does he think that if he expresses that those risks are real he’ll lose political capital? People won’t put him or his friends in positions of power, because he’ll be branded as a kook?
Is he just in the habit of side-stepping the weird possibilities?
This looks to me, from the outside, like an unforced error. They were asking the question, about some core beliefs, pretty directly. It seems like it would help if, in every such instance, the EA people who think that the world might be destroyed by AGI in the next 20 years, say that they think that the world might be destroyed by AGI in the next 20 years.
As Ben said, this seems incongruent with the responses that the other two people gave, neither of which talked that much about timelines, but did seem to directly respond to the concern about catastrophic/apocalyptic risk from AGI.
I do agree that it’s plausible that Matheny somehow understood the question differently from the other two people, and interpreted it in a more timelines focused way, though he also heard the other two people talk, which makes that somewhat less likely. I do agree that the question wasn’t asked in the most cogent way.
Thanks for checking this! I mostly agree with all your original comment now (except the first part suggesting it was point blank, but we’re quibbling over definitions at this point), this does seem like a case of intentionally not discussing risk
A few other examples off the top of my head:
ARC graph on RSPs with the “safe zone” part
Anthropic calling ASL-4 accidental risks “speculative”
the recent TIME article saying there’s no trade off between progress and safety
More generally, for having talked to many AI policy/safety members, I can say it’s a very common pattern. At the eve of the FLI open letter, one of the most senior persons in the AI governance & policy X risk community was explaining that it was stupid to write this letter and that it would make future policy efforts much more difficult etc.
I agree that it is important to be clear about the potential for catastrophic AI risk, and I am somewhat disappointed in the answer above (though I think calling “I don’t know” lying is a bit of a stretch). But on the whole, I think people have been pretty upfront about catastrophic risk, e.g. Dario has given an explicit P(doom) publicly, all the lab heads have signed the CAIS letter, etc.
Notably, though, that’s not what the original post is primarily asking for: it’s asking for people to clearly state that they agree that we should pause/stop AI development, not to clearly state that that they think AI poses a catastrophic risk. I agree that people should clearly state that they think there’s a catastrophic risk, but I disagree that people should clearly state that they think we should pause.
Primarily, that’s because I don’t actually think trying to get governments to enact some sort of a generic pause would make good policy. Analogizing to climate change, I think getting scientists to say publicly that they think climate change is a real risk helped the cause, but putting pressure on scientists to publicly say that environmentalism/degrowth/etc. would solve the problem has substantially hurt the cause (despite the fact that a magic button that halved consumption would probably solve climate change).