I have seen/heard from at least two sources something to the effect that MIRI/CFAR leadership (and Anna in particular) has very short AI timelines and high probability of doom (and apparently having high confidence in these beliefs). Here is the only public example that I can recall seeing. (Of the two examples I can specifically recall, this is not the better one, but the other was not posted publicly.) Is there any truth to these claims?
Riceissa’s question was brief, so I’ll add a bunch of my thoughts on this topic.
I also remember there was something of a hush around the broader x-risk network on the topic of timelines, sometime around the time of FLI’s second AI conference. Since then I’ve received weird mixed signals about what people think, with hushed tones of being very worried/scared. The explicit content is of a similar type to Sam Altman’s line “if you believe what I believe about the timeline to AGI and the effect it will have on the world, it is hard to spend a lot of mental cycles thinking about anything else” but rarely accompanied with an explanation of the reasoning that lead to that view.
I think that you can internalise models of science, progress, computation, ML, and geopolitics, and start to feel like “AGI being built” is part of your reality, your world-model, and then figure out what actions you want to take in the world. I’ve personally thought about it a bit and come to some of my own conclusions, and I’ve generally focused on plans designed for making sure AGI goes well. This is the important and difficult work of incorporating abstract, far ideas into your models of near-mode reality.
But it’s also seems to me that a number of x-risk people looked at many of the leaders getting scared, and that is why they believe the timeline is short. This is how a herd turns around and runs in fear from an oncoming jaguar—most members of the herd don’t stop to check for themselves, they trust that everyone else is running for good reason. More formally, it’s known as an info cascade. This is often the rational thing to do when people you trust act as if something dangerous is coming at you. You don’t stop and actually pay attention to the evidence oneself.
(I personally experience such herd behaviour commonly when using the train systems in the UK. When a train is cancelled and 50 people are waiting beside it to get on, I normally don’t see the board that announces which platform to go to for the replacement train, as it’s only visible to a few of the people, but very quickly the whole 50 people are moving to the new platform for the replacement train. I also see it when getting off a train at a new train station, where lots of people don’t really know which way to walk to get out of the building: immediately coming off the train, is it left or right? But the first few people tend to make a judgement, and basically everyone else follows them. I’ve sometimes done it myself, been the first off and started walking confidently in a direction, and have everyone start confidently follow me, and it always feels a little magical for a moment, because I know I just took a guess.)
But the unusual thing about our situation, is that when you ask the leaders of the pack why they think a jaguar is coming, they’re very secretive about it. In my experience many clued-in people will explicitly recommend not sharing information about timelines. I’m thinking about OpenPhil, OpenAI, MIRI, FHI, and so on. I don’t think I’ve ever talked to people at CFAR about timelines.
To add more detail to my saying it’s considered ‘the’ decision-relevant variable by many, here’s two quotes. Ray Arnold is a colleague and a friend of mine, and two years ago he wrote a good post on his general updates about such subjects, that said the following:
Claim 1: Whatever your estimates two years ago for AGI timelines, they should probably be shorter and more explicit this year.
Claim 2: Relatedly, if you’ve been waiting for concrete things to happen for you to get worried enough to take AGI x-risk seriously, that time has come. Whatever your timelines currently are, they should probably be influencing your decisions in ways more specific than periodically saying “Well, this sounds concerning.”
[Timelines] are the decision-relevant question. At some point timelines get short enough that it’s pointless to save for retirement. At some point timelines get short enough that it may be morally irresponsible to have children...
Ray talks in his post about how much of his beliefs on this topic comes from trusting another person closer to the action, which is perfectly reasonable thing to do, though I’ll point out again it’s also (if lots of people do it) herd behaviour. Qiaochu talks about how he never figured out the timeline to AGI with an explicit model, even though he takes short timelines very seriously, which also sounds like a process that involves trusting others a bunch.
It’s okay to keep secrets, and in a number cases it’s of crucial importance. Much of Nick Bostrom’s career is about how some information can be hazardous, and about how not all ideas are safe at our current level of wisdom. But it’s important to note that “short timelines” is a particular idea that has had the herd turn around and running in fear to solve an urgent problem, and there’s been a lot of explicit recommendations to not give people the info they’d need to make a good judgment about it. And those two things together are always worrying.
It’s also very unusual for this community. We’ve been trying to make things go well wrt AGI for over a decade, and until recently we’ve put all our reasoning out in the open. Eliezer and Bostrom published so much. And yet now this central decision-node, “the decision-relevant variable”, is hidden from the view of most people involved. It’s quite strange, and generally is the sort of thing that is at risk for abuse by whatever process is deciding what the ‘ground truth’ is. I don’t believe the group of people involved in being secretive about AI timelines have spent at all as much time thinking about the downsides of secrecy or put in the work to mitigate them. Of course I can’t really tell, given the secrecy.
All that said, as you can see in the quotes/links that I and Robby provided elsewhere in this thread, I think Eliezer has made the greatest attempt of basically anyone to explain how he models timelines, and wrote very explicitly about his updates after AlphaGo Zero. And the Fire Alarm post was really, really great. In my personal experience the things in the quotes above is fairly consistent with how Eliezer reasoned about timelines before the deep learning revolution.
I think a factor that is likely to be highly relevant is that companies like DeepMind face a natural incentive to obscure understanding their progress and to be the sole arbiters of what is going to happen. I know that they’re very careful about requiring all visitors to their offices to sign NDAs, and requiring employees to get permission for any blogposts they’re planning to write on the internet about AI. I’d guess a substantial amount of this effect comes from there, but I’m not sure.
Edit: I edited this comment a bunch of times because I initially wrote it quickly, and didn’t quite like how it came out. Sorry if anyone was writing a reply. I’m not likely to edit it again.
Edit: I think it’s likely I’ll turn this into a top level post at some point.
For the record, parts of that ratanon post seem extremely inaccurate to me; for example, the claim that MIRI people are deferring to Dario Amodei on timelines is not even remotely reasonable. So I wouldn’t take it that seriously.
Agreed I wouldn’t take the ratanon post too seriously. For another example, I know from living with Dario that his motives do not resemble those ascribed to him in that post.
Huh, thanks for the info, I’m surprised to hear that.
I myself had heard that rumour, saying that at the second FLI conference Dario had spoken a lot about short timelines and now everyone including MIRI was scared. IIRC I heard it from some people involved in ML who were in attendance of that conference, but I didn’t hear it from anyone at MIRI. I never heard much disconfirmatory evidence, and it’s certainly been a sort-of-belief that’s bounced around my head for the past two or so years.
Certainly MIRI has written about this, for example see the relevant part of their 2018 update:
The latter scenario is relatively less important in worlds where AGI timelines are short. If current deep learning research is already on the brink of AGI, for example, then it becomes less plausible that the results of MIRI’s deconfusion work could become a relevant influence on AI capabilities research, and most of the potential impact of our work would come from its direct applicability to deep-learning-based systems. While many of us at MIRI believe that short timelines are at least plausible, there is significant uncertainty and disagreement about timelines inside MIRI, and I would not feel comfortable committing to a course of action that is safe only in worlds where timelines are short.
Also see Eliezer’s top-notch piece on timelines, which includes the relevant quote:
Of course, the future is very hard to predict in detail. It’s so hard that not only do I confess my own inability, I make the far stronger positive statement that nobody else can do it either.
Eliezer also updated after losing a bet that AlphaGo would not be able to beat humans so well, which he wrote about in AlphaGo Zero and the Foom Debate. It ends with the line:
I wouldn’t have predicted AlphaGo and lost money betting against the speed of its capability gains, because reality held a more extreme position than I did on the Yudkowsky-Hanson spectrum.
More timeline statements, from Eliezer in March 2016:
That said, timelines are the hardest part of AGI issues to forecast, by which I mean that if you ask me for a specific year, I throw up my hands and say “Not only do I not know, I make the much stronger statement that nobody else has good knowledge either.” Fermi said that positive-net-energy from nuclear power wouldn’t be possible for 50 years, two years before he oversaw the construction of the first pile of uranium bricks to go critical. The way these things work is that they look fifty years off to the slightly skeptical, and ten years later, they still look fifty years off, and then suddenly there’s a breakthrough and they look five years off, at which point they’re actually 2 to 20 years off.
If you hold a gun to my head and say “Infer your probability distribution from your own actions, you self-proclaimed Bayesian” then I think I seem to be planning for a time horizon between 8 and 40 years, but some of that because there’s very little I think I can do in less than 8 years, and, you know, if it takes longer than 40 years there’ll probably be some replanning to do anyway over that time period.
Since [August], senior staff at MIRI have reassessed their views on how far off artificial general intelligence (AGI) is and concluded that shorter timelines are more likely than they were previously thinking. [...]
There’s no consensus among MIRI researchers on how long timelines are, and our aggregated estimate puts medium-to-high probability on scenarios in which the research community hasn’t developed AGI by, e.g., 2035. On average, however, research staff now assign moderately higher probability to AGI’s being developed before 2035 than we did a year or two ago.
I talked to Nate last month and he outlined the same concepts and arguments from Eliezer’s Oct. 2017 There’s No Fire Alarm for AGI (mentioned by Ben above) to describe his current view of timelines, in particular (quoting Eliezer’s post):
History shows that for the general public, and even for scientists not in a key inner circle, and even for scientists in that key circle, it is very often the case that key technological developments still seem decades away, five years before they show up. [...]
And again, that’s not to say that people saying “fifty years” is a certain sign that something is happening in a squash court; they were saying “fifty years” sixty years ago too. It’s saying that anyone who thinks technological timelines are actually forecastable, in advance, by people who are not looped in to the leading project’s progress reports and who don’t share all the best ideas about exactly how to do the thing and how much effort is required for that, is learning the wrong lesson from history. In particular, from reading history books that neatly lay out lines of progress and their visible signs that we all know now were important and evidential. It’s sometimes possible to say useful conditional things about the consequences of the big development whenever it happens, but it’s rarely possible to make confident predictions about the timing of those developments, beyond a one- or two-year horizon. And if you are one of the rare people who can call the timing, if people like that even exist, nobody else knows to pay attention to you and not to the Excited Futurists or Sober Skeptics. [...]
So far as I can presently estimate, now that we’ve had AlphaGo and a couple of other maybe/maybe-not shots across the bow, and seen a huge explosion of effort invested into machine learning and an enormous flood of papers, we are probably going to occupy our present epistemic state until very near the end.
By saying we’re probably going to be in roughly this epistemic state until almost the end, I don’t mean to say we know that AGI is imminent, or that there won’t be important new breakthroughs in AI in the intervening time. I mean that it’s hard to guess how many further insights are needed for AGI, or how long it will take to reach those insights. After the next breakthrough, we still won’t know how many more breakthroughs are needed, leaving us in pretty much the same epistemic state as before. Whatever discoveries and milestones come next, it will probably continue to be hard to guess how many further insights are needed, and timelines will continue to be similarly murky. Maybe researcher enthusiasm and funding will rise further, and we’ll be able to say that timelines are shortening; or maybe we’ll hit another AI winter, and we’ll know that’s a sign indicating that things will take longer than they would otherwise; but we still won’t know how long.
I had already seen all of those quotes/links, all of the quotes/links that Rob Bensinger posts in the sibling comment, as well as this tweet from Eliezer. I asked my question because those public quotes don’t sound like the private information I referred to in my question, and I wanted insight into the discrepancy.
Okay. I was responding to “Is there any truth to these claims?” which sounded like it would be a big shock to discover MIRI/CFAR staff were considering short timelines a lot in their actions, when they’d actually stated it out loud in many places.
While I agree that I’m confused about MIRI/CFAR’s timelines and think that info-cascades around this have likely occurred, I want to mention that the thing you linked to is pretty hyperbolic.
To the best of my understanding, part of why the MIRI leadership (Nate Soares, Eliezer Yudkowsky, and Anna Salamon) have been delusionally spewing nonsense about the destruction of the world within a decade is because they’ve been misled by Dario Amodei, an untrustworthy, blatant status-seeker recently employed at Google Brain. I am unaware of the existence of even a single concrete, object-level reason to believe these claims; I, and many others, suspect that Dario is intentionally embellishing the facts because he revels in attention.
I want to say that I think that Dario is not obviously untrustworthy; I think well of him for being an early EA who put in the work to write up their reasoning about donations (see his extensive writeup on the GiveWell blog from 2009) which I always take as a good sign about someone’s soul; the quote also says there’s no reason or argument to believe in short timelines, but the analyses above in Eliezer’s posts on AlphaGo Zero and Fire Alarm provide plenty of reasons for thinking AI could come within a decade. Don’t forget that Shane Legg, one of the cofounders of DeepMind, has been consistently predicting AGI with 50% probability by 2028 (e.g. he said it here in 2011).
Don’t forget that Shane Legg, one of the cofounders of DeepMind, has been consistently predicting AGI with 50% probability by 2028 (e.g. he said it here in 2011).
Just noting that since then, half the time to 2028 has elapsed. If he’s still giving 50%, that’s kind of surprising.
Why is that surprising? Doesn’t it just mean that the pace of development in the last decade has been approximately equal to the average over Shane_{2011}’s distribution of development speeds?
I don’t think it’s that simple. The uncertainty isn’t just about pace of development but about how much development needs to be done.
But even if it does mean that, would that not be surprising? Perhaps not if he’d originally given a narrow confidence internal, but his 10% estimate was in 2018. For us to be hitting the average precisely enough to not move the 50% estimate much… I haven’t done any arithmetic here, but I think that would be surprising, yeah.
And my sense is that the additional complexity makes it more surprising, not less.
Yes, I agree that the space of things to be uncertain about is multidimensional. We project the uncertainty onto a one-dimensional space parameterized by “probability of <event> by <time>”.
It would be surprising for a sophisticated person to show a market of 49 @ 51 on this event. (Unpacking jargon, showing this market means being willing to buy for 49 or sell at 51 a contract which is worth 100 if the hypothesis is true and 0 if it is false)
(it’s somewhat similar saying that your 2-sigma confidence interval around the “true probability” of the event is 49 to 51. The market language can be interpreted with just decision theory while the confidence interval idea also requires some notion of statistics)
My interpretation of the second-hand evidence about Shane Legg’s opinion suggests that Shane would quote a market like 40 @ 60. (The only thing I know about Shane is that they apparently summarized their belief as 50% a number of years ago and hasn’t publicly changed their opinion since)
Perhaps I’m misinterpreting you, but I feel like this was intended as disagreement? If so, I’d appreciate clarification. It seems basically correct to me, and consistent with what I said previously. I still think that: if, in 2011, you gave 10% probability by 2018 and 50% by 2028; and if, in 2019, you still give 50% by 2028 (as an explicit estimate, i.e. you haven’t just not-given an updated estimate); then this is surprising, even acknowledging that 50% is probably not very precise in either case.
I realised after writing that I didn’t give a quote to show he that still believed it. I have the recollection that he still says 2028, I think someone more connected to AI/ML probably told me, but I can’t think of anywhere to quote him saying it.
I have seen/heard from at least two sources something to the effect that MIRI/CFAR leadership (and Anna in particular) has very short AI timelines and high probability of doom (and apparently having high confidence in these beliefs). Here is the only public example that I can recall seeing. (Of the two examples I can specifically recall, this is not the better one, but the other was not posted publicly.) Is there any truth to these claims?
Riceissa’s question was brief, so I’ll add a bunch of my thoughts on this topic.
I also remember there was something of a hush around the broader x-risk network on the topic of timelines, sometime around the time of FLI’s second AI conference. Since then I’ve received weird mixed signals about what people think, with hushed tones of being very worried/scared. The explicit content is of a similar type to Sam Altman’s line “if you believe what I believe about the timeline to AGI and the effect it will have on the world, it is hard to spend a lot of mental cycles thinking about anything else” but rarely accompanied with an explanation of the reasoning that lead to that view.
I think that you can internalise models of science, progress, computation, ML, and geopolitics, and start to feel like “AGI being built” is part of your reality, your world-model, and then figure out what actions you want to take in the world. I’ve personally thought about it a bit and come to some of my own conclusions, and I’ve generally focused on plans designed for making sure AGI goes well. This is the important and difficult work of incorporating abstract, far ideas into your models of near-mode reality.
But it’s also seems to me that a number of x-risk people looked at many of the leaders getting scared, and that is why they believe the timeline is short. This is how a herd turns around and runs in fear from an oncoming jaguar—most members of the herd don’t stop to check for themselves, they trust that everyone else is running for good reason. More formally, it’s known as an info cascade. This is often the rational thing to do when people you trust act as if something dangerous is coming at you. You don’t stop and actually pay attention to the evidence oneself.
(I personally experience such herd behaviour commonly when using the train systems in the UK. When a train is cancelled and 50 people are waiting beside it to get on, I normally don’t see the board that announces which platform to go to for the replacement train, as it’s only visible to a few of the people, but very quickly the whole 50 people are moving to the new platform for the replacement train. I also see it when getting off a train at a new train station, where lots of people don’t really know which way to walk to get out of the building: immediately coming off the train, is it left or right? But the first few people tend to make a judgement, and basically everyone else follows them. I’ve sometimes done it myself, been the first off and started walking confidently in a direction, and have everyone start confidently follow me, and it always feels a little magical for a moment, because I know I just took a guess.)
But the unusual thing about our situation, is that when you ask the leaders of the pack why they think a jaguar is coming, they’re very secretive about it. In my experience many clued-in people will explicitly recommend not sharing information about timelines. I’m thinking about OpenPhil, OpenAI, MIRI, FHI, and so on. I don’t think I’ve ever talked to people at CFAR about timelines.
To add more detail to my saying it’s considered ‘the’ decision-relevant variable by many, here’s two quotes. Ray Arnold is a colleague and a friend of mine, and two years ago he wrote a good post on his general updates about such subjects, that said the following:
Qiaochu also talked about it as the decision-relevant question:
Ray talks in his post about how much of his beliefs on this topic comes from trusting another person closer to the action, which is perfectly reasonable thing to do, though I’ll point out again it’s also (if lots of people do it) herd behaviour. Qiaochu talks about how he never figured out the timeline to AGI with an explicit model, even though he takes short timelines very seriously, which also sounds like a process that involves trusting others a bunch.
It’s okay to keep secrets, and in a number cases it’s of crucial importance. Much of Nick Bostrom’s career is about how some information can be hazardous, and about how not all ideas are safe at our current level of wisdom. But it’s important to note that “short timelines” is a particular idea that has had the herd turn around and running in fear to solve an urgent problem, and there’s been a lot of explicit recommendations to not give people the info they’d need to make a good judgment about it. And those two things together are always worrying.
It’s also very unusual for this community. We’ve been trying to make things go well wrt AGI for over a decade, and until recently we’ve put all our reasoning out in the open. Eliezer and Bostrom published so much. And yet now this central decision-node, “the decision-relevant variable”, is hidden from the view of most people involved. It’s quite strange, and generally is the sort of thing that is at risk for abuse by whatever process is deciding what the ‘ground truth’ is. I don’t believe the group of people involved in being secretive about AI timelines have spent at all as much time thinking about the downsides of secrecy or put in the work to mitigate them. Of course I can’t really tell, given the secrecy.
All that said, as you can see in the quotes/links that I and Robby provided elsewhere in this thread, I think Eliezer has made the greatest attempt of basically anyone to explain how he models timelines, and wrote very explicitly about his updates after AlphaGo Zero. And the Fire Alarm post was really, really great. In my personal experience the things in the quotes above is fairly consistent with how Eliezer reasoned about timelines before the deep learning revolution.
I think a factor that is likely to be highly relevant is that companies like DeepMind face a natural incentive to obscure understanding their progress and to be the sole arbiters of what is going to happen. I know that they’re very careful about requiring all visitors to their offices to sign NDAs, and requiring employees to get permission for any blogposts they’re planning to write on the internet about AI. I’d guess a substantial amount of this effect comes from there, but I’m not sure.
Edit: I edited this comment a bunch of times because I initially wrote it quickly, and didn’t quite like how it came out. Sorry if anyone was writing a reply. I’m not likely to edit it again.
Edit: I think it’s likely I’ll turn this into a top level post at some point.
FWIW, I don’t feel this way about timelines anymore. Lot more pessimistic about estimates being mostly just noise.
For the record, parts of that ratanon post seem extremely inaccurate to me; for example, the claim that MIRI people are deferring to Dario Amodei on timelines is not even remotely reasonable. So I wouldn’t take it that seriously.
Agreed I wouldn’t take the ratanon post too seriously. For another example, I know from living with Dario that his motives do not resemble those ascribed to him in that post.
I don’t know Dario well, but I know enough to be able to tell that the anon here doesn’t know what they’re talking about re Dario.
Huh, thanks for the info, I’m surprised to hear that.
I myself had heard that rumour, saying that at the second FLI conference Dario had spoken a lot about short timelines and now everyone including MIRI was scared. IIRC I heard it from some people involved in ML who were in attendance of that conference, but I didn’t hear it from anyone at MIRI. I never heard much disconfirmatory evidence, and it’s certainly been a sort-of-belief that’s bounced around my head for the past two or so years.
Certainly MIRI has written about this, for example see the relevant part of their 2018 update:
Also see Eliezer’s top-notch piece on timelines, which includes the relevant quote:
Eliezer also updated after losing a bet that AlphaGo would not be able to beat humans so well, which he wrote about in AlphaGo Zero and the Foom Debate. It ends with the line:
More timeline statements, from Eliezer in March 2016:
And from me in April 2017:
I talked to Nate last month and he outlined the same concepts and arguments from Eliezer’s Oct. 2017 There’s No Fire Alarm for AGI (mentioned by Ben above) to describe his current view of timelines, in particular (quoting Eliezer’s post):
I had already seen all of those quotes/links, all of the quotes/links that Rob Bensinger posts in the sibling comment, as well as this tweet from Eliezer. I asked my question because those public quotes don’t sound like the private information I referred to in my question, and I wanted insight into the discrepancy.
Okay. I was responding to “Is there any truth to these claims?” which sounded like it would be a big shock to discover MIRI/CFAR staff were considering short timelines a lot in their actions, when they’d actually stated it out loud in many places.
While I agree that I’m confused about MIRI/CFAR’s timelines and think that info-cascades around this have likely occurred, I want to mention that the thing you linked to is pretty hyperbolic.
I want to say that I think that Dario is not obviously untrustworthy; I think well of him for being an early EA who put in the work to write up their reasoning about donations (see his extensive writeup on the GiveWell blog from 2009) which I always take as a good sign about someone’s soul; the quote also says there’s no reason or argument to believe in short timelines, but the analyses above in Eliezer’s posts on AlphaGo Zero and Fire Alarm provide plenty of reasons for thinking AI could come within a decade. Don’t forget that Shane Legg, one of the cofounders of DeepMind, has been consistently predicting AGI with 50% probability by 2028 (e.g. he said it here in 2011).
Just noting that since then, half the time to 2028 has elapsed. If he’s still giving 50%, that’s kind of surprising.
Why is that surprising? Doesn’t it just mean that the pace of development in the last decade has been approximately equal to the average over Shane_{2011}’s distribution of development speeds?
I don’t think it’s that simple. The uncertainty isn’t just about pace of development but about how much development needs to be done.
But even if it does mean that, would that not be surprising? Perhaps not if he’d originally given a narrow confidence internal, but his 10% estimate was in 2018. For us to be hitting the average precisely enough to not move the 50% estimate much… I haven’t done any arithmetic here, but I think that would be surprising, yeah.
And my sense is that the additional complexity makes it more surprising, not less.
Yes, I agree that the space of things to be uncertain about is multidimensional. We project the uncertainty onto a one-dimensional space parameterized by “probability of <event> by <time>”.
It would be surprising for a sophisticated person to show a market of 49 @ 51 on this event. (Unpacking jargon, showing this market means being willing to buy for 49 or sell at 51 a contract which is worth 100 if the hypothesis is true and 0 if it is false)
(it’s somewhat similar saying that your 2-sigma confidence interval around the “true probability” of the event is 49 to 51. The market language can be interpreted with just decision theory while the confidence interval idea also requires some notion of statistics)
My interpretation of the second-hand evidence about Shane Legg’s opinion suggests that Shane would quote a market like 40 @ 60. (The only thing I know about Shane is that they apparently summarized their belief as 50% a number of years ago and hasn’t publicly changed their opinion since)
Perhaps I’m misinterpreting you, but I feel like this was intended as disagreement? If so, I’d appreciate clarification. It seems basically correct to me, and consistent with what I said previously. I still think that: if, in 2011, you gave 10% probability by 2018 and 50% by 2028; and if, in 2019, you still give 50% by 2028 (as an explicit estimate, i.e. you haven’t just not-given an updated estimate); then this is surprising, even acknowledging that 50% is probably not very precise in either case.
I realised after writing that I didn’t give a quote to show he that still believed it. I have the recollection that he still says 2028, I think someone more connected to AI/ML probably told me, but I can’t think of anywhere to quote him saying it.