I think this question is too vague. MIRI could turn out to be useless for any number of reasons, leading to different empirical disconfirmations. (A lot of these will look like the end of human life.) E.g.:
MIRI is useless because FAI research is very useful, but MIRI’s basic methodology or research orientation is completely and irredeemably the wrong approach to FAI. Expected evidence: MIRI’s research starts seeing diminishing returns, even as they attempt a wide variety of strategies; non-MIRI researchers make surprising amounts of progress into the issues MIRI is interested in; reputable third parties that assess MIRI’s results consistently disagree with its foundational assumptions or methodology; the researchers MIRI attracts come to be increasingly seen by the academic establishment as irrelevant to FAI research or even as cranks, for substantive, mathy reasons.
MIRI is useless because FAI is impossible—we simply lack the resources to engineer a benign singularity, no matter how hard we try. Expected evidence: Demonstrations that a self-modifying AGI can’t have stable, predictable values; demonstrations that coding anything like indirect normativity is unfeasible; increasingly slow progress, e.g., due to fixed hardware limitations.
MIRI is useless because some defeater will inevitably kill us all (or otherwise halt technological progress) before we have time to produce self-enhancing AGI. (Alternatively: UFAI is completely impossible to prevent, by any means.) Expected evidence: Discovery of some weapon (e.g., a supervirus or bomb) that can reliably kill everyone and has widely known and easy-to-follow engineering specifications.
MIRI is useless because some defeater will kill us all before we build an AGI, with a probability high enough to suck all the expected utility out of FAI research. I.e., our survival until the intelligence explosion singularity is a possibility, but not a large enough possibility to make the value of surviving past that sieve worth paying attention to. Expected evidence: Discovery that near-term existential threats are vastly more threatening than currently thought.
MIRI is useless because FAI is inevitable. Expected evidence: Black-swan disproof of fragility-of-value and complexity-of-value, revolutionizing human psychology.
Note that these scenarios not only would be indicated by different evidence, but also call for very different responses.
If 1 is true: Instead of funding MIRI, we should fund some radically different FAI research program.
If 2 is true: Instead of funding MIRI, we should pour all our resources into permanently stopping AI research, by any available means.
If 3 is true: Instead of funding MIRI, we should just enjoy what little time we have left before the apocalypse.
If 4 is true: Instead of funding MIRI, we should combat other existential risks, or act as in 3.
If 5 is true: Instead of funding MIRI, we should speed along AI research and fund cryonic and life-extension technologies in the hopes of surviving into the intelligence explosion.
RE: #1: do you have a suggestion for how someone who is not an AI researcher could tell if MIRI’s work is diminishing? I think your suggestion is to ask experts—apart from Holden, have any experts reviewed MIRI?
Explicit reviews of MIRI as an organization aren’t the only kind of review of MIRI. It also counts as a review of MIRI, at least weakly, if anyone competent enough to evaluate any of MIRI’s core claims comes out in favor of (or opposition to) any of those claims, or chooses to work with MIRI. David Chalmers’ The Singularity: A Philosophical Analysis and the follow-up collectively provide very strong evidence that analytic philosophers have no compelling objection to the intelligence explosion prediction, for example, and that a number of them share it. Reviews of MIRI’s five theses and specific published works are likely to give us better long-term insight into whether MIRI’s on the right track (relative to potential FAI researcher competitors) than a review focused on, e.g., MIRI’s organizational structure or use of funding, both of which are more malleable than its basic epistemic methodology and outlook.
It’s also important to keep in mind that the best way to figure out whether MIRI’s useless is probably to fund MIRI. Give them $150M, earmarked for foundational research that will see clear results within a decade, and wait 15-20 years. If what they’re doing is useless, it will be far more obvious when we’ve seen them do a lot more of it; and any blind alleys they go down will help clarify what they (or a rival research team) should be working on instead. At this point MIRI is very much a neonate, if not a partly-developed fetus. Speeding its development would both make us able to fairly evaluate it much more quickly, and encourage other researchers to get into the business.
No. I’m pretty confident the chance of MIRI failing due to cobalt bombs is <<1%, given that none exist, there are no known plans to build any, and it would still need to be used to halt progress. Also, the use of enough cobalt bombs to destroy MIRI’s relevance (remember, MIRI has supports in many different nations, who would presumably a) remain concerned and b) attempt to carry on research if there aren’t pressing concerns) presupposes a global nuclear exchange, which would make MIRI irrelevant either way.
(Irrelevant in the sense that all of MIRI’s research and writings would be lost, and there wouldn’t be enough tech left for people to remember MIRI’s research program by the time they would be able to restart research again. I am not claiming that a global nuclear exchange would be an existential risk.)
That isn’t what you said though. You were talking about the discovery, the very existence of a weapon able to reliably kill everyone. You’d need a lot fewer cobalt bombs to salt the earth with lethal amounts of fallout than you’d need to melt everything to slag, too.
1: The methods for constructing a nuclear bomb are by no means “widely known and easy to follow.” Witness the often unsuccessful struggles of many nations for decades to acquire them. Cobalt bombs are even more advanced and difficult to construct than ‘regular’ nuclear weapons.
The scenario RobBB was presumably envisioning was one in which private individuals have gained the ability to essentially destroy society using, e.g. a super-pandemic or something. A sufficient number of people have always been able to destroy human society; no new technology would be needed for everybody in the world to simultaneously commit suicide, or, for that matter, for a massive nuclear exchange. Spontaneous collective suicide is not likely. However, actions by a small group of individuals (c.f. al-Qaeda) are far more likely. Such groups do not at present have the ability to end human society as we know it; RobBB is envisioning a scenario where they gain it.
2: In the above comment, I wasn’t talking about existential risks [*]; I am not claiming that a nuclear war would be an existential risk. For any conventional nuclear war of significant size, SF/Berkley would almost certainly be targeted, killing everybody at MIRI. While I am unsure where the physical location of the servers storing their website/other data is, it’s overwhelmingly likely that EMP from nuclear detonations would destroy the ability of anybody to access that data. Given the geographic distribution of LWers, it is likely that only a small-ish number would survive a massive nuclear exchange. Presumably at least a few of these individuals would attempt to carry on research and to recopy MIRI’s research/ideas for future generations, assuming that humanity will eventually recover. However, it is extremely unlikely that they will get very far, and very likely that whatever they do write down will be lost or ignored.
[*] Rereading the comment, I actually was talking about existential risks. I have edited it for clarity; I was not intending to, but adopted RobBB’s phrasing of something killing us all, while I regard nuclear risks as more likely to render MIRI useless by collapsing society. My bad.
Sorry about that. I got confused. s/you/RobBB/. I understand better now. I still believe that of the five, 3 is probably the most likely. I also 2-believe that I might overestimate that probability. (Sorry if I sound a bit strange. I’m starting to study lojban.)
I think this question is too vague. MIRI could turn out to be useless for any number of reasons, leading to different empirical disconfirmations. (A lot of these will look like the end of human life.) E.g.:
MIRI is useless because FAI research is very useful, but MIRI’s basic methodology or research orientation is completely and irredeemably the wrong approach to FAI. Expected evidence: MIRI’s research starts seeing diminishing returns, even as they attempt a wide variety of strategies; non-MIRI researchers make surprising amounts of progress into the issues MIRI is interested in; reputable third parties that assess MIRI’s results consistently disagree with its foundational assumptions or methodology; the researchers MIRI attracts come to be increasingly seen by the academic establishment as irrelevant to FAI research or even as cranks, for substantive, mathy reasons.
MIRI is useless because FAI is impossible—we simply lack the resources to engineer a benign singularity, no matter how hard we try. Expected evidence: Demonstrations that a self-modifying AGI can’t have stable, predictable values; demonstrations that coding anything like indirect normativity is unfeasible; increasingly slow progress, e.g., due to fixed hardware limitations.
MIRI is useless because some defeater will inevitably kill us all (or otherwise halt technological progress) before we have time to produce self-enhancing AGI. (Alternatively: UFAI is completely impossible to prevent, by any means.) Expected evidence: Discovery of some weapon (e.g., a supervirus or bomb) that can reliably kill everyone and has widely known and easy-to-follow engineering specifications.
MIRI is useless because some defeater will kill us all before we build an AGI, with a probability high enough to suck all the expected utility out of FAI research. I.e., our survival until the intelligence explosion singularity is a possibility, but not a large enough possibility to make the value of surviving past that sieve worth paying attention to. Expected evidence: Discovery that near-term existential threats are vastly more threatening than currently thought.
MIRI is useless because FAI is inevitable. Expected evidence: Black-swan disproof of fragility-of-value and complexity-of-value, revolutionizing human psychology.
Note that these scenarios not only would be indicated by different evidence, but also call for very different responses.
If 1 is true: Instead of funding MIRI, we should fund some radically different FAI research program.
If 2 is true: Instead of funding MIRI, we should pour all our resources into permanently stopping AI research, by any available means.
If 3 is true: Instead of funding MIRI, we should just enjoy what little time we have left before the apocalypse.
If 4 is true: Instead of funding MIRI, we should combat other existential risks, or act as in 3.
If 5 is true: Instead of funding MIRI, we should speed along AI research and fund cryonic and life-extension technologies in the hopes of surviving into the intelligence explosion.
This is great RobBB, thanks!
RE: #1: do you have a suggestion for how someone who is not an AI researcher could tell if MIRI’s work is diminishing? I think your suggestion is to ask experts—apart from Holden, have any experts reviewed MIRI?
Explicit reviews of MIRI as an organization aren’t the only kind of review of MIRI. It also counts as a review of MIRI, at least weakly, if anyone competent enough to evaluate any of MIRI’s core claims comes out in favor of (or opposition to) any of those claims, or chooses to work with MIRI. David Chalmers’ The Singularity: A Philosophical Analysis and the follow-up collectively provide very strong evidence that analytic philosophers have no compelling objection to the intelligence explosion prediction, for example, and that a number of them share it. Reviews of MIRI’s five theses and specific published works are likely to give us better long-term insight into whether MIRI’s on the right track (relative to potential FAI researcher competitors) than a review focused on, e.g., MIRI’s organizational structure or use of funding, both of which are more malleable than its basic epistemic methodology and outlook.
It’s also important to keep in mind that the best way to figure out whether MIRI’s useless is probably to fund MIRI. Give them $150M, earmarked for foundational research that will see clear results within a decade, and wait 15-20 years. If what they’re doing is useless, it will be far more obvious when we’ve seen them do a lot more of it; and any blind alleys they go down will help clarify what they (or a rival research team) should be working on instead. At this point MIRI is very much a neonate, if not a partly-developed fetus. Speeding its development would both make us able to fairly evaluate it much more quickly, and encourage other researchers to get into the business.
Isn’t 3 already pretty much the case thanks to things like cobalt bombs?
No. I’m pretty confident the chance of MIRI failing due to cobalt bombs is <<1%, given that none exist, there are no known plans to build any, and it would still need to be used to halt progress. Also, the use of enough cobalt bombs to destroy MIRI’s relevance (remember, MIRI has supports in many different nations, who would presumably a) remain concerned and b) attempt to carry on research if there aren’t pressing concerns) presupposes a global nuclear exchange, which would make MIRI irrelevant either way.
(Irrelevant in the sense that all of MIRI’s research and writings would be lost, and there wouldn’t be enough tech left for people to remember MIRI’s research program by the time they would be able to restart research again. I am not claiming that a global nuclear exchange would be an existential risk.)
That isn’t what you said though. You were talking about the discovery, the very existence of a weapon able to reliably kill everyone. You’d need a lot fewer cobalt bombs to salt the earth with lethal amounts of fallout than you’d need to melt everything to slag, too.
(I am not RobbBB.)
1: The methods for constructing a nuclear bomb are by no means “widely known and easy to follow.” Witness the often unsuccessful struggles of many nations for decades to acquire them. Cobalt bombs are even more advanced and difficult to construct than ‘regular’ nuclear weapons.
The scenario RobBB was presumably envisioning was one in which private individuals have gained the ability to essentially destroy society using, e.g. a super-pandemic or something. A sufficient number of people have always been able to destroy human society; no new technology would be needed for everybody in the world to simultaneously commit suicide, or, for that matter, for a massive nuclear exchange. Spontaneous collective suicide is not likely. However, actions by a small group of individuals (c.f. al-Qaeda) are far more likely. Such groups do not at present have the ability to end human society as we know it; RobBB is envisioning a scenario where they gain it.
2: In the above comment, I wasn’t talking about existential risks [*]; I am not claiming that a nuclear war would be an existential risk. For any conventional nuclear war of significant size, SF/Berkley would almost certainly be targeted, killing everybody at MIRI. While I am unsure where the physical location of the servers storing their website/other data is, it’s overwhelmingly likely that EMP from nuclear detonations would destroy the ability of anybody to access that data. Given the geographic distribution of LWers, it is likely that only a small-ish number would survive a massive nuclear exchange. Presumably at least a few of these individuals would attempt to carry on research and to recopy MIRI’s research/ideas for future generations, assuming that humanity will eventually recover. However, it is extremely unlikely that they will get very far, and very likely that whatever they do write down will be lost or ignored.
[*] Rereading the comment, I actually was talking about existential risks. I have edited it for clarity; I was not intending to, but adopted RobBB’s phrasing of something killing us all, while I regard nuclear risks as more likely to render MIRI useless by collapsing society. My bad.
Sorry about that. I got confused. s/you/RobBB/. I understand better now. I still believe that of the five, 3 is probably the most likely. I also 2-believe that I might overestimate that probability. (Sorry if I sound a bit strange. I’m starting to study lojban.)