I give them about 10 percent probability, and statements I’ve read from Eliezer and Luke cause me to believe they’re below medium in self-assessment nowadays, or their definition of “medium” may be lower than you think. I support them to increase the chance.
That said, low chances of success aren’t good for recruiting or motivation, and maximizing the probability requires as much effort and skill as possible to obtain.
Eliezer is very much against the idea of supporting MIRI based on a “low probability of really high impact” argument.
I hate to put words in his mouth, but I think
he means 0.0001% chance, not 10% chance. 10% is well within the range of probabilities humans can reason about (to the extent that humans can reason about any probabilities).
Eliezer thinks the case for MIRI does not depend on very small chances, and furthermore, is sceptical that these arguments are used in practice by Xrisk organisations, etc. He wouldn’t necessary turn away someone’s money who said “I’m donating because of a 10^-10 chance.” (though equally he might for PR/paternalistic reasons)
he means 0.0001% chance, not 10% chance. 10% is well within the range of probabilities humans can reason about (to the extent that humans can reason about any probabilities).
I base it on everything I’ve read and seen on technology, human nature, historical uses of power, trends in tech capabilities, the effects of intelligence, MIRI’s mission, team, and focus, and the greater realm of philanthropic endeavors.
If you mean, ‘you pulled that number out of your butt, and therefore I call you on it,’ then I’ll have to admit defeat due to inability to break it down quantitatively. Sorry.
2.) Eliezer is very much against the idea of supporting MIRI based on a “low probability of really high impact” argument. What do you think?
I think that’s taken out of context. The way I understand it, he means superintelligence will have a really high impact regardless (near 100% probability), and is therefore a ‘lever point’ which can have a higher probability of being impacted by anyone paying attention to it, and since MIRI is one of very few groups paying attention, they have a medium probability of being such an impactor.
If you mean, ‘you pulled that number out of your butt, and therefore I call you on it,’ then I’ll have to admit defeat due to inability to break it down quantitatively. Sorry.
Yeah. On one hand, I think there is something to be said about needing to make these fast and loose estimates and that there’s some basis for them. But on the other hand, I think one needs to recognize just how fast and loose they are. I think our error bars on MIRI’s chance of success is really high.
~
I think that’s taken out of context. The way I understand it, he means superintelligence will have a really high impact regardless (near 100% probability), and is therefore a ‘lever point’ which can have a higher probability of being impacted by anyone paying attention to it, and since MIRI is one of very few groups paying attention, they have a medium probability of being such an impactor.
Let me put that in premise-conclusion form:
P1: Superintelligence will, with probability greater than 99.999%, dramatically impact the future.
P2: One can change how superintelligence will unfold by working on superintelligence.
C3: Therefore from P1 and P2, working on superintelligence will dramatically impact the future.
P4: MIRI is one of the only groups working on superintelligence.
C5: Therefore from C3 and P4, MIRI will dramatically impact the future.
Do you think that’s right?
If so, I think P2 could be false, but I’ll accept it for the sake of argument. The real problem is, I think, C5 is a fallacy. It either assumes that any work in the domain will affect how superintelligence unfolds in a controlled way (which seems false) or that MIRI’s work will have impact (which seems unproven).
P1 is almost certainly an overestimate: independent of everything else, there’s almost certainly a greater than 0.001% chance that a civilization-ending event will occur before anyone gets around to building a superintelligence. The potential importance of AI research by way of this chain of logic wouldn’t be lowered too much if you used 80 or 90%, though.
I’m not sure which fallacy you’re invoking, but saying (to paraphrase), ‘superintelligence is likely difficult to aim’ and ‘MIRI’s work may not have an impact’ are certainly possible, and already contribute to my estimates.
The method is even more important (practice vs. perfect practice, philanthropy vs. givewell). I believe in the mission, not MIRI per se. If Eliezer decided that magic was the best way to achieve FAI and started searching for the right wand and hand gestures rather than math and decision theory, I would look elsewhere.
Your PredictionBook link says you reckon 10% probability of humanity still being around in ~100 years, but that’s not the same thing as MIRI succeeding. Superhuman AI might turn out to be beyond human capability, so we could survive without MIRI achieving anything. Superhuman UFAI might be feasible, and MIRI might successfully stop it happening (in which case I’d say they succeeded), but FAI might be just too hard or too weak and we might then get wiped out by something else. (I agree that that seems low probability.)
The full sentence reads: “MIRI exists to ensure that the creation of smarter-than-human intelligence has a positive impact.” (emphasis added) Clearly, if smarter-than-human intelligence ends up having a positive impact independently (or in spite of) MIRI’s efforts, that would count as a success only in a Pickwickian sort of sense. To succeed in the sense obviously intended by the authors of the mission statement, MIRI would have to be at least partially causally implicated in the process leading to the creation of FAI.
So the question remains: on what grounds do you believe that, if smarter-than-human intelligence ends up having a positive impact, this will be necessarily at least partly due to MIRI’s efforts? I find that view implausible, and instead agree with Carl Shulman that “the impact of MIRI in particular has to be far smaller subset of the expected impact of the cause as a whole,” for the reasons he mentions.
Surveys at the FHI’s Winter Intelligence conference, and sent out to AI experts showed median estimates of existential risk from AI as less likely than good outcomes from AI.
I would say we are probably not doomed by that channel, as would, e.g. Paul Christiano, although I would say (as do the surveys, including of neutral AI experts) the risk is significant.
I would say we are probably not doomed by that channel, as would, e.g. Paul Christiano, although I would say (as do the surveys, including of neutral AI experts) the risk is significant.
I agree with this, but I would be interested in knowing your own reasons for saying that we’re probably not doomed via AI risk. I spelled out some reasons for my own belief here.
Personally, I don’t think that there is that high of a chance that we’re doomed via AI risk because I think that the odds of “AI goes foom” are significantly lower then the MIRI people do. I think it’s somewhat more likely that subhuman-level GAI’s and vaugly human level AI’s increase the rate of AI research somewhat, but not to the dramatic levels the EY thinks. With a slower paced rate of AI development, we’re probably not dealing with a singleton AI, we have time to work to perfect friendly AI research while getting assistance from roughly human level AI’s, while slowly developing the slightly smarter AI’s, and while improving friendly-AI software with every step.
I don’t think the “AI goes foom” scenario is impossible, but I don’t put it at a high likelihood, maybe 10%-20%. I just don’t think that it’s all that likely that a human level GAI (say, one with the equivilent of IQ-90 to IQ-100) is able to rapidly turn itself into a IQ-300 GAI; if we can’t do it that quickly, then a AI of similar intelligence shouldn’t be able to do much better. And in the slower AI takeoff scenario I think is more likely, there is still some AI risk, but research in friendly AI we do now at this early stage is likely to be mostly made obsolete by research we do at that stage, and we’re more likely to adapt and tweak solutions rather then having to have a totally flawless solution ready the day before someone develops the first moderately useful-level GAI.
So i would probably put the total odds of MIRI directly having a big impact to be maybe around 5%. Which, considering the stakes involved, is still very significant and worth the cost, and there is also a significant change that their work has other useful spinoffs even if it turns out to not be necessary for the reason intended.
I give them about 10 percent probability, and statements I’ve read from Eliezer and Luke cause me to believe they’re below medium in self-assessment nowadays, or their definition of “medium” may be lower than you think. I support them to increase the chance.
That said, low chances of success aren’t good for recruiting or motivation, and maximizing the probability requires as much effort and skill as possible to obtain.
1.) What do you base the 10% estimate on?
2.) Eliezer is very much against the idea of supporting MIRI based on a “low probability of really high impact” argument. What do you think?
I hate to put words in his mouth, but I think
he means 0.0001% chance, not 10% chance. 10% is well within the range of probabilities humans can reason about (to the extent that humans can reason about any probabilities).
Eliezer thinks the case for MIRI does not depend on very small chances, and furthermore, is sceptical that these arguments are used in practice by Xrisk organisations, etc. He wouldn’t necessary turn away someone’s money who said “I’m donating because of a 10^-10 chance.” (though equally he might for PR/paternalistic reasons)
Where does this 10% probability come from?
Anchoring from my butt-number?
I believe the correct term is “ass-pull number.” :)
I base it on everything I’ve read and seen on technology, human nature, historical uses of power, trends in tech capabilities, the effects of intelligence, MIRI’s mission, team, and focus, and the greater realm of philanthropic endeavors.
If you mean, ‘you pulled that number out of your butt, and therefore I call you on it,’ then I’ll have to admit defeat due to inability to break it down quantitatively. Sorry.
I think that’s taken out of context. The way I understand it, he means superintelligence will have a really high impact regardless (near 100% probability), and is therefore a ‘lever point’ which can have a higher probability of being impacted by anyone paying attention to it, and since MIRI is one of very few groups paying attention, they have a medium probability of being such an impactor.
Yeah. On one hand, I think there is something to be said about needing to make these fast and loose estimates and that there’s some basis for them. But on the other hand, I think one needs to recognize just how fast and loose they are. I think our error bars on MIRI’s chance of success is really high.
~
Let me put that in premise-conclusion form:
P1: Superintelligence will, with probability greater than 99.999%, dramatically impact the future.
P2: One can change how superintelligence will unfold by working on superintelligence.
C3: Therefore from P1 and P2, working on superintelligence will dramatically impact the future.
P4: MIRI is one of the only groups working on superintelligence.
C5: Therefore from C3 and P4, MIRI will dramatically impact the future.
Do you think that’s right?
If so, I think P2 could be false, but I’ll accept it for the sake of argument. The real problem is, I think, C5 is a fallacy. It either assumes that any work in the domain will affect how superintelligence unfolds in a controlled way (which seems false) or that MIRI’s work will have impact (which seems unproven).
P1 is almost certainly an overestimate: independent of everything else, there’s almost certainly a greater than 0.001% chance that a civilization-ending event will occur before anyone gets around to building a superintelligence. The potential importance of AI research by way of this chain of logic wouldn’t be lowered too much if you used 80 or 90%, though.
I’m not sure which fallacy you’re invoking, but saying (to paraphrase), ‘superintelligence is likely difficult to aim’ and ‘MIRI’s work may not have an impact’ are certainly possible, and already contribute to my estimates.
I think a fair amount of people argue that because a cause is important, anyone working on that cause must be doing important work.
The method is even more important (practice vs. perfect practice, philanthropy vs. givewell). I believe in the mission, not MIRI per se. If Eliezer decided that magic was the best way to achieve FAI and started searching for the right wand and hand gestures rather than math and decision theory, I would look elsewhere.
Could you clarify your definition of success?
Your PredictionBook link says you reckon 10% probability of humanity still being around in ~100 years, but that’s not the same thing as MIRI succeeding. Superhuman AI might turn out to be beyond human capability, so we could survive without MIRI achieving anything. Superhuman UFAI might be feasible, and MIRI might successfully stop it happening (in which case I’d say they succeeded), but FAI might be just too hard or too weak and we might then get wiped out by something else. (I agree that that seems low probability.)
From MIRI’s mission statement: “the creation of smarter-than-human intelligence has a positive impact.”
I see smarter-than-human intelligence as required to overcome the combined threat of existential risks in the long run.
The full sentence reads: “MIRI exists to ensure that the creation of smarter-than-human intelligence has a positive impact.” (emphasis added) Clearly, if smarter-than-human intelligence ends up having a positive impact independently (or in spite of) MIRI’s efforts, that would count as a success only in a Pickwickian sort of sense. To succeed in the sense obviously intended by the authors of the mission statement, MIRI would have to be at least partially causally implicated in the process leading to the creation of FAI.
So the question remains: on what grounds do you believe that, if smarter-than-human intelligence ends up having a positive impact, this will be necessarily at least partly due to MIRI’s efforts? I find that view implausible, and instead agree with Carl Shulman that “the impact of MIRI in particular has to be far smaller subset of the expected impact of the cause as a whole,” for the reasons he mentions.
I subscribe to the view that AGI is bad by default, and don’t see anyone else working on the friendliness problem.
So we’re probably doomed then?
Surveys at the FHI’s Winter Intelligence conference, and sent out to AI experts showed median estimates of existential risk from AI as less likely than good outcomes from AI.
I would say we are probably not doomed by that channel, as would, e.g. Paul Christiano, although I would say (as do the surveys, including of neutral AI experts) the risk is significant.
I agree with this, but I would be interested in knowing your own reasons for saying that we’re probably not doomed via AI risk. I spelled out some reasons for my own belief here.
Personally, I don’t think that there is that high of a chance that we’re doomed via AI risk because I think that the odds of “AI goes foom” are significantly lower then the MIRI people do. I think it’s somewhat more likely that subhuman-level GAI’s and vaugly human level AI’s increase the rate of AI research somewhat, but not to the dramatic levels the EY thinks. With a slower paced rate of AI development, we’re probably not dealing with a singleton AI, we have time to work to perfect friendly AI research while getting assistance from roughly human level AI’s, while slowly developing the slightly smarter AI’s, and while improving friendly-AI software with every step.
I don’t think the “AI goes foom” scenario is impossible, but I don’t put it at a high likelihood, maybe 10%-20%. I just don’t think that it’s all that likely that a human level GAI (say, one with the equivilent of IQ-90 to IQ-100) is able to rapidly turn itself into a IQ-300 GAI; if we can’t do it that quickly, then a AI of similar intelligence shouldn’t be able to do much better. And in the slower AI takeoff scenario I think is more likely, there is still some AI risk, but research in friendly AI we do now at this early stage is likely to be mostly made obsolete by research we do at that stage, and we’re more likely to adapt and tweak solutions rather then having to have a totally flawless solution ready the day before someone develops the first moderately useful-level GAI.
So i would probably put the total odds of MIRI directly having a big impact to be maybe around 5%. Which, considering the stakes involved, is still very significant and worth the cost, and there is also a significant change that their work has other useful spinoffs even if it turns out to not be necessary for the reason intended.
I’m pessimistic and depressed.