You are asking other people for their money and time, when they have other opportunities. To do that they need an estimate of the chance of MIRI succeeding
No they don’t; they could be checking relative plausibility of causing an OK outcome without trying to put absolute numbers on a probability estimate, and this is reasonable due to the following circumstances:
The life lesson I’ve learned is that by the time you really get anywhere, if you get anywhere, you’ll have encountered some positive miracles, some negative miracles, your plans will have changed, you’ll have found that the parts which took the longest weren’t what you thought they would be, and that other things proved to be much easier than expected. Your successes won’t come through foreseen avenues, and neither will your failures. But running through it all will be the fundamental realization that everything you accomplished, and all the unforeseen opportunities you took advantage of, were things that you would never have received if you hadn’t attacked the hardest part of the problem that you knew about straight-on, without distraction.
How do you estimate probabilities like that? I honestly haven’t a clue. Now, we all still have to maximize expected utility, but the heuristic I’m applying to do that (which at the meta level I think is the planning heuristic with the best chance of actually working) is to ask “Is there any better way of attacking the hardest part of the problem?” or “Is there any better course of action which doesn’t rely on someone else performing a miracle?” So far as I can tell, these other proposed courses of action don’t attack the hardest part of the problem for humanity’s survival, but rely on someone else performing a miracle. I cannot make myself believe that this would really actually work. (And System 2 agrees that System 1′s inability to really believe seems well-founded.)
Since I’m acting on such reasons and heuristics as “If you don’t attack the hardest part of the problem, no one else will” and “Beware of taking the easy way out” and “Don’t rely on someone else to perform a miracle”, I am indeed willing to term what I’m doing “heroic epistemology”. It’s just that I think such reasoning is, you know, actually correct and normative under these conditions.
If you don’t mind mixing the meta-level and the object-level, then I find any reasoning along the lines of “The probability of our contributing to solving FAI is too low, maybe we can have a larger impact by working on synthetic biology defense and hoping a miracle happens elsewhere” much less convincing than the meta-level observation, “That’s a complete Hail Mary pass, if there’s something you think is going to wipe out humanity then just work on that directly as your highest priority.” All the side cleverness, on my view, just adds up to losing the chance that you get by engaging directly with the problem and everything unforeseen that happens from there.
Another way of phrasing this is that if we actually win, I fully expect the counterfactual still-arguing-about-this version of 2013-Carl to say, “But we succeeded through avenue X, while you were then advocating avenue Y, which I was right to say wouldn’t work.” And to this the counterfactual reply of Eliezer will be, “But Carl, if I’d taken your advice back then, I wouldn’t have stayed engaged with the problem long enough to discover and comprehend avenue X and seize that opportunity, and this part of our later conversation was totally foreseeable in advance.” Hypothetical oblivious!Carl then replies, “But the foreseeable probability should still have been very low” or “Maybe you or someone else would’ve tried Y without that detour, if you’d worked on Z earlier” where Z was not actually uniquely suggested as the single best alternative course of action at the time. If there’s a reply that counterfactual non-oblivious Carl can make, I can’t foresee it from here, under those hypothetical circumstances unfolding as I describe (and you shouldn’t really be trying to justify yourself under those hypothetical circumstances, any more than I should be making excuses in advance for what counterfactual Eliezer says after failing, besides “Oops”).
My reasoning here is, from my internal perspective, very crude, because I’m not sure I really actually trust non-crude reasoning. There’s this killer problem that’s going to make all that other stuff pointless. I see a way to make progress on it, on the object level; the next problem up is visible and can be attacked. (Even this wasn’t always true, and I stuck with the problem anyway long enough to get to the point where I could state the tiling problem.) Resources should go to attacking this visible next step on the hardest problem. An exception to this as top priority maximization was CFAR, via “teaching rationality demonstrably channels more resources toward FAI; and CFAR which will later be self-sustaining is just starting up; plus CFAR might be useful for a general saving throw bonus; plus if a rational EA community had existed in 1996 it would have shaved ten years off the timeline and we could easily run into that situation again; plus I’m not sure MIRI will survive without CFAR”. Generalizing, young but hopefully self-sustaining initiatives can be plausibly competitive with MIRI for small numbers of marginal dollars, provided that they’re sufficiently directly linked to FAI down the road. Short of that, it doesn’t really make sense to ignore the big killer problem and hope somebody else handles it later. Not really actually.
If the year was 1960, which would you rather have?
10 smart people trying to build FAI for 20 years, 1960-1980
A billion dollars, a large supporting movement, prediction markets and science courts that make the state of the evidence on AI transparent, and teams working on FAI, brain emulations, cognitive enhancement, and more but starting in 1990 (in expectation closer to AI)
At any given time there are many problems where solutions are very important, but the time isn’t yet right to act on them, rather than on the capabilities to act on them, and also to deal with the individually unexpected problems that come along so regularly. Investment-driven and movement-building-driven discount rates are relevant even for existential risk.
GiveWell has grown in influence much faster than the x-risk community while working on global health, and are now in the process of investigating and pivoting towards higher leverage causes, with global catastrophic risk among the top three under consideration.
I’d rather have both, hence diverting some marginal resources to CFAR until it was launched, then switching back to MIRI. Is there a third thing that MIRI should divert marginal resources to right now?
I have just spent a month in England interacting extensively with the EA movement here (maybe your impressions from the California EA summit differ, I’d be curious to hear). Donors interested in the far future are also considering donations to the following (all of these are from talks with actual people making concrete short-term choices; in addition to donations, people are also considering career choices post-college):
80,000 Hours, CEA and other movement building and capacity-increasing organizations (including CFAR), which also increase non-charity options (e.g. 80k helping people going into scientific funding agencies and political careers where they will be in a position to affect research and policy reactions to technologies relevant to x-risk and other trajectory changes)
AMF/GiveWell charities to keep GiveWell and the EA movement growing while actors like GiveWell, Paul Christiano, Nick Beckstead and others at FHI, investigate the intervention options and cause prioritization, followed by organization-by-organization analysis of the GiveWell variety, laying the groundwork for massive support for the top far future charities and organizations identified by said processes
Finding ways to fund such evaluation with RFMF, e.g. by paying for FHI or CEA hires to work on them
The FHI’s other work
A donor-advised fund investing the returns until such evaluations or more promising opportunities present themselves or are elicited by the fund (possibilities like Drexler’s nanotech panel, extensions of the DAGGRE methods, a Bayesian aggregation algorithm that greatly improves extraction of scientific expert opinion or science courts that could mobilize much more talent and resources to neglected problems with good cases, some key steps in biotech enhancement)
That’s why Peter Hurford posted the OP, because he’s an EA considering all these options, and wants to compare them to MIRI.
That is a sort of discussion my brain puts in a completely different category. Peter and Carl, please always give me a concrete alternative policy option that (allegedly) depends on a debate, if such is available; my brain is then far less likely to label the conversation “annoying useless meta objections that I want to just get over with as fast as possible”.
AMF/GiveWell charities to keep GiveWell and the EA movement growing while actors like GiveWell, Paul Christiano, Nick Beckstead and others at FHI, investigate the intervention options and cause prioritization, followed by organization-by-organization analysis of the GiveWell variety, laying the groundwork for massive support for the top far future charities and organizations identified by said processes
Cool, if MIRI keeps going, they might be able to show FAI as top focus with adequate evidence by the time all of this comes together.
Well, in collaboration with FHI. As soon as Bostrom’s Superintelligence is released, we’ll probably be building on and around that to make whatever cases we think are reasonable to make.
they could be checking relative plausibility of causing an OK outcome without trying to put absolute numbers on a probability estimate, and this is reasonable due to the following circumstances
No they don’t; they could be checking relative plausibility of causing an OK outcome without trying to put absolute numbers on a probability estimate, and this is reasonable due to the following circumstances:
The life lesson I’ve learned is that by the time you really get anywhere, if you get anywhere, you’ll have encountered some positive miracles, some negative miracles, your plans will have changed, you’ll have found that the parts which took the longest weren’t what you thought they would be, and that other things proved to be much easier than expected. Your successes won’t come through foreseen avenues, and neither will your failures. But running through it all will be the fundamental realization that everything you accomplished, and all the unforeseen opportunities you took advantage of, were things that you would never have received if you hadn’t attacked the hardest part of the problem that you knew about straight-on, without distraction.
How do you estimate probabilities like that? I honestly haven’t a clue. Now, we all still have to maximize expected utility, but the heuristic I’m applying to do that (which at the meta level I think is the planning heuristic with the best chance of actually working) is to ask “Is there any better way of attacking the hardest part of the problem?” or “Is there any better course of action which doesn’t rely on someone else performing a miracle?” So far as I can tell, these other proposed courses of action don’t attack the hardest part of the problem for humanity’s survival, but rely on someone else performing a miracle. I cannot make myself believe that this would really actually work. (And System 2 agrees that System 1′s inability to really believe seems well-founded.)
Since I’m acting on such reasons and heuristics as “If you don’t attack the hardest part of the problem, no one else will” and “Beware of taking the easy way out” and “Don’t rely on someone else to perform a miracle”, I am indeed willing to term what I’m doing “heroic epistemology”. It’s just that I think such reasoning is, you know, actually correct and normative under these conditions.
If you don’t mind mixing the meta-level and the object-level, then I find any reasoning along the lines of “The probability of our contributing to solving FAI is too low, maybe we can have a larger impact by working on synthetic biology defense and hoping a miracle happens elsewhere” much less convincing than the meta-level observation, “That’s a complete Hail Mary pass, if there’s something you think is going to wipe out humanity then just work on that directly as your highest priority.” All the side cleverness, on my view, just adds up to losing the chance that you get by engaging directly with the problem and everything unforeseen that happens from there.
Another way of phrasing this is that if we actually win, I fully expect the counterfactual still-arguing-about-this version of 2013-Carl to say, “But we succeeded through avenue X, while you were then advocating avenue Y, which I was right to say wouldn’t work.” And to this the counterfactual reply of Eliezer will be, “But Carl, if I’d taken your advice back then, I wouldn’t have stayed engaged with the problem long enough to discover and comprehend avenue X and seize that opportunity, and this part of our later conversation was totally foreseeable in advance.” Hypothetical oblivious!Carl then replies, “But the foreseeable probability should still have been very low” or “Maybe you or someone else would’ve tried Y without that detour, if you’d worked on Z earlier” where Z was not actually uniquely suggested as the single best alternative course of action at the time. If there’s a reply that counterfactual non-oblivious Carl can make, I can’t foresee it from here, under those hypothetical circumstances unfolding as I describe (and you shouldn’t really be trying to justify yourself under those hypothetical circumstances, any more than I should be making excuses in advance for what counterfactual Eliezer says after failing, besides “Oops”).
My reasoning here is, from my internal perspective, very crude, because I’m not sure I really actually trust non-crude reasoning. There’s this killer problem that’s going to make all that other stuff pointless. I see a way to make progress on it, on the object level; the next problem up is visible and can be attacked. (Even this wasn’t always true, and I stuck with the problem anyway long enough to get to the point where I could state the tiling problem.) Resources should go to attacking this visible next step on the hardest problem. An exception to this as top priority maximization was CFAR, via “teaching rationality demonstrably channels more resources toward FAI; and CFAR which will later be self-sustaining is just starting up; plus CFAR might be useful for a general saving throw bonus; plus if a rational EA community had existed in 1996 it would have shaved ten years off the timeline and we could easily run into that situation again; plus I’m not sure MIRI will survive without CFAR”. Generalizing, young but hopefully self-sustaining initiatives can be plausibly competitive with MIRI for small numbers of marginal dollars, provided that they’re sufficiently directly linked to FAI down the road. Short of that, it doesn’t really make sense to ignore the big killer problem and hope somebody else handles it later. Not really actually.
If the year was 1960, which would you rather have?
10 smart people trying to build FAI for 20 years, 1960-1980
A billion dollars, a large supporting movement, prediction markets and science courts that make the state of the evidence on AI transparent, and teams working on FAI, brain emulations, cognitive enhancement, and more but starting in 1990 (in expectation closer to AI)
At any given time there are many problems where solutions are very important, but the time isn’t yet right to act on them, rather than on the capabilities to act on them, and also to deal with the individually unexpected problems that come along so regularly. Investment-driven and movement-building-driven discount rates are relevant even for existential risk.
GiveWell has grown in influence much faster than the x-risk community while working on global health, and are now in the process of investigating and pivoting towards higher leverage causes, with global catastrophic risk among the top three under consideration.
I’d rather have both, hence diverting some marginal resources to CFAR until it was launched, then switching back to MIRI. Is there a third thing that MIRI should divert marginal resources to right now?
I have just spent a month in England interacting extensively with the EA movement here (maybe your impressions from the California EA summit differ, I’d be curious to hear). Donors interested in the far future are also considering donations to the following (all of these are from talks with actual people making concrete short-term choices; in addition to donations, people are also considering career choices post-college):
80,000 Hours, CEA and other movement building and capacity-increasing organizations (including CFAR), which also increase non-charity options (e.g. 80k helping people going into scientific funding agencies and political careers where they will be in a position to affect research and policy reactions to technologies relevant to x-risk and other trajectory changes)
AMF/GiveWell charities to keep GiveWell and the EA movement growing while actors like GiveWell, Paul Christiano, Nick Beckstead and others at FHI, investigate the intervention options and cause prioritization, followed by organization-by-organization analysis of the GiveWell variety, laying the groundwork for massive support for the top far future charities and organizations identified by said processes
Finding ways to fund such evaluation with RFMF, e.g. by paying for FHI or CEA hires to work on them
The FHI’s other work
A donor-advised fund investing the returns until such evaluations or more promising opportunities present themselves or are elicited by the fund (possibilities like Drexler’s nanotech panel, extensions of the DAGGRE methods, a Bayesian aggregation algorithm that greatly improves extraction of scientific expert opinion or science courts that could mobilize much more talent and resources to neglected problems with good cases, some key steps in biotech enhancement)
That’s why Peter Hurford posted the OP, because he’s an EA considering all these options, and wants to compare them to MIRI.
That is a sort of discussion my brain puts in a completely different category. Peter and Carl, please always give me a concrete alternative policy option that (allegedly) depends on a debate, if such is available; my brain is then far less likely to label the conversation “annoying useless meta objections that I want to just get over with as fast as possible”.
Can we have a new top-level comment on this?
I edited my top-level comment to include the list and explanation.
Cool, if MIRI keeps going, they might be able to show FAI as top focus with adequate evidence by the time all of this comes together.
Well, in collaboration with FHI. As soon as Bostrom’s Superintelligence is released, we’ll probably be building on and around that to make whatever cases we think are reasonable to make.
Thanks; I found this comment illuminating about your views.
Also, because this.