I know that FHI plans to produce a particular set of policy recommendations relevant to superintelligence upon the release of Nick’s book or shortly thereafter. FHI has given no timeline for Nick’s book but I expect it to be published in mid or late 2013.
The comparably detailed document from SI will be the AI risk wiki. We think the wiki format makes even more sense than a book for these purposes, though an OUP book on superintelligence from Nick Bostrom sounds great to us. Certainly, we will be busy with other projects, but even still I think the AI risk wiki (a fairly comprehensive version 1.0, anyway) could be finished within 2 years. I’m not that confident it will be finished in 2 years, though, given that we’ve barely begun. Six months from now I’ll be more confidently able to predict the likelihood of finishing the AI risk wiki version 1.0 within 2 years.
Despite this, I would describe the current situation as “bogged down” when it comes to singularity strategy. Luckily, the situation is changing due to 2 recent game-shifting events: (1) FHI decided to spend a few years focusing on AI risk strategy while Nick wrote a monograph on the subject, and (2) shortly thereafter, SI began to rapidly grow its research team (at first, mostly through part-time remote researchers) and use that team to produce a lot more research writing than before (only a small fraction of which you’ve seen thus far).
And no, I don’t know in advance what strategic recommendations FHI will arrive at, nor which strategic recommendations SI’s scholarly AI risk wiki will arrive at, except to say that SI’s proposals will probably include Friendly AI research as one of the very important things humanity should be doing right now about AI risk.
ETA: My answer to your original question — “Why haven’t SI and LW attracted or produced any good strategists?” — is that it’s very difficult and time-consuming to acquire all the domain knowledge required to be good at singularity strategy, especially when things like a book-length treatment of AI risk and an AI risk wiki don’t yet exist. It will be easier for someone to become good at singularity strategy once those things exist, but even still they’ll have to know a lot about technological development and forecasting, narrow AI, AGI, FAI open problems, computer science, maths, the physical sciences, economics, philosophy of science, value theory, anthropics, and quite a bit more.
What methodology will be used to produce SI’s strategic recommendations (and FHI’s, if you know the answer)? As far as I can tell, we currently don’t have a way to make the many known strategic considerations/arguments commensurable (e.g., suitable for integrating into a quantitative strategic framework) except by using our intuitions which seem especially unreliable on matters related to Singularity strategy. The fact that you think the AI risk wiki can be finished in 2 years seems to indicate that you either disagree with this evaluation of the current state of affairs, or think we can make very rapid progress in strategic reasoning. Can you explain?
We certainly could integrate known strategic arguments into a quantitative framework like this, but I’m worried that, for example, “putting so many made-up probabilities into a probability tree like this is not actually that helpful.”
I think for now both SI and FHI are still in the qualitative stage that normally precedes quantitative analysis. Big projects like Nick’s monograph and SI’s AI risk wiki will indeed constitute “rapid progress” in strategic reasoning, but it will be rapid progress toward more quantitative analyses, not rapid progress within a quantitative framework that we have already built.
Of course, some of the work on strategic sub-problems is already at the quantitative/formal stage, so quantitative/formal progress can be made on them immediately if SI/FHI can raise the resources to find and hire the right people to work on them. Two examples: (1) What do reasonable economic models of past jumps in optimization power imply about what would happen once we get self-improving AGI? (2) If we add lots more AI-related performance curve data to Nagy’s Performance Curve Database and use his improved tech forecasting methods, what does it all imply about AI and WBE timelines?
I think for now both SI and FHI are still in the qualitative stage that normally precedes quantitative analysis.
There are many strategic considerations that greatly differ in nature from one another. It seems to me that at best they will require diverse novel methods to analyze quantitatively, and at worst a large fraction may resist attempts at quantitative analysis until the Singularity occurs.
For example we can see that there is an upper bound on how confident a small FAI team, working in secret and with limited time, can be (assuming it’s rational) about the correctness of an FAI design, due to the issue raised in my comment quoted by Holden, and this is of obvious strategic importance. But I have no idea what method we can use to derive this bound, other than to “make it up”. Solving this problem alone could easily take a team several years to accomplish, so how do you hope to produce the strategic recommendations, which must take into account many such issues, in 2 years?
Solving this problem alone could easily take a team several years to accomplish, so how do you hope to produce the strategic recommendations, which must take into account many such issues, in 2 years?
Two answers:
Obviously, our recommendations won’t be final, and we’ll try to avoid being overconfident — especially where the recommendations depend on highly uncertain variables.
In many (most?) cases, I suspect our recommendations will be for policies that play a dual role of (1) making progress in directions that look promising from where we stand now, and also (2) purchasing highly valuable information, like how feasible an NGO FAI team is, how hard FAI really is, what the failure modes look like, how plausible alternative approaches are, etc.
SI, FHI, you, others — we’re working on tough problems with many unknown and uncertain strategic variables. Those challenges are not unique to AI risk. Humans have manytools for doing the best they can while running on spaghetti code and facing decision problems under uncertainty, and we’re gaining new tools all the time.
I don’t mean to minimize your concerns, though. Right now I expect to fail. I expect us all to get paperclipped (or turned off), though I’ll be happy to update in favor of positive outcomes if (1) research shows the problem isn’t as hard as I now think, (2) financial support for x-risk reduction increases, (3) etc.
I don’t mean to minimize your concerns, though. Right now I expect to fail. I expect us all to get paperclipped (or turned off), though I’ll be happy to update in favor of positive outcomes if (1) research shows the problem isn’t as hard as I now think, (2) financial support for x-risk reduction increases, (3) etc.
I think you may have misunderstood my intent here. I’m not trying to make you more pessimistic about our overall prospects but arguing (i.e., trying to figure out) the absolute and relative importance of solving various strategic problems.
Another point was to suggest that perhaps SI ought to give higher priority to recruiting/training “hero strategists” as opposed to “hero mathematicians”. For example your So You Want to Save the World says:
No, the world must be saved by mathematicians, computer scientists, and philosophers.
which fails to credit the importance of strategic contributions (even though later in the post there is a large section on strategic problems).
which fails to credit the importance of strategic contributions
Sorry if that was unclear; I mean to identify the strategists as “philosophers”, like this. As you say, I went on to include a large section on strategy.
I certainly agree on the importance of strategy. Most of the research SI and FHI have done is strategic, after all — and most of the work in progress is strategic, too.
I do tend to talk a lot about “hero mathematicians,” though. Maybe that’s because “hero mathematician” is more concrete (to me) than “hero strategist.”
Anyway, it seems like we may be failing to disagree on anything, here.
Sorry if that was unclear; I mean to identify the strategists as “philosophers”, like this.
I see. I had interpreted you to mean philosophers as part of a team to build FAI.
I do tend to talk a lot about “hero mathematicians,” though. Maybe that’s because “hero mathematician” is more concrete (to me) than “hero strategist.”
What do you mean by “more concrete”, and do you think it’s a good reason to talk a lot more about “hero mathematicians”?
I had interpreted you to mean philosophers as part of a team to build FAI.
That could also be true, but I’m not sure.
Re: “hero mathematicians” and “hero strategists”, here’s a more detailed version of what I currently think.
Result of saying we need “hero mathematicians”? A few mathematicians (perhaps primed by HPMoR to be rationality heroes) come to us and learn what the technical research program looks like, help put our memes into the math community, etc.
Result of saying we need “hero strategists”? I’m inundated with people who say they can contribute to singularity strategy after thinking about the issues for one month and reading less than 100 pages on the subject. SI staff wastes valuable time trying to steer amateur strategists along more valuable paths before giving up due to low ROI.
Basically, the recruiting problem is different for mathematicians and strategists, and I think these problems can be tackled more effectively by tackling them separately. Mathematicians can prove themselves useful rather quickly, by offering constructive comments on the problems we will (in the next 12 months) have written up somewhat formally, or by spreading our memes in their research communities.
But to tell whether someone can be a useful strategist they need to read 500 pages of material and spend months chatting regularly with SI and/or FHI, and that’s very costly for both them and for SI+FHI.
The best result might be if some of the mathematicians themselves turn out to be good strategists. I don’t know that I can count on that, but for example I already count both you and Paul Christiano as among the few strategists whose strategy work I would spend my time reading, even though your primary life work has been in math and compsci (and not, say, civil engineering, business management, political science, or economics).
Result of saying we need “hero strategists”? I’m inundated with people who say they can contribute to singularity strategy after thinking about the issues for one month and reading less than 100 pages on the subject. SI staff wastes valuable time trying to steer amateur strategists along more valuable paths before giving up due to low ROI.
But to tell whether someone can be a useful strategist they need to read 500 pages of material and spend months chatting regularly with SI and/or FHI, and that’s very costly for both them and for SI+FHI.
You could direct them to LW and let them prove their mettle here?
I just tried to picture what “hero strategist” could mean, if distinct from ‘person who knows LW rationality’ or ‘practical guy like Luke’. I came up with someone who could hire the world’s best mathematicians plus a professional cat-herder and base the strategy on the result.
Right now I expect to fail. I expect us all to get paperclipped
So, you’re currently thinking hard about the best way to approach someone like Terence Tao? (Doesn’t have to be him, someone else’s blog might also have comments and give you a better opportunity to raise the issue.)
I know that FHI plans to produce a particular set of policy recommendations relevant to superintelligence upon the release of Nick’s book or shortly thereafter. FHI has given no timeline for Nick’s book but I expect it to be published in mid or late 2013.
The comparably detailed document from SI will be the AI risk wiki. We think the wiki format makes even more sense than a book for these purposes, though an OUP book on superintelligence from Nick Bostrom sounds great to us. Certainly, we will be busy with other projects, but even still I think the AI risk wiki (a fairly comprehensive version 1.0, anyway) could be finished within 2 years. I’m not that confident it will be finished in 2 years, though, given that we’ve barely begun. Six months from now I’ll be more confidently able to predict the likelihood of finishing the AI risk wiki version 1.0 within 2 years.
Despite this, I would describe the current situation as “bogged down” when it comes to singularity strategy. Luckily, the situation is changing due to 2 recent game-shifting events: (1) FHI decided to spend a few years focusing on AI risk strategy while Nick wrote a monograph on the subject, and (2) shortly thereafter, SI began to rapidly grow its research team (at first, mostly through part-time remote researchers) and use that team to produce a lot more research writing than before (only a small fraction of which you’ve seen thus far).
And no, I don’t know in advance what strategic recommendations FHI will arrive at, nor which strategic recommendations SI’s scholarly AI risk wiki will arrive at, except to say that SI’s proposals will probably include Friendly AI research as one of the very important things humanity should be doing right now about AI risk.
ETA: My answer to your original question — “Why haven’t SI and LW attracted or produced any good strategists?” — is that it’s very difficult and time-consuming to acquire all the domain knowledge required to be good at singularity strategy, especially when things like a book-length treatment of AI risk and an AI risk wiki don’t yet exist. It will be easier for someone to become good at singularity strategy once those things exist, but even still they’ll have to know a lot about technological development and forecasting, narrow AI, AGI, FAI open problems, computer science, maths, the physical sciences, economics, philosophy of science, value theory, anthropics, and quite a bit more.
What methodology will be used to produce SI’s strategic recommendations (and FHI’s, if you know the answer)? As far as I can tell, we currently don’t have a way to make the many known strategic considerations/arguments commensurable (e.g., suitable for integrating into a quantitative strategic framework) except by using our intuitions which seem especially unreliable on matters related to Singularity strategy. The fact that you think the AI risk wiki can be finished in 2 years seems to indicate that you either disagree with this evaluation of the current state of affairs, or think we can make very rapid progress in strategic reasoning. Can you explain?
We certainly could integrate known strategic arguments into a quantitative framework like this, but I’m worried that, for example, “putting so many made-up probabilities into a probability tree like this is not actually that helpful.”
I think for now both SI and FHI are still in the qualitative stage that normally precedes quantitative analysis. Big projects like Nick’s monograph and SI’s AI risk wiki will indeed constitute “rapid progress” in strategic reasoning, but it will be rapid progress toward more quantitative analyses, not rapid progress within a quantitative framework that we have already built.
Of course, some of the work on strategic sub-problems is already at the quantitative/formal stage, so quantitative/formal progress can be made on them immediately if SI/FHI can raise the resources to find and hire the right people to work on them. Two examples: (1) What do reasonable economic models of past jumps in optimization power imply about what would happen once we get self-improving AGI? (2) If we add lots more AI-related performance curve data to Nagy’s Performance Curve Database and use his improved tech forecasting methods, what does it all imply about AI and WBE timelines?
There are many strategic considerations that greatly differ in nature from one another. It seems to me that at best they will require diverse novel methods to analyze quantitatively, and at worst a large fraction may resist attempts at quantitative analysis until the Singularity occurs.
For example we can see that there is an upper bound on how confident a small FAI team, working in secret and with limited time, can be (assuming it’s rational) about the correctness of an FAI design, due to the issue raised in my comment quoted by Holden, and this is of obvious strategic importance. But I have no idea what method we can use to derive this bound, other than to “make it up”. Solving this problem alone could easily take a team several years to accomplish, so how do you hope to produce the strategic recommendations, which must take into account many such issues, in 2 years?
Two answers:
Obviously, our recommendations won’t be final, and we’ll try to avoid being overconfident — especially where the recommendations depend on highly uncertain variables.
In many (most?) cases, I suspect our recommendations will be for policies that play a dual role of (1) making progress in directions that look promising from where we stand now, and also (2) purchasing highly valuable information, like how feasible an NGO FAI team is, how hard FAI really is, what the failure modes look like, how plausible alternative approaches are, etc.
SI, FHI, you, others — we’re working on tough problems with many unknown and uncertain strategic variables. Those challenges are not unique to AI risk. Humans have many tools for doing the best they can while running on spaghetti code and facing decision problems under uncertainty, and we’re gaining new tools all the time.
I don’t mean to minimize your concerns, though. Right now I expect to fail. I expect us all to get paperclipped (or turned off), though I’ll be happy to update in favor of positive outcomes if (1) research shows the problem isn’t as hard as I now think, (2) financial support for x-risk reduction increases, (3) etc.
I think you may have misunderstood my intent here. I’m not trying to make you more pessimistic about our overall prospects but arguing (i.e., trying to figure out) the absolute and relative importance of solving various strategic problems.
Another point was to suggest that perhaps SI ought to give higher priority to recruiting/training “hero strategists” as opposed to “hero mathematicians”. For example your So You Want to Save the World says:
which fails to credit the importance of strategic contributions (even though later in the post there is a large section on strategic problems).
Sorry if that was unclear; I mean to identify the strategists as “philosophers”, like this. As you say, I went on to include a large section on strategy.
I certainly agree on the importance of strategy. Most of the research SI and FHI have done is strategic, after all — and most of the work in progress is strategic, too.
I do tend to talk a lot about “hero mathematicians,” though. Maybe that’s because “hero mathematician” is more concrete (to me) than “hero strategist.”
Anyway, it seems like we may be failing to disagree on anything, here.
I see. I had interpreted you to mean philosophers as part of a team to build FAI.
What do you mean by “more concrete”, and do you think it’s a good reason to talk a lot more about “hero mathematicians”?
That could also be true, but I’m not sure.
Re: “hero mathematicians” and “hero strategists”, here’s a more detailed version of what I currently think.
Result of saying we need “hero mathematicians”? A few mathematicians (perhaps primed by HPMoR to be rationality heroes) come to us and learn what the technical research program looks like, help put our memes into the math community, etc.
Result of saying we need “hero strategists”? I’m inundated with people who say they can contribute to singularity strategy after thinking about the issues for one month and reading less than 100 pages on the subject. SI staff wastes valuable time trying to steer amateur strategists along more valuable paths before giving up due to low ROI.
Basically, the recruiting problem is different for mathematicians and strategists, and I think these problems can be tackled more effectively by tackling them separately. Mathematicians can prove themselves useful rather quickly, by offering constructive comments on the problems we will (in the next 12 months) have written up somewhat formally, or by spreading our memes in their research communities.
But to tell whether someone can be a useful strategist they need to read 500 pages of material and spend months chatting regularly with SI and/or FHI, and that’s very costly for both them and for SI+FHI.
The best result might be if some of the mathematicians themselves turn out to be good strategists. I don’t know that I can count on that, but for example I already count both you and Paul Christiano as among the few strategists whose strategy work I would spend my time reading, even though your primary life work has been in math and compsci (and not, say, civil engineering, business management, political science, or economics).
You could direct them to LW and let them prove their mettle here?
I just tried to picture what “hero strategist” could mean, if distinct from ‘person who knows LW rationality’ or ‘practical guy like Luke’. I came up with someone who could hire the world’s best mathematicians plus a professional cat-herder and base the strategy on the result.
So, you’re currently thinking hard about the best way to approach someone like Terence Tao? (Doesn’t have to be him, someone else’s blog might also have comments and give you a better opportunity to raise the issue.)
Actually, yes. We had a meeting about that a couple weeks ago. Tao was specifically named. :)