Why haven’t SI and LW attracted or produced any good strategists? I’ve been given to understand (from someone close to SI) that various people within SI have worked on Singularity strategy but only produced lots of writings that are not of an organized, publishable form. Others have attempted to organize them but also failed, and there seems to be a general feeling that strategy work is bogged down or going in circles and any further effort will not be very productive. The situation on LW seems similar, with people arguing in various directions without much feeling of progress. Why are we so bad at this, given that strategic thinking must be a core part of rationality?
There are some but not lots of “writings” produced internally by SingInst that are not available to the public. There’s lots of scribbled notes and half-finished models and thoughts in brains and stuff like that. We’re working to push them out into written form, but that takes time, money, and people — and we’re short on all three.
The other problem is that to talk about strategy we first have to explain lots of things that are basic (to a veteran like you but not to most interested parties) in clear, well-organized language for the first time, since much of this hasn’t been done yet (these SI papers definitely help, though: 1, 2, 3, 4, 5, 6). To solve this problem we are (1) adding/improving lots of articles on the LW wiki like you suggested a while back (you’ll see a report on what we did, later), and (2) working on the AI risk wiki (we’re creating the map of articles right now). Once those resources are available it will be easier to speak clearly in public about strategic issues.
We hit a temporary delay in pushing out strategy stuff at SI because two of our most knowledgable researchers & strategists became unavailable for different reasons: Anna took over launching CFAR and Carl took an extended (unpaid) leave of absence to take care of some non-SI things. Also, I haven’t been able to continue my own AI risk strategy series due to other priorities, and because I got to the point where it was going to be a lot of work to continue that sequence if I didn’t already have clear, well-organized write-ups of lots of standard material. (So, it’ll be easier for me to continue once the LW wiki has been improved and once the AI risk wiki exists, both of which we’ve got people working on right now.)
Moreover, there are several papers in the works — mostly by Kaj (who is now a staff researcher), with some help from myself — but you won’t see them for a while. You did see this and this, however. Those are the product of months of part-time work from several remote researchers, plus analysis by Kaj and Stuart. Remote researchers are currently doing large lit reviews for other paper projects (at the pace of part-time work), but we aren’t at the stage of analyzing those large data sets yet so that we can write papers about what we found.
Also, a lot of work currently scattered around in papers and posts by SI and FHI people is being collected and tightly organized in Nick Bostrom’s forthcoming monograph on superintelligence, which will be a few hundred pages entirely about singularity strategy. Having much of “the basics” organized in that way will also make it easier to produce additional strategy work.
Luke, with the existing people at SI and FHI’s disposal, how long do you think it would take (assuming they’re not busy with other projects) to produce a document that lays out a cogent argument for some specific Singularity strategy? An argument that takes into account all of the important considerations that have already been raised (for example my comment that Holden quoted)? I will concede that strategy work is not bogged down if you think it can be done in a reasonable time frame. (2 years, perhaps?) But if SI and FHI are merely producing writings that explain the strategic considerations, but which we can’t foresee forming into an overall argument for some specific strategy, that seems very weak evidence at best against my claim that we are bad at strategic thinking.
I know that FHI plans to produce a particular set of policy recommendations relevant to superintelligence upon the release of Nick’s book or shortly thereafter. FHI has given no timeline for Nick’s book but I expect it to be published in mid or late 2013.
The comparably detailed document from SI will be the AI risk wiki. We think the wiki format makes even more sense than a book for these purposes, though an OUP book on superintelligence from Nick Bostrom sounds great to us. Certainly, we will be busy with other projects, but even still I think the AI risk wiki (a fairly comprehensive version 1.0, anyway) could be finished within 2 years. I’m not that confident it will be finished in 2 years, though, given that we’ve barely begun. Six months from now I’ll be more confidently able to predict the likelihood of finishing the AI risk wiki version 1.0 within 2 years.
Despite this, I would describe the current situation as “bogged down” when it comes to singularity strategy. Luckily, the situation is changing due to 2 recent game-shifting events: (1) FHI decided to spend a few years focusing on AI risk strategy while Nick wrote a monograph on the subject, and (2) shortly thereafter, SI began to rapidly grow its research team (at first, mostly through part-time remote researchers) and use that team to produce a lot more research writing than before (only a small fraction of which you’ve seen thus far).
And no, I don’t know in advance what strategic recommendations FHI will arrive at, nor which strategic recommendations SI’s scholarly AI risk wiki will arrive at, except to say that SI’s proposals will probably include Friendly AI research as one of the very important things humanity should be doing right now about AI risk.
ETA: My answer to your original question — “Why haven’t SI and LW attracted or produced any good strategists?” — is that it’s very difficult and time-consuming to acquire all the domain knowledge required to be good at singularity strategy, especially when things like a book-length treatment of AI risk and an AI risk wiki don’t yet exist. It will be easier for someone to become good at singularity strategy once those things exist, but even still they’ll have to know a lot about technological development and forecasting, narrow AI, AGI, FAI open problems, computer science, maths, the physical sciences, economics, philosophy of science, value theory, anthropics, and quite a bit more.
What methodology will be used to produce SI’s strategic recommendations (and FHI’s, if you know the answer)? As far as I can tell, we currently don’t have a way to make the many known strategic considerations/arguments commensurable (e.g., suitable for integrating into a quantitative strategic framework) except by using our intuitions which seem especially unreliable on matters related to Singularity strategy. The fact that you think the AI risk wiki can be finished in 2 years seems to indicate that you either disagree with this evaluation of the current state of affairs, or think we can make very rapid progress in strategic reasoning. Can you explain?
We certainly could integrate known strategic arguments into a quantitative framework like this, but I’m worried that, for example, “putting so many made-up probabilities into a probability tree like this is not actually that helpful.”
I think for now both SI and FHI are still in the qualitative stage that normally precedes quantitative analysis. Big projects like Nick’s monograph and SI’s AI risk wiki will indeed constitute “rapid progress” in strategic reasoning, but it will be rapid progress toward more quantitative analyses, not rapid progress within a quantitative framework that we have already built.
Of course, some of the work on strategic sub-problems is already at the quantitative/formal stage, so quantitative/formal progress can be made on them immediately if SI/FHI can raise the resources to find and hire the right people to work on them. Two examples: (1) What do reasonable economic models of past jumps in optimization power imply about what would happen once we get self-improving AGI? (2) If we add lots more AI-related performance curve data to Nagy’s Performance Curve Database and use his improved tech forecasting methods, what does it all imply about AI and WBE timelines?
I think for now both SI and FHI are still in the qualitative stage that normally precedes quantitative analysis.
There are many strategic considerations that greatly differ in nature from one another. It seems to me that at best they will require diverse novel methods to analyze quantitatively, and at worst a large fraction may resist attempts at quantitative analysis until the Singularity occurs.
For example we can see that there is an upper bound on how confident a small FAI team, working in secret and with limited time, can be (assuming it’s rational) about the correctness of an FAI design, due to the issue raised in my comment quoted by Holden, and this is of obvious strategic importance. But I have no idea what method we can use to derive this bound, other than to “make it up”. Solving this problem alone could easily take a team several years to accomplish, so how do you hope to produce the strategic recommendations, which must take into account many such issues, in 2 years?
Solving this problem alone could easily take a team several years to accomplish, so how do you hope to produce the strategic recommendations, which must take into account many such issues, in 2 years?
Two answers:
Obviously, our recommendations won’t be final, and we’ll try to avoid being overconfident — especially where the recommendations depend on highly uncertain variables.
In many (most?) cases, I suspect our recommendations will be for policies that play a dual role of (1) making progress in directions that look promising from where we stand now, and also (2) purchasing highly valuable information, like how feasible an NGO FAI team is, how hard FAI really is, what the failure modes look like, how plausible alternative approaches are, etc.
SI, FHI, you, others — we’re working on tough problems with many unknown and uncertain strategic variables. Those challenges are not unique to AI risk. Humans have manytools for doing the best they can while running on spaghetti code and facing decision problems under uncertainty, and we’re gaining new tools all the time.
I don’t mean to minimize your concerns, though. Right now I expect to fail. I expect us all to get paperclipped (or turned off), though I’ll be happy to update in favor of positive outcomes if (1) research shows the problem isn’t as hard as I now think, (2) financial support for x-risk reduction increases, (3) etc.
I don’t mean to minimize your concerns, though. Right now I expect to fail. I expect us all to get paperclipped (or turned off), though I’ll be happy to update in favor of positive outcomes if (1) research shows the problem isn’t as hard as I now think, (2) financial support for x-risk reduction increases, (3) etc.
I think you may have misunderstood my intent here. I’m not trying to make you more pessimistic about our overall prospects but arguing (i.e., trying to figure out) the absolute and relative importance of solving various strategic problems.
Another point was to suggest that perhaps SI ought to give higher priority to recruiting/training “hero strategists” as opposed to “hero mathematicians”. For example your So You Want to Save the World says:
No, the world must be saved by mathematicians, computer scientists, and philosophers.
which fails to credit the importance of strategic contributions (even though later in the post there is a large section on strategic problems).
which fails to credit the importance of strategic contributions
Sorry if that was unclear; I mean to identify the strategists as “philosophers”, like this. As you say, I went on to include a large section on strategy.
I certainly agree on the importance of strategy. Most of the research SI and FHI have done is strategic, after all — and most of the work in progress is strategic, too.
I do tend to talk a lot about “hero mathematicians,” though. Maybe that’s because “hero mathematician” is more concrete (to me) than “hero strategist.”
Anyway, it seems like we may be failing to disagree on anything, here.
Sorry if that was unclear; I mean to identify the strategists as “philosophers”, like this.
I see. I had interpreted you to mean philosophers as part of a team to build FAI.
I do tend to talk a lot about “hero mathematicians,” though. Maybe that’s because “hero mathematician” is more concrete (to me) than “hero strategist.”
What do you mean by “more concrete”, and do you think it’s a good reason to talk a lot more about “hero mathematicians”?
I had interpreted you to mean philosophers as part of a team to build FAI.
That could also be true, but I’m not sure.
Re: “hero mathematicians” and “hero strategists”, here’s a more detailed version of what I currently think.
Result of saying we need “hero mathematicians”? A few mathematicians (perhaps primed by HPMoR to be rationality heroes) come to us and learn what the technical research program looks like, help put our memes into the math community, etc.
Result of saying we need “hero strategists”? I’m inundated with people who say they can contribute to singularity strategy after thinking about the issues for one month and reading less than 100 pages on the subject. SI staff wastes valuable time trying to steer amateur strategists along more valuable paths before giving up due to low ROI.
Basically, the recruiting problem is different for mathematicians and strategists, and I think these problems can be tackled more effectively by tackling them separately. Mathematicians can prove themselves useful rather quickly, by offering constructive comments on the problems we will (in the next 12 months) have written up somewhat formally, or by spreading our memes in their research communities.
But to tell whether someone can be a useful strategist they need to read 500 pages of material and spend months chatting regularly with SI and/or FHI, and that’s very costly for both them and for SI+FHI.
The best result might be if some of the mathematicians themselves turn out to be good strategists. I don’t know that I can count on that, but for example I already count both you and Paul Christiano as among the few strategists whose strategy work I would spend my time reading, even though your primary life work has been in math and compsci (and not, say, civil engineering, business management, political science, or economics).
Result of saying we need “hero strategists”? I’m inundated with people who say they can contribute to singularity strategy after thinking about the issues for one month and reading less than 100 pages on the subject. SI staff wastes valuable time trying to steer amateur strategists along more valuable paths before giving up due to low ROI.
But to tell whether someone can be a useful strategist they need to read 500 pages of material and spend months chatting regularly with SI and/or FHI, and that’s very costly for both them and for SI+FHI.
You could direct them to LW and let them prove their mettle here?
I just tried to picture what “hero strategist” could mean, if distinct from ‘person who knows LW rationality’ or ‘practical guy like Luke’. I came up with someone who could hire the world’s best mathematicians plus a professional cat-herder and base the strategy on the result.
Right now I expect to fail. I expect us all to get paperclipped
So, you’re currently thinking hard about the best way to approach someone like Terence Tao? (Doesn’t have to be him, someone else’s blog might also have comments and give you a better opportunity to raise the issue.)
I love working on problems like these. If the specifics of the “circles” are written down anywhere, or if you care to describe them, I’d happily give it a whack. I won’t claim to be an expert, but I enjoy complex problem solving tasks like this one too much not to offer.
My understanding is that there is a lot of writings produced internally by SingInst that are not available to the public. If you think you are up to the task of organizing and polishing them into publishable form (which I guess probably also requires filling in lots of missing pieces) you should contact them and volunteer. (But I guess they probably don’t want to hand their unfinished work to just anyone so you’ll have to prove yourself somehow.)
If you just want to get an idea of the issues involved, here are some of my writings on the topic:
From time to time people ask lukeprog about SI writings. He talk about AI papers they are working, and at some point, for the sake of security, he stop and says “it’s confidential” or something similar.
Evaluating who can do a good work is importante, 60% fail, besides the historical aversion to formated papers.
Why haven’t SI and LW attracted or produced any good strategists? I’ve been given to understand (from someone close to SI) that various people within SI have worked on Singularity strategy but only produced lots of writings that are not of an organized, publishable form. Others have attempted to organize them but also failed, and there seems to be a general feeling that strategy work is bogged down or going in circles and any further effort will not be very productive. The situation on LW seems similar, with people arguing in various directions without much feeling of progress. Why are we so bad at this, given that strategic thinking must be a core part of rationality?
There are some but not lots of “writings” produced internally by SingInst that are not available to the public. There’s lots of scribbled notes and half-finished models and thoughts in brains and stuff like that. We’re working to push them out into written form, but that takes time, money, and people — and we’re short on all three.
The other problem is that to talk about strategy we first have to explain lots of things that are basic (to a veteran like you but not to most interested parties) in clear, well-organized language for the first time, since much of this hasn’t been done yet (these SI papers definitely help, though: 1, 2, 3, 4, 5, 6). To solve this problem we are (1) adding/improving lots of articles on the LW wiki like you suggested a while back (you’ll see a report on what we did, later), and (2) working on the AI risk wiki (we’re creating the map of articles right now). Once those resources are available it will be easier to speak clearly in public about strategic issues.
We hit a temporary delay in pushing out strategy stuff at SI because two of our most knowledgable researchers & strategists became unavailable for different reasons: Anna took over launching CFAR and Carl took an extended (unpaid) leave of absence to take care of some non-SI things. Also, I haven’t been able to continue my own AI risk strategy series due to other priorities, and because I got to the point where it was going to be a lot of work to continue that sequence if I didn’t already have clear, well-organized write-ups of lots of standard material. (So, it’ll be easier for me to continue once the LW wiki has been improved and once the AI risk wiki exists, both of which we’ve got people working on right now.)
Moreover, there are several papers in the works — mostly by Kaj (who is now a staff researcher), with some help from myself — but you won’t see them for a while. You did see this and this, however. Those are the product of months of part-time work from several remote researchers, plus analysis by Kaj and Stuart. Remote researchers are currently doing large lit reviews for other paper projects (at the pace of part-time work), but we aren’t at the stage of analyzing those large data sets yet so that we can write papers about what we found.
Also, a lot of work currently scattered around in papers and posts by SI and FHI people is being collected and tightly organized in Nick Bostrom’s forthcoming monograph on superintelligence, which will be a few hundred pages entirely about singularity strategy. Having much of “the basics” organized in that way will also make it easier to produce additional strategy work.
Luke, with the existing people at SI and FHI’s disposal, how long do you think it would take (assuming they’re not busy with other projects) to produce a document that lays out a cogent argument for some specific Singularity strategy? An argument that takes into account all of the important considerations that have already been raised (for example my comment that Holden quoted)? I will concede that strategy work is not bogged down if you think it can be done in a reasonable time frame. (2 years, perhaps?) But if SI and FHI are merely producing writings that explain the strategic considerations, but which we can’t foresee forming into an overall argument for some specific strategy, that seems very weak evidence at best against my claim that we are bad at strategic thinking.
I know that FHI plans to produce a particular set of policy recommendations relevant to superintelligence upon the release of Nick’s book or shortly thereafter. FHI has given no timeline for Nick’s book but I expect it to be published in mid or late 2013.
The comparably detailed document from SI will be the AI risk wiki. We think the wiki format makes even more sense than a book for these purposes, though an OUP book on superintelligence from Nick Bostrom sounds great to us. Certainly, we will be busy with other projects, but even still I think the AI risk wiki (a fairly comprehensive version 1.0, anyway) could be finished within 2 years. I’m not that confident it will be finished in 2 years, though, given that we’ve barely begun. Six months from now I’ll be more confidently able to predict the likelihood of finishing the AI risk wiki version 1.0 within 2 years.
Despite this, I would describe the current situation as “bogged down” when it comes to singularity strategy. Luckily, the situation is changing due to 2 recent game-shifting events: (1) FHI decided to spend a few years focusing on AI risk strategy while Nick wrote a monograph on the subject, and (2) shortly thereafter, SI began to rapidly grow its research team (at first, mostly through part-time remote researchers) and use that team to produce a lot more research writing than before (only a small fraction of which you’ve seen thus far).
And no, I don’t know in advance what strategic recommendations FHI will arrive at, nor which strategic recommendations SI’s scholarly AI risk wiki will arrive at, except to say that SI’s proposals will probably include Friendly AI research as one of the very important things humanity should be doing right now about AI risk.
ETA: My answer to your original question — “Why haven’t SI and LW attracted or produced any good strategists?” — is that it’s very difficult and time-consuming to acquire all the domain knowledge required to be good at singularity strategy, especially when things like a book-length treatment of AI risk and an AI risk wiki don’t yet exist. It will be easier for someone to become good at singularity strategy once those things exist, but even still they’ll have to know a lot about technological development and forecasting, narrow AI, AGI, FAI open problems, computer science, maths, the physical sciences, economics, philosophy of science, value theory, anthropics, and quite a bit more.
What methodology will be used to produce SI’s strategic recommendations (and FHI’s, if you know the answer)? As far as I can tell, we currently don’t have a way to make the many known strategic considerations/arguments commensurable (e.g., suitable for integrating into a quantitative strategic framework) except by using our intuitions which seem especially unreliable on matters related to Singularity strategy. The fact that you think the AI risk wiki can be finished in 2 years seems to indicate that you either disagree with this evaluation of the current state of affairs, or think we can make very rapid progress in strategic reasoning. Can you explain?
We certainly could integrate known strategic arguments into a quantitative framework like this, but I’m worried that, for example, “putting so many made-up probabilities into a probability tree like this is not actually that helpful.”
I think for now both SI and FHI are still in the qualitative stage that normally precedes quantitative analysis. Big projects like Nick’s monograph and SI’s AI risk wiki will indeed constitute “rapid progress” in strategic reasoning, but it will be rapid progress toward more quantitative analyses, not rapid progress within a quantitative framework that we have already built.
Of course, some of the work on strategic sub-problems is already at the quantitative/formal stage, so quantitative/formal progress can be made on them immediately if SI/FHI can raise the resources to find and hire the right people to work on them. Two examples: (1) What do reasonable economic models of past jumps in optimization power imply about what would happen once we get self-improving AGI? (2) If we add lots more AI-related performance curve data to Nagy’s Performance Curve Database and use his improved tech forecasting methods, what does it all imply about AI and WBE timelines?
There are many strategic considerations that greatly differ in nature from one another. It seems to me that at best they will require diverse novel methods to analyze quantitatively, and at worst a large fraction may resist attempts at quantitative analysis until the Singularity occurs.
For example we can see that there is an upper bound on how confident a small FAI team, working in secret and with limited time, can be (assuming it’s rational) about the correctness of an FAI design, due to the issue raised in my comment quoted by Holden, and this is of obvious strategic importance. But I have no idea what method we can use to derive this bound, other than to “make it up”. Solving this problem alone could easily take a team several years to accomplish, so how do you hope to produce the strategic recommendations, which must take into account many such issues, in 2 years?
Two answers:
Obviously, our recommendations won’t be final, and we’ll try to avoid being overconfident — especially where the recommendations depend on highly uncertain variables.
In many (most?) cases, I suspect our recommendations will be for policies that play a dual role of (1) making progress in directions that look promising from where we stand now, and also (2) purchasing highly valuable information, like how feasible an NGO FAI team is, how hard FAI really is, what the failure modes look like, how plausible alternative approaches are, etc.
SI, FHI, you, others — we’re working on tough problems with many unknown and uncertain strategic variables. Those challenges are not unique to AI risk. Humans have many tools for doing the best they can while running on spaghetti code and facing decision problems under uncertainty, and we’re gaining new tools all the time.
I don’t mean to minimize your concerns, though. Right now I expect to fail. I expect us all to get paperclipped (or turned off), though I’ll be happy to update in favor of positive outcomes if (1) research shows the problem isn’t as hard as I now think, (2) financial support for x-risk reduction increases, (3) etc.
I think you may have misunderstood my intent here. I’m not trying to make you more pessimistic about our overall prospects but arguing (i.e., trying to figure out) the absolute and relative importance of solving various strategic problems.
Another point was to suggest that perhaps SI ought to give higher priority to recruiting/training “hero strategists” as opposed to “hero mathematicians”. For example your So You Want to Save the World says:
which fails to credit the importance of strategic contributions (even though later in the post there is a large section on strategic problems).
Sorry if that was unclear; I mean to identify the strategists as “philosophers”, like this. As you say, I went on to include a large section on strategy.
I certainly agree on the importance of strategy. Most of the research SI and FHI have done is strategic, after all — and most of the work in progress is strategic, too.
I do tend to talk a lot about “hero mathematicians,” though. Maybe that’s because “hero mathematician” is more concrete (to me) than “hero strategist.”
Anyway, it seems like we may be failing to disagree on anything, here.
I see. I had interpreted you to mean philosophers as part of a team to build FAI.
What do you mean by “more concrete”, and do you think it’s a good reason to talk a lot more about “hero mathematicians”?
That could also be true, but I’m not sure.
Re: “hero mathematicians” and “hero strategists”, here’s a more detailed version of what I currently think.
Result of saying we need “hero mathematicians”? A few mathematicians (perhaps primed by HPMoR to be rationality heroes) come to us and learn what the technical research program looks like, help put our memes into the math community, etc.
Result of saying we need “hero strategists”? I’m inundated with people who say they can contribute to singularity strategy after thinking about the issues for one month and reading less than 100 pages on the subject. SI staff wastes valuable time trying to steer amateur strategists along more valuable paths before giving up due to low ROI.
Basically, the recruiting problem is different for mathematicians and strategists, and I think these problems can be tackled more effectively by tackling them separately. Mathematicians can prove themselves useful rather quickly, by offering constructive comments on the problems we will (in the next 12 months) have written up somewhat formally, or by spreading our memes in their research communities.
But to tell whether someone can be a useful strategist they need to read 500 pages of material and spend months chatting regularly with SI and/or FHI, and that’s very costly for both them and for SI+FHI.
The best result might be if some of the mathematicians themselves turn out to be good strategists. I don’t know that I can count on that, but for example I already count both you and Paul Christiano as among the few strategists whose strategy work I would spend my time reading, even though your primary life work has been in math and compsci (and not, say, civil engineering, business management, political science, or economics).
You could direct them to LW and let them prove their mettle here?
I just tried to picture what “hero strategist” could mean, if distinct from ‘person who knows LW rationality’ or ‘practical guy like Luke’. I came up with someone who could hire the world’s best mathematicians plus a professional cat-herder and base the strategy on the result.
So, you’re currently thinking hard about the best way to approach someone like Terence Tao? (Doesn’t have to be him, someone else’s blog might also have comments and give you a better opportunity to raise the issue.)
Actually, yes. We had a meeting about that a couple weeks ago. Tao was specifically named. :)
I love working on problems like these. If the specifics of the “circles” are written down anywhere, or if you care to describe them, I’d happily give it a whack. I won’t claim to be an expert, but I enjoy complex problem solving tasks like this one too much not to offer.
My understanding is that there is a lot of writings produced internally by SingInst that are not available to the public. If you think you are up to the task of organizing and polishing them into publishable form (which I guess probably also requires filling in lots of missing pieces) you should contact them and volunteer. (But I guess they probably don’t want to hand their unfinished work to just anyone so you’ll have to prove yourself somehow.)
If you just want to get an idea of the issues involved, here are some of my writings on the topic:
http://lesswrong.com/r/discussion/lw/6mi/some_thoughts_on_singularity_strategies/
http://lesswrong.com/r/discussion/lw/b10/modest_superintelligences/
http://lesswrong.com/r/discussion/lw/e2k/cynical_explanations_of_fai_critics_including/
And see also the unfinished AI risk sequence by lukeprog.
From time to time people ask lukeprog about SI writings. He talk about AI papers they are working, and at some point, for the sake of security, he stop and says “it’s confidential” or something similar.
Evaluating who can do a good work is importante, 60% fail, besides the historical aversion to formated papers.
Note: I’m still waiting EY books.