I’m one of the people who think AI is probably farther away than Eliezer does, and I think I owe a reply.
So let me go point by point here.
Do I think that nothing productive can be done on AI safety now? No; I think the MIRI and OpenAI research programs probably already include work relevant to safety.
Am I personally doing safety now? Nope; I’m doing applications. Mainly because it’s the job I can do. But I assume that’s not very relevant to Eliezer’s point.
Do I think that one ought to feel a sense of urgency about AI right now? Well, that’s a weird way of thinking about it, and I get stuck on the “one ought to feel” part—is it ever true that you should feel a certain way? I have the sense that this is the road to Crazytown.
Eliezer points out that “the timing of the big development is a function of the peak knowledge in the field”—i.e. just because I don’t see any evidence in the literature that we currently have a strong AI in a secret lab in Google or Baidu or something, doesn’t mean it might not already be here. This is technically true, but doesn’t seem probabilistic enough. Absence of evidence is (weak) evidence of absence. “The real state of the art is much more advanced than the public state of the art” is always technically possible, and you can get Pascal’s Mugged that way.
I find that I get negatively surprised about as often as positively surprised by reading new ML papers from top conferences—half the time I’m like “we can do that now?!” and half the time I’m like “I’m surprised we couldn’t do that already!” (For example, image segmentation with occlusions is way worse than I expected it to be.) If the secret research was much more advanced than the revealed research, I don’t think I’d see that distribution of outcomes, unless the secretive researchers were much more strategic than I expect human beings to actually be.
Most of my “AI is far away” intuitions don’t actually come with timelines. Embarrassingly enough, when people have pressed me for timelines, my asspull numbers have turned out to be inconsistent with each other. My asspull number guesses are not very good.
My intuitions come from a few sources:
a.) When I look at AI performance on particular benchmarks where we’ve seen progress, I usually see performance improvement that’s linear in processing power and hence exponential in time. Basically, we’re getting better because we’re throwing more power at the problem (and because the combination of neural nets and GPUs allows us to grow with processing power rather than lag behind it, by parallelizing the problem.) The exceptions seem to be problems in their infancy, like speech recognition and machine translation, which may be improving faster than Moore’s Law, but are so young that it’s hard to tell. This tells me steady progress is happening, more or less one domain at a time, but calls into question whether the algorithmic improvements since backprop have made much difference at all.
b.) For a general intelligence, you’d need generalization ability from one problem to another, and most of what I’ve seen on meta-learning and generalization ability is that it’s very very weak.
c.) I have some roughly cog-sci-influenced intuitions that intelligence is going to require something more “conceptual” or “model-like” than the “pattern recognition” we see in neural nets right now. (Basically, I believe what Joshua Tenenbaum believes.) In other words, I think we’ll need novel theoretical breakthroughs, not just scaling up the algorithms we have today, to get strong AI. Can I prognosticate how long that’ll take? No, not really. But when people make the strong claim of “5-10 years to strong AI, requiring no concepts not in already-published papers as of 2017″, I feel confident that this is not true.
Hi Sarah, not sure why you felt compelled to answer. Nothing in your reply suggests a contrary logical argument to the Fire Alarm; the only thing I can think of is Eliezer vaguely implying a shorter timeline and you vaguely implying a longer (or at least more diffuse) one. I didn’t get the feeling EY implied AGI is possible by scaling current state of art. The argument about peak knowledge was also to explain the Fire Alarm mechanics, rather than imply that top people at Google have “it” already.
As far as your intuitions, I feel similarly about the cogsci stuff (from a lesser base of knowledge) but it should be noted that there’s some idea exchange between the graphical models people like Josh and NN people. Also it’s possible that NNs can be constructed to learn graphical models. (as an aside would be interesting to ask Josh what his distribution is. Josh begat Noah Goodman, Noah beget Andreas Struhmuller who is quite reachable and in the LW network)
I guess I don’t disagree with the “no fire alarm” thing. I have a policy that if it looks like I might be somebody’s villain, I should show up and make myself available to get smited.
Good point re: talking to Andreas, I may do that one of these days.
I want to pursue this slightly. Before recent evidence—which caused me to update in a vague way towards shorter timelines—my uncertainty looked like a near-uniform distribution over the next century with 5% reserved for the rest of time (conditional on us surviving to AGI). This could obviously give less than a 10% probability for the claim “5-10 years to strong AI” and the likely destruction of humanity at that time. Are you really arguing for something lower, or are you “confident” the way people were certain (~80%) Hillary Clinton would win?
I think Eliezer is implying here that timelines may be short or at least that the left tail is fatter than people want to admit, but I think the thing that Sarah feels compelled to respond to is more the vibe that you have no right to think there are long timelines. He’s saying that in order to be confident in no strong AI within a few years you need lots of concrete predictions and probabilities or else you’re just pulling things out of [the air] on request without a model and not updating on evidence, and implying that recent evidence should update you in favor of sooner being more likely rather than AGI getting one day later in expectation each day. In particular, his fifth point in response to the conference.
It felt off-putting enough to me that I decided to respond at length here to the associated analysis and logic, even though I too fully agree with no fire alarm and the need to act now and the fact that most people don’t have models and so on.
I don’t have enough knowledge of current ML to offer short term predictions that are worth anything, which is something I want to try and change, but in the meantime I don’t think that means I can’t make meaningful long term predictions, just that they’ll be worse than they would otherwise be.
My take is that Eliezer is saying that we should be aware of the significant probability that AGI takes us unaware, and also that people don’t tend to think enough about their claims. He’s not saying “be certain that it will be soon,” but rather “any claim that it will almost certainly take centuries is suspect if it cannot be backed up with specific, lower-level difficulty claims expressed through estimated times for certain goals to be reached.” I’m not sure if this goes against your reading of the post, though.
Yeah, I was also confused what disagreement Sarah was pointing to, but I thought maybe she was arguing that there was in fact a fire alarm, as she currently has models of AI development that say it’s very far away without a conceptual breakthrough i.e. that conceptual breakthrough would be a fire alarm.
But this seems false, given that I’ve not heard many others state this fire alarm in particular (with all the details regarding “performance improvement that’s linear in processing power and hence exponential in time” etc). Nonetheless I’d be happy to find out that there sort of is such a consensus.
“Do I think that one ought to feel a sense of urgency about AI right now? Well, that’s a weird way of thinking about it, and I get stuck on the “one ought to feel” part—is it ever true that you should feel a certain way? I have the sense that this is the road to Crazytown.”
If the room is on fire, one ought to feel at least mildly concerned. If the room has a significant chance of being set on fire, one ought to feel somewhat less concerned but still not entirely okay with the prospect of a fiery death. It seems clear that one ought to be worried about future events to a degree proportional to their likelihood and adverse effects, or else face a greater chance of knowing about but ignoring a significant danger.
I’m one of the people who think AI is probably farther away than Eliezer does, and I think I owe a reply.
So let me go point by point here.
Do I think that nothing productive can be done on AI safety now? No; I think the MIRI and OpenAI research programs probably already include work relevant to safety.
Am I personally doing safety now? Nope; I’m doing applications. Mainly because it’s the job I can do. But I assume that’s not very relevant to Eliezer’s point.
Do I think that one ought to feel a sense of urgency about AI right now? Well, that’s a weird way of thinking about it, and I get stuck on the “one ought to feel” part—is it ever true that you should feel a certain way? I have the sense that this is the road to Crazytown.
Eliezer points out that “the timing of the big development is a function of the peak knowledge in the field”—i.e. just because I don’t see any evidence in the literature that we currently have a strong AI in a secret lab in Google or Baidu or something, doesn’t mean it might not already be here. This is technically true, but doesn’t seem probabilistic enough. Absence of evidence is (weak) evidence of absence. “The real state of the art is much more advanced than the public state of the art” is always technically possible, and you can get Pascal’s Mugged that way.
I find that I get negatively surprised about as often as positively surprised by reading new ML papers from top conferences—half the time I’m like “we can do that now?!” and half the time I’m like “I’m surprised we couldn’t do that already!” (For example, image segmentation with occlusions is way worse than I expected it to be.) If the secret research was much more advanced than the revealed research, I don’t think I’d see that distribution of outcomes, unless the secretive researchers were much more strategic than I expect human beings to actually be.
Most of my “AI is far away” intuitions don’t actually come with timelines. Embarrassingly enough, when people have pressed me for timelines, my asspull numbers have turned out to be inconsistent with each other. My asspull number guesses are not very good.
My intuitions come from a few sources:
a.) When I look at AI performance on particular benchmarks where we’ve seen progress, I usually see performance improvement that’s linear in processing power and hence exponential in time. Basically, we’re getting better because we’re throwing more power at the problem (and because the combination of neural nets and GPUs allows us to grow with processing power rather than lag behind it, by parallelizing the problem.) The exceptions seem to be problems in their infancy, like speech recognition and machine translation, which may be improving faster than Moore’s Law, but are so young that it’s hard to tell. This tells me steady progress is happening, more or less one domain at a time, but calls into question whether the algorithmic improvements since backprop have made much difference at all.
b.) For a general intelligence, you’d need generalization ability from one problem to another, and most of what I’ve seen on meta-learning and generalization ability is that it’s very very weak.
c.) I have some roughly cog-sci-influenced intuitions that intelligence is going to require something more “conceptual” or “model-like” than the “pattern recognition” we see in neural nets right now. (Basically, I believe what Joshua Tenenbaum believes.) In other words, I think we’ll need novel theoretical breakthroughs, not just scaling up the algorithms we have today, to get strong AI. Can I prognosticate how long that’ll take? No, not really. But when people make the strong claim of “5-10 years to strong AI, requiring no concepts not in already-published papers as of 2017″, I feel confident that this is not true.
Hi Sarah, not sure why you felt compelled to answer. Nothing in your reply suggests a contrary logical argument to the Fire Alarm; the only thing I can think of is Eliezer vaguely implying a shorter timeline and you vaguely implying a longer (or at least more diffuse) one. I didn’t get the feeling EY implied AGI is possible by scaling current state of art. The argument about peak knowledge was also to explain the Fire Alarm mechanics, rather than imply that top people at Google have “it” already.
As far as your intuitions, I feel similarly about the cogsci stuff (from a lesser base of knowledge) but it should be noted that there’s some idea exchange between the graphical models people like Josh and NN people. Also it’s possible that NNs can be constructed to learn graphical models. (as an aside would be interesting to ask Josh what his distribution is. Josh begat Noah Goodman, Noah beget Andreas Struhmuller who is quite reachable and in the LW network)
I guess I don’t disagree with the “no fire alarm” thing. I have a policy that if it looks like I might be somebody’s villain, I should show up and make myself available to get smited.
Good point re: talking to Andreas, I may do that one of these days.
I want to pursue this slightly. Before recent evidence—which caused me to update in a vague way towards shorter timelines—my uncertainty looked like a near-uniform distribution over the next century with 5% reserved for the rest of time (conditional on us surviving to AGI). This could obviously give less than a 10% probability for the claim “5-10 years to strong AI” and the likely destruction of humanity at that time. Are you really arguing for something lower, or are you “confident” the way people were certain (~80%) Hillary Clinton would win?
I think Eliezer is implying here that timelines may be short or at least that the left tail is fatter than people want to admit, but I think the thing that Sarah feels compelled to respond to is more the vibe that you have no right to think there are long timelines. He’s saying that in order to be confident in no strong AI within a few years you need lots of concrete predictions and probabilities or else you’re just pulling things out of [the air] on request without a model and not updating on evidence, and implying that recent evidence should update you in favor of sooner being more likely rather than AGI getting one day later in expectation each day. In particular, his fifth point in response to the conference.
It felt off-putting enough to me that I decided to respond at length here to the associated analysis and logic, even though I too fully agree with no fire alarm and the need to act now and the fact that most people don’t have models and so on.
I don’t have enough knowledge of current ML to offer short term predictions that are worth anything, which is something I want to try and change, but in the meantime I don’t think that means I can’t make meaningful long term predictions, just that they’ll be worse than they would otherwise be.
My take is that Eliezer is saying that we should be aware of the significant probability that AGI takes us unaware, and also that people don’t tend to think enough about their claims. He’s not saying “be certain that it will be soon,” but rather “any claim that it will almost certainly take centuries is suspect if it cannot be backed up with specific, lower-level difficulty claims expressed through estimated times for certain goals to be reached.” I’m not sure if this goes against your reading of the post, though.
Yeah, I was also confused what disagreement Sarah was pointing to, but I thought maybe she was arguing that there was in fact a fire alarm, as she currently has models of AI development that say it’s very far away without a conceptual breakthrough i.e. that conceptual breakthrough would be a fire alarm.
But this seems false, given that I’ve not heard many others state this fire alarm in particular (with all the details regarding “performance improvement that’s linear in processing power and hence exponential in time” etc). Nonetheless I’d be happy to find out that there sort of is such a consensus.
“Do I think that one ought to feel a sense of urgency about AI right now? Well, that’s a weird way of thinking about it, and I get stuck on the “one ought to feel” part—is it ever true that you should feel a certain way? I have the sense that this is the road to Crazytown.”
If the room is on fire, one ought to feel at least mildly concerned. If the room has a significant chance of being set on fire, one ought to feel somewhat less concerned but still not entirely okay with the prospect of a fiery death. It seems clear that one ought to be worried about future events to a degree proportional to their likelihood and adverse effects, or else face a greater chance of knowing about but ignoring a significant danger.