EDIT: Note, I don’t think these people have given explicit probabilities. But they seem much less worried than people from the AI alignment community. EDIT^2: Also, only the links to Hanson and Jacob’s stuff have comparable detail to what you requested.
Bryan Caplan is one. Tyler Cowen too, if you take his claims of nuclear war being a much greater large-scale risk by far seriously and assign standard numbers for that. I think David Friedman might agree, though I’ll get back to you on that. Geoffery Hinton seems more worried about autonomous machines than AI taking over. He thinks deep learning will be enough, but quite a few more conceptual breakthroughs on the order of transformers will be needed.
Maybe Jacob Cannell? He seems quite optimistic that alignment is on track to be solved. Though I doubt his P(doom) is less than 1%.
Strong disagree. Hanson believes that there’s more than a 1% chance of AI destroying all value.
Even if he didn’t see an inside view argument, he makes an outside view argument about the Great Filter.
He probably believes that there’s a much larger chance of it killing everyone, and his important disagreement with Yudkowsky is that thinks that it will have value in itself, rather than be a paperclip maximizer. In particular, in the Em scenario, he argues that property rights will keep humans alive for 2 years. Maybe you should read that as <1% of all humans being killed in that first phase, but at some point the Ems evolve into something truly alien and he stops predicting that they don’t kill everyone. But that’s OK because he values the descendants.
Also note that iirc he only assigns about 10% to the EM scenario happening in general? At least, as of the writing of the book. I get the impression he just thinks about it a lot because it is the scenario that he, a human economist, can think about.
I have not read the book, but my memory is that in a blog post he said that the probability is “at least” 10%. I think he holds a much higher number, but doesn’t want to speak about it and just wants to insist that his hostile reader should accept at least 10%. In particular, if people say “no it won’t happen, 10%,” then that’s not a rebuttal at all. But maybe I’m confusing that with other numbers, eg, here where he says that it’s worth talking about even if it is only 1%.
Conditional on my key assumptions, I expect at least 30 percent of future situations to be usefully informed by my analysis. Unconditionally, I expect at least 5 percent.
I now estimate an unconditional 80% chance of it being a useful guide,
I think that means he previously put 15% on ems in general and 5% on his em scenario (ie, you were right).
80% on the specific scenario leaves little room for AI, let alone AI destroying all value. So maybe he now puts that <1%. But maybe he has just removed non-em non-AI scenarios. In particular, you have to put a lot of weight on completely unanticipated scenarios; perhaps that has gone from 80% to 10%.
I’d expect his “useful guide” claim to be compatible with worlds that’re entirely AGIs? He seems to think they’ll be subject to the same sorts of dynamics as humans, coordination problems and all that. I’m not convinced, but he seems quite confident.
(personally I think some coordination problems and legibility issues will always persist, but they’d be relatively unimportant, and focusing on them wont tell us much about the overall shape of AGI societies.)
OK, fair. I didn’t actually read the post in detail. There’s a good chance Hanson assigns >1% chance of AI killing everyone, if you don’t include EMs. But two points
1) Hanson’s view of EMs results in vast numbers of very human like minds continuing to exist for a long subjective period of time. That’s not really an x-risk, though Hanson does think it plausible that biological humans may suffer greatly in the transition. He doesn’t give a detailed picture of what happens after, besides some stuff like colonozing the sun etc. Yet, there could still be humans hanging around in the Age of EM. To me, Age of EM paints a picture that makes OP’s question seem kind of poorly phrased. Like, if someone believed solving alignment would result in all humans being uploaded, then gradually becoming transhuman entities, would that qualify as a >1% chance of human extinction? I think most here would say no.
2) Working on capabilities doesn’t seem to be nearly as big an issue in a Hansonian worldview as it would be in e.g. Yudkowsky’s, or even Christiano’s. So I feel like pointing out Hanson would still be worthwhile, especially as he’s a person who engaged heavily with the early AI alignment people.
I claim that Hanson has >1% chance of Yudkowsky’s scenario that AI comes first and destroys all value and also a >1% chance that Ems come first and a scenario that a lot of people would say killed all people, including the Ems. This is not directly relevant to the question about AI, but it suggests that he is sanguine about analogous AI scenarios, soft takeoff scenarios not covered by Yudkowsky.
Yes, during the 2 years of wallclock time, the Ems exist for 1000 subjective years. Is that so long? This is not “longtermism.” Yes, you should probably count the Ems as humans, so if they kill all the biological humans, they don’t “kill everyone,” but after this period they are outcompeted by something more alien. Does this count as killing everyone?
Working on capabilities isn’t a problem in his mainline, but the question was not about mainline, but about tail events. If Ems are going to come first, then you could punt alignment to their millennium of work. But if it’s not guaranteed who comes first and AI is worse than Ems, working on AI could cause it to come first. Or maybe not. Maybe one is so much easier than the other and nothing is decision relevant.
Yes, Hanson sees value drift as inevitable. The Ems will be outcompeted by something better adapted that we should see some value in. He thinks it’s parochial to dislike the Ems evolving under Malthusian pressures. Maybe, but it’s important not to confuse the factual questions with the moral questions. “It’s OK because there’s no risk of X” is different from “X is OK, actually.” Yes, he talks about the Dreamtime. Part of that is the delusion that we can steer the future more than Malthusian forces. But part of it is that because we are not yet under strict competition, we have excess resources that we can use to steer the future, if only a little.
I think this is a good summary of Hanson’s views, and your answer is correct as pertains to the question that was actually asked. That said, I think reading Hanson counts as a skeptic for the need for more AI-safety researchers on the margin. And, I think he’d be skeptical of the marginal person claiming a large impact via working on AI capabilities relative to most counterfactuals. I am not sure if we disagree there, but I’m going to tap out anyway.
I think this is his latest comment, but it is on FOOM. Hanson’s opinion is that, on the margin, the current amount of people working on AI safety seems adequate. Why? Because there’s not much useful work you can do without access to advanced AI, and he thinks the latter is a long time in coming. Again, why? Hanson thinks that FOOM is the main reason to worry about AI risk. He prefers an outside view to predict technologies which we have little empirical information on and so believes FOOM is unlikely because he thinks progress historically doesn’t come in huge chunks but gradually. You might question the speed of progress, if not its lumpiness, as deep learning seems to pump out advance after advance. Hanson argues that people are estimating progress poorly and talk of deep learning is over-blown.
What would it take to get Hanson to sit up and pay more attention to AI? AI self-monologue used to guide and improves its ability to perform useful tasks.
One thing I didn’t manage to fit in here is that I feel like another crux for Hanson would be how the brain works. If the brain tackles most useful tasks using a simple learning algorithm, like Steve Byrnes argues, instead of a grab bag of specialized modules with distinct algorithms for each of them, then I think that would be a big update. But that is mostly my impression, and I can’t find the sources I used to generate it.
I’ve argued at length (1234567) against the plausibility of this scenario. Its not that its impossible, or that no one should work on it, but that far too many take it as a default future scenario.
Yeah, I think he assigns ~5% chance to FOOM, if I had to make a tenative guess. 10% seems too high to me. In general, my first impression as to Hanson’s credences on a topic won’t be accurate unless I really scrutinize his claims. So its not weird to me that someone might wind up thinking Hanson believes there’s a <1% of AI x-risks.
Do you mean hard take off, or Yudkowsky’s worry that foom causes rapid value drift and destroys all value? I think Hanson puts maybe 5% on that and a much larger number on hard take off, 10 or 20%.
Really? My impression was the opposite. He’s said stuff to the effect of “there’s nothing you can do to prevent value drift”, and seems to think that whether we create EMs or not, our successors will hold values quite different to our own. See all the stuff about the current era being a dreamtime, on the values of grabby aliens etc.
Hanson is the most obvious answer, to me.
EDIT: Note, I don’t think these people have given explicit probabilities. But they seem much less worried than people from the AI alignment community.
EDIT^2: Also, only the links to Hanson and Jacob’s stuff have comparable detail to what you requested.
Bryan Caplan is one. Tyler Cowen too, if you take his claims of nuclear war being a much greater large-scale risk by far seriously and assign standard numbers for that. I think David Friedman might agree, though I’ll get back to you on that. Geoffery Hinton seems more worried about autonomous machines than AI taking over. He thinks deep learning will be enough, but quite a few more conceptual breakthroughs on the order of transformers will be needed.
Maybe Jacob Cannell? He seems quite optimistic that alignment is on track to be solved. Though I doubt his P(doom) is less than 1%.
Strong disagree. Hanson believes that there’s more than a 1% chance of AI destroying all value.
Even if he didn’t see an inside view argument, he makes an outside view argument about the Great Filter.
He probably believes that there’s a much larger chance of it killing everyone, and his important disagreement with Yudkowsky is that thinks that it will have value in itself, rather than be a paperclip maximizer. In particular, in the Em scenario, he argues that property rights will keep humans alive for 2 years. Maybe you should read that as <1% of all humans being killed in that first phase, but at some point the Ems evolve into something truly alien and he stops predicting that they don’t kill everyone. But that’s OK because he values the descendants.
Also note that iirc he only assigns about 10% to the EM scenario happening in general? At least, as of the writing of the book. I get the impression he just thinks about it a lot because it is the scenario that he, a human economist, can think about.
I have not read the book, but my memory is that in a blog post he said that the probability is “at least” 10%. I think he holds a much higher number, but doesn’t want to speak about it and just wants to insist that his hostile reader should accept at least 10%. In particular, if people say “no it won’t happen, 10%,” then that’s not a rebuttal at all. But maybe I’m confusing that with other numbers, eg, here where he says that it’s worth talking about even if it is only 1%.
Here he reports old numbers and new:
I think that means he previously put 15% on ems in general and 5% on his em scenario (ie, you were right).
80% on the specific scenario leaves little room for AI, let alone AI destroying all value. So maybe he now puts that <1%. But maybe he has just removed non-em non-AI scenarios. In particular, you have to put a lot of weight on completely unanticipated scenarios; perhaps that has gone from 80% to 10%.
I’d expect his “useful guide” claim to be compatible with worlds that’re entirely AGIs? He seems to think they’ll be subject to the same sorts of dynamics as humans, coordination problems and all that. I’m not convinced, but he seems quite confident.
(personally I think some coordination problems and legibility issues will always persist, but they’d be relatively unimportant, and focusing on them wont tell us much about the overall shape of AGI societies.)
OK, fair. I didn’t actually read the post in detail. There’s a good chance Hanson assigns >1% chance of AI killing everyone, if you don’t include EMs. But two points
1) Hanson’s view of EMs results in vast numbers of very human like minds continuing to exist for a long subjective period of time. That’s not really an x-risk, though Hanson does think it plausible that biological humans may suffer greatly in the transition. He doesn’t give a detailed picture of what happens after, besides some stuff like colonozing the sun etc. Yet, there could still be humans hanging around in the Age of EM. To me, Age of EM paints a picture that makes OP’s question seem kind of poorly phrased. Like, if someone believed solving alignment would result in all humans being uploaded, then gradually becoming transhuman entities, would that qualify as a >1% chance of human extinction? I think most here would say no.
2) Working on capabilities doesn’t seem to be nearly as big an issue in a Hansonian worldview as it would be in e.g. Yudkowsky’s, or even Christiano’s. So I feel like pointing out Hanson would still be worthwhile, especially as he’s a person who engaged heavily with the early AI alignment people.
I claim that Hanson has >1% chance of Yudkowsky’s scenario that AI comes first and destroys all value and also a >1% chance that Ems come first and a scenario that a lot of people would say killed all people, including the Ems. This is not directly relevant to the question about AI, but it suggests that he is sanguine about analogous AI scenarios, soft takeoff scenarios not covered by Yudkowsky.
Yes, during the 2 years of wallclock time, the Ems exist for 1000 subjective years. Is that so long? This is not “longtermism.” Yes, you should probably count the Ems as humans, so if they kill all the biological humans, they don’t “kill everyone,” but after this period they are outcompeted by something more alien. Does this count as killing everyone?
Working on capabilities isn’t a problem in his mainline, but the question was not about mainline, but about tail events. If Ems are going to come first, then you could punt alignment to their millennium of work. But if it’s not guaranteed who comes first and AI is worse than Ems, working on AI could cause it to come first. Or maybe not. Maybe one is so much easier than the other and nothing is decision relevant.
Yes, Hanson sees value drift as inevitable. The Ems will be outcompeted by something better adapted that we should see some value in. He thinks it’s parochial to dislike the Ems evolving under Malthusian pressures. Maybe, but it’s important not to confuse the factual questions with the moral questions. “It’s OK because there’s no risk of X” is different from “X is OK, actually.” Yes, he talks about the Dreamtime. Part of that is the delusion that we can steer the future more than Malthusian forces. But part of it is that because we are not yet under strict competition, we have excess resources that we can use to steer the future, if only a little.
I think this is a good summary of Hanson’s views, and your answer is correct as pertains to the question that was actually asked. That said, I think reading Hanson counts as a skeptic for the need for more AI-safety researchers on the margin. And, I think he’d be skeptical of the marginal person claiming a large impact via working on AI capabilities relative to most counterfactuals. I am not sure if we disagree there, but I’m going to tap out anyway.
Here, for instance.
I think this is his latest comment, but it is on FOOM. Hanson’s opinion is that, on the margin, the current amount of people working on AI safety seems adequate. Why? Because there’s not much useful work you can do without access to advanced AI, and he thinks the latter is a long time in coming. Again, why? Hanson thinks that FOOM is the main reason to worry about AI risk. He prefers an outside view to predict technologies which we have little empirical information on and so believes FOOM is unlikely because he thinks progress historically doesn’t come in huge chunks but gradually. You might question the speed of progress, if not its lumpiness, as deep learning seems to pump out advance after advance. Hanson argues that people are estimating progress poorly and talk of deep learning is over-blown.
What would it take to get Hanson to sit up and pay more attention to AI? AI self-monologue used to guide and improves its ability to perform useful tasks.
One thing I didn’t manage to fit in here is that I feel like another crux for Hanson would be how the brain works. If the brain tackles most useful tasks using a simple learning algorithm, like Steve Byrnes argues, instead of a grab bag of specialized modules with distinct algorithms for each of them, then I think that would be a big update. But that is mostly my impression, and I can’t find the sources I used to generate it.
That sounds like a lot more than 1% chance.
Yeah, I think he assigns ~5% chance to FOOM, if I had to make a tenative guess. 10% seems too high to me. In general, my first impression as to Hanson’s credences on a topic won’t be accurate unless I really scrutinize his claims. So its not weird to me that someone might wind up thinking Hanson believes there’s a <1% of AI x-risks.
Do you mean hard take off, or Yudkowsky’s worry that foom causes rapid value drift and destroys all value? I think Hanson puts maybe 5% on that and a much larger number on hard take off, 10 or 20%.
Really? My impression was the opposite. He’s said stuff to the effect of “there’s nothing you can do to prevent value drift”, and seems to think that whether we create EMs or not, our successors will hold values quite different to our own. See all the stuff about the current era being a dreamtime, on the values of grabby aliens etc.