We’re not. There’s a spread of perspectives and opinions and lack-of-opinions. If you’re judging from the upvotes, might be worth keeping in mind that some of us think “upvote” should mean “this seems like it helps the conversation access relevant considerations/arguments” rather than “I agree with the conclusions.”
Still, my shortest reply to “Why expect there’s at least some risk if an AI is created that’s way more powerful than humanity?” is something like: “It seems pretty common-sensical to think that alien entities that’re much more powerful than us may pose risk to us. Plus, we thought about it longer and that didn’t make the common sense of that ‘sounds risky’ go away, for many/most of us.”
The spread of opinions seems narrow compared to what I would expect. OP makes some bold predictions in his post. I see more debate over less controversial claims all of the time.
The spread of opinions seems narrow compared to what I would expect. OP makes some bold predictions in his post. I see more debate over less controversial claims all of the time.
That’s fair.
what do aliens have to do with AI?
Sorry, I said it badly/unclearly. What I meant was: most ways to design powerful AI will, on my best guess, be “alien” intelligences, in the sense that they are different from us (think differently, have different goals/values, etc.).
There’s an analogy being drawn between the power of a hypothetical advanced alien civilization and the power of a superintelligent AI. If you agree that the hypothetical AI would be more powerful, and that an alien civilization capable of travelling to Earth would be a threat, then it follows that superintelligent AI is a threat.
I think most people here are in agreement that AI poses a huge risk, but are differ on how likely it is that we’re all going to die. A 20% chance we’re all going to die is very much worth trying to mitigate sensibly, and the OP says still it’s worth trying to mitigate a 99.9999% chance of human extinction in a similarly level-headed manner (even if the mechanics of doing the work are slightly different at that point).
“There’s no hope for survival” is an overstatement; the OP is arguing “successfully navigating AGI looks very hard, enough that we should reconcile ourselves with the reality that we’re probably not going to make it”, not “successfully navigating AGI looks impossible / negligibly likely, such that we should give up”.
Yes! Verymuch+1! I’ve been hanging around here for almost 10 years and am getting value from the response to the 101-level questions.
Superintelligence, “AI Alignment: Why It’s Hard, and Where to Start”, “There’s No Fire Alarm for Artificial General Intelligence”, and the “Security Mindset” dialogues (part one, part two) do a good job of explaining why people are super worried about AGI.
Honestly, the Wait But Why posts are my favorite and what I would recommend to a newcomer. (I was careful to say things like “favorite” and “what I’d recommend to a newcomer” instead of “best”.)
Furthermore, I feel like Karolina and people like them who are asking this sort of question deserve an answer that doesn’t require an investment of multiple hours of effortful reading and thinking. I’m thinking something like three paragraphs. Here is my attempt at one. Take it with the appropriate grain of salt. I’m just someone who’s hung around the community for a while but doesn’t deal with this stuff professionally.
Think about how much smarter humans are than dogs. Our intelligence basically gives us full control over them. We’re technologically superior. Plus we understand their psychology and can manipulate them into doing what we want pretty reliably. Now take that and multiply by, uh, a really big number. An AGI would be wayyyyy smarter than us.
Ok, but why do we assume there will be this super powerful AGI? Well, first of all, if you look at surveys, it looks like pretty much everyone does in fact believe it. It’s not something that people disagree on. They just disagree on when exactly it will happen. 10 years? 50 years? 200 years? But to address the question of how, well, the idea is that the smarter a thing is, the more capable it is of improving it’s own smartness. So suppose something starts off with 10 intelligence points and tries to improve and goes to 11. At 11 it’s more capable of improving than it was at 10, so it goes to a 13. +2 instead of +1. At 13 it’s even more capable of improving, so it goes to a 16. Then a 20. Then a 25. So on and so forth. Accelerating growth is powerful.
Ok, so suppose we have this super powerful thing that is inevitable and is going to have complete control over us and the universe. Why can’t we just program the thing from the start to be nice? This is the alignment problem. I don’t have the best understanding of it tbh, but I like to think about the field of law and contracts. Y’know how those terms of service documents are so long and no one reads them? It’s hard to specify all of the things you want upfront. “Oh yeah, please don’t do A. Oh wait I forgot B, don’t do that either. Oh and C, yeah, that’s another important one.” Even I understand that this is a pretty bad description of the alignment problem, but hopefully it at least serves the purpose of providing some amount of intuition.
“There’s no hope for survival” is an overstatement; the OP is arguing “successfully navigating AGI looks very hard, enough that we should reconcile ourselves with the reality that we’re probably not going to make it”, not “successfully navigating AGI looks impossible / negligibly likely, such that we should give up”.
Really? That wasn’t my read on it. Although I’m not very confident in my read on it. I came away from the post feeling confused.
For example, 0% was mentioned in a few places, like:
When Earth’s prospects are that far underwater in the basement of the logistic success curve, it may be hard to feel motivated about continuing to fight, since doubling our chances of survival will only take them from 0% to 0%.
Was that April Fools context exaggeration? My understanding is that what Vaniver said here is accurate: that Eliezer is saying that we’re doomed, but that he’s saying it on April Fools so that people who don’t want to believe it for mental health reasons (nervously starts raising hand) have an out. Maybe that’s not accurate though? Maybe it’s more like “We’re pretty doomed but I’m exaggerating/joking about how doomed we are because it’s April Fools Day.”
I think the Wait But Why posts are quite good, though I usually link them alongside Luke Muehlhauser’s reply.
For example, 0% was mentioned in a few places
It’s obviously not literally 0%, and the post is explicitly about ‘how do we succeed?’, with a lengthy discussion of the possible worlds where we do in fact succeed:
[...] The surviving worlds look like people who lived inside their awful reality and tried to shape up their impossible chances; until somehow, somewhere, a miracle appeared—the model broke in a positive direction, for once, as does not usually occur when you are trying to do something very difficult and hard to understand, but might still be so—and they were positioned with the resources and the sanity to take advantage of that positive miracle, because they went on living inside uncomfortable reality. Positive model violations do ever happen, but it’s much less likely that somebody’s specific desired miracle that “we’re all dead anyways if not...” will happen; these people have just walked out of the reality where any actual positive miracles might occur. [...]
The whole idea of ‘let’s maximize dignity’ is that it’s just a reframe of ‘let’s maximize the probability that we survive and produce a flourishing civilization’ (the goal of the reframe being to guard against wishful thinking):
Obviously, the measuring units of dignity are over humanity’s log odds of survival—the graph on which the logistic success curve is a straight line. [...] But if enough people can contribute enough bits of dignity like that, wouldn’t that mean we didn’t die at all? Yes, but again, don’t get your hopes up.
Hence:
Q1: Does ‘dying with dignity’ in this context mean accepting the certainty of your death, and not childishly regretting that or trying to fight a hopeless battle?
Don’t be ridiculous. How would that increase the log odds of Earth’s survival?
I dunno what the ‘0%’ means exactly, but it’s obviously not literal. My read of it was something like ‘low enough that it’s hard enough to be calibrated about exactly how low it is’, plus ‘low enough that you can make a lot of progress and still not have double-digit success probability’.
I think the Wait But Why posts are quite good, though I usually link them alongside Luke Muehlhauser’s reply.
Cool! Good to get your endorsement. And thanks for pointing me to Muehlhauser’s reply. I’ll check it out.
I dunno what the ‘0%’ means exactly, but it’s obviously not literal. My read of it was something like ‘low enough that it’s hard enough to be calibrated about exactly how low it is’, plus ‘low enough that you can make a lot of progress and still not have double-digit success probability’.
Ok, yeah, that does sound pretty plausible. It still encompasses a pretty wide range though. Like, it could mean one in a billion, I guess? There is a grim tone here. And Eliezer has spoken about his pessimism elsewhere with a similarly grim tone. Maybe I am overreacting to that. I dunno. Like Raemon, I am still feeling confused.
It’s definitely worth noting though that even at a low number like one in a billion, it is still worth working on for sure. And I see that Eliezer believes this as well. So in that sense I take back what I said in my initial comment.
One in a billion seems way, way, way too low to me. (Like, I think that’s a crazy p(win) to have, and I’d be shocked beyond shocked if Eliezer’s p(win) were that low. Like, if he told me that I don’t think I’d believe him.)
FYI I am finding myself fairly confused about the “0%” line. I don’t see a reason not to take Eliezer at his word that he meant 0%. “Obviously not literal” feels pretty strong, if he meant a different thing I’d prefer the post say whatever he meant.
Eliezer seemed quite clear to me when he said (paraphrased) “we are on the left side of the logistical success curve, where success is measured in significant digits of leading 0s you are removing from your probability of success”. The whole post seems to clearly imply that Eliezer thinks that marginal dignity is possible, which he defines as a unit of logistical movement on the probability of success. This clearly implies the probability is not literally 0, but it does clearly argue that the probability (on a linear scale) can be rounded to 0.
Mostly I had no idea if he meant like 0.1, 0.001, or 0.00000001. Also not sure if he’s more like “survival chance is 0%, probably, with some margin of error, maybe it’s 1%”, or “no, I’m making the confident claim that it’s more like 0.0001″
(This was combined with some confusion about Nate Soares saying something in the vein of “if you don’t whole heartedly believe in your plans, you should multiply their EV by 0, and you’re not supposed to pick the plan who’s epsilon value is “least epsilon”)
Also, MIRI isn’t (necessarily) a hive mind, so not sure if Rob, Nate or Abram actually share the same estimate of how doomed we are as Eliezer.
Also, MIRI isn’t (necessarily) a hive mind, so not sure if Rob, Nate or Abram actually share the same estimate of how doomed we are as Eliezer.
Indeed, I expect that the views of at least some individuals working at MIRI vary considerably.
In some ways, the post would seem more accurate to me if it had the Onion-esque headline: Eliezer announces on MIRI’s behalf that “MIRI adopts new ‘Death with Dignity’ strategy.”
Still, I love the post a lot. Also, Eliezer has always been pivotal in MIRI.
This sequence covers some chunk of it, though it does already assume a lot of context. I think this sequence is the basic case for AI Risk, and doesn’t assume a lot of context.
People should be able to think the world is at risk, or even doomed, and that solving that is (naturally) something like the most pressing problem, without being whatever it means to be “apocalyptic death cultists”. Otherwise you run into foreseeable issues like new technologies actually presenting existential risks and nobody taking taking those risks seriously for social reasons.
Lmao, I’m the author of the top level comment, and I honestly came to the same conclusion about this place being a cult (or at the very least, cult-like with a TON of red flags). I recently started lurking here again after not thinking about this group for months because I was curious how the rationalists were rationalizing the FTX collapse and also wanted to see their reaction to that openai chat thing.
So AI will destroy the planet and there’s no hope for survival?
Why is everyone here in agreement that AI will inevitably kill off humanity and destroy the planet?
Sorry I’m new to LessWrong and clicked on this post because I recognized the author’s name from the series on rationality.
We’re not. There’s a spread of perspectives and opinions and lack-of-opinions. If you’re judging from the upvotes, might be worth keeping in mind that some of us think “upvote” should mean “this seems like it helps the conversation access relevant considerations/arguments” rather than “I agree with the conclusions.”
Still, my shortest reply to “Why expect there’s at least some risk if an AI is created that’s way more powerful than humanity?” is something like: “It seems pretty common-sensical to think that alien entities that’re much more powerful than us may pose risk to us. Plus, we thought about it longer and that didn’t make the common sense of that ‘sounds risky’ go away, for many/most of us.”
The spread of opinions seems narrow compared to what I would expect. OP makes some bold predictions in his post. I see more debate over less controversial claims all of the time.
Sorry, but what do aliens have to do with AI?
Part of the reason the spread seems small is that people are correctly inferring that this comment section is not a venue for debating the object-level question of Probability(doom via AI), but rather for discussing EY’s viewpoint as written in the post. See e.g. https://www.lesswrong.com/posts/34Gkqus9vusXRevR8/late-2021-miri-conversations-ama-discussion for more of a debate.
Debating p(doom) here seems fine to me, unless there’s an explicit request to talk about that elsewhere.
That’s fair.
Sorry, I said it badly/unclearly. What I meant was: most ways to design powerful AI will, on my best guess, be “alien” intelligences, in the sense that they are different from us (think differently, have different goals/values, etc.).
I just want to say I don’t think that was unclear at all. It’s fair to expect people to know the wider meaning of the word ‘alien’.
There’s an analogy being drawn between the power of a hypothetical advanced alien civilization and the power of a superintelligent AI. If you agree that the hypothetical AI would be more powerful, and that an alien civilization capable of travelling to Earth would be a threat, then it follows that superintelligent AI is a threat.
I think most people here are in agreement that AI poses a huge risk, but are differ on how likely it is that we’re all going to die. A 20% chance we’re all going to die is very much worth trying to mitigate sensibly, and the OP says still it’s worth trying to mitigate a 99.9999% chance of human extinction in a similarly level-headed manner (even if the mechanics of doing the work are slightly different at that point).
+1 for asking the 101-level questions! Superintelligence, “AI Alignment: Why It’s Hard, and Where to Start”, “There’s No Fire Alarm for Artificial General Intelligence”, and the “Security Mindset” dialogues (part one, part two) do a good job of explaining why people are super worried about AGI.
“There’s no hope for survival” is an overstatement; the OP is arguing “successfully navigating AGI looks very hard, enough that we should reconcile ourselves with the reality that we’re probably not going to make it”, not “successfully navigating AGI looks impossible / negligibly likely, such that we should give up”.
If you want specific probabilities, here’s a survey I ran last year: https://www.lesswrong.com/posts/QvwSr5LsxyDeaPK5s/existential-risk-from-ai-survey-results. Eliezer works at MIRI (as do I), and MIRI views tended to be the most pessimistic.
Yes! Very much +1! I’ve been hanging around here for almost 10 years and am getting value from the response to the 101-level questions.
Honestly, the Wait But Why posts are my favorite and what I would recommend to a newcomer. (I was careful to say things like “favorite” and “what I’d recommend to a newcomer” instead of “best”.)
Furthermore, I feel like Karolina and people like them who are asking this sort of question deserve an answer that doesn’t require an investment of multiple hours of effortful reading and thinking. I’m thinking something like three paragraphs. Here is my attempt at one. Take it with the appropriate grain of salt. I’m just someone who’s hung around the community for a while but doesn’t deal with this stuff professionally.
Think about how much smarter humans are than dogs. Our intelligence basically gives us full control over them. We’re technologically superior. Plus we understand their psychology and can manipulate them into doing what we want pretty reliably. Now take that and multiply by, uh, a really big number. An AGI would be wayyyyy smarter than us.
Ok, but why do we assume there will be this super powerful AGI? Well, first of all, if you look at surveys, it looks like pretty much everyone does in fact believe it. It’s not something that people disagree on. They just disagree on when exactly it will happen. 10 years? 50 years? 200 years? But to address the question of how, well, the idea is that the smarter a thing is, the more capable it is of improving it’s own smartness. So suppose something starts off with 10 intelligence points and tries to improve and goes to 11. At 11 it’s more capable of improving than it was at 10, so it goes to a 13. +2 instead of +1. At 13 it’s even more capable of improving, so it goes to a 16. Then a 20. Then a 25. So on and so forth. Accelerating growth is powerful.
Ok, so suppose we have this super powerful thing that is inevitable and is going to have complete control over us and the universe. Why can’t we just program the thing from the start to be nice? This is the alignment problem. I don’t have the best understanding of it tbh, but I like to think about the field of law and contracts. Y’know how those terms of service documents are so long and no one reads them? It’s hard to specify all of the things you want upfront. “Oh yeah, please don’t do A. Oh wait I forgot B, don’t do that either. Oh and C, yeah, that’s another important one.” Even I understand that this is a pretty bad description of the alignment problem, but hopefully it at least serves the purpose of providing some amount of intuition.
Really? That wasn’t my read on it. Although I’m not very confident in my read on it. I came away from the post feeling confused.
For example, 0% was mentioned in a few places, like:
Was that April Fools context exaggeration? My understanding is that what Vaniver said here is accurate: that Eliezer is saying that we’re doomed, but that he’s saying it on April Fools so that people who don’t want to believe it for mental health reasons (nervously starts raising hand) have an out. Maybe that’s not accurate though? Maybe it’s more like “We’re pretty doomed but I’m exaggerating/joking about how doomed we are because it’s April Fools Day.”
I think the Wait But Why posts are quite good, though I usually link them alongside Luke Muehlhauser’s reply.
It’s obviously not literally 0%, and the post is explicitly about ‘how do we succeed?’, with a lengthy discussion of the possible worlds where we do in fact succeed:
The whole idea of ‘let’s maximize dignity’ is that it’s just a reframe of ‘let’s maximize the probability that we survive and produce a flourishing civilization’ (the goal of the reframe being to guard against wishful thinking):
Hence:
I dunno what the ‘0%’ means exactly, but it’s obviously not literal. My read of it was something like ‘low enough that it’s hard enough to be calibrated about exactly how low it is’, plus ‘low enough that you can make a lot of progress and still not have double-digit success probability’.
Cool! Good to get your endorsement. And thanks for pointing me to Muehlhauser’s reply. I’ll check it out.
Ok, yeah, that does sound pretty plausible. It still encompasses a pretty wide range though. Like, it could mean one in a billion, I guess? There is a grim tone here. And Eliezer has spoken about his pessimism elsewhere with a similarly grim tone. Maybe I am overreacting to that. I dunno. Like Raemon, I am still feeling confused.
It’s definitely worth noting though that even at a low number like one in a billion, it is still worth working on for sure. And I see that Eliezer believes this as well. So in that sense I take back what I said in my initial comment.
One in a billion seems way, way, way too low to me. (Like, I think that’s a crazy p(win) to have, and I’d be shocked beyond shocked if Eliezer’s p(win) were that low. Like, if he told me that I don’t think I’d believe him.)
Ok, that’s good to hear! Just checking, do you feel similarly about 1 in 100k?
No, that’s a lot lower than my probability but it doesn’t trigger the same ‘that can’t be right, we must be miscommunicating somehow’ reaction.
I see. Thanks for the response.
FYI I am finding myself fairly confused about the “0%” line. I don’t see a reason not to take Eliezer at his word that he meant 0%. “Obviously not literal” feels pretty strong, if he meant a different thing I’d prefer the post say whatever he meant.
Eliezer seemed quite clear to me when he said (paraphrased) “we are on the left side of the logistical success curve, where success is measured in significant digits of leading 0s you are removing from your probability of success”. The whole post seems to clearly imply that Eliezer thinks that marginal dignity is possible, which he defines as a unit of logistical movement on the probability of success. This clearly implies the probability is not literally 0, but it does clearly argue that the probability (on a linear scale) can be rounded to 0.
Mostly I had no idea if he meant like 0.1, 0.001, or 0.00000001. Also not sure if he’s more like “survival chance is 0%, probably, with some margin of error, maybe it’s 1%”, or “no, I’m making the confident claim that it’s more like 0.0001″
(This was combined with some confusion about Nate Soares saying something in the vein of “if you don’t whole heartedly believe in your plans, you should multiply their EV by 0, and you’re not supposed to pick the plan who’s epsilon value is “least epsilon”)
Also, MIRI isn’t (necessarily) a hive mind, so not sure if Rob, Nate or Abram actually share the same estimate of how doomed we are as Eliezer.
Indeed, I expect that the views of at least some individuals working at MIRI vary considerably.
In some ways, the post would seem more accurate to me if it had the Onion-esque headline: Eliezer announces on MIRI’s behalf that “MIRI adopts new ‘Death with Dignity’ strategy.”
Still, I love the post a lot. Also, Eliezer has always been pivotal in MIRI.
The five MIRI responses in my AI x-risk survey (marked with orange dots) show a lot of variation in P(doom):
(Albeit it’s still only five people; maybe a lot of MIRI optimists didn’t reply, or maybe a lot of pessimists didn’t, for some reason.)
Personally, I took it to be 0% within an implied # of significant digits, perhaps in the ballpark of three.
This sequence covers some chunk of it, though it does already assume a lot of context. I think this sequence is the basic case for AI Risk, and doesn’t assume a lot of context.
Because it is an apocalyptic cult and don’t you dare let them pretend otherwise.
People should be able to think the world is at risk, or even doomed, and that solving that is (naturally) something like the most pressing problem, without being whatever it means to be “apocalyptic death cultists”. Otherwise you run into foreseeable issues like new technologies actually presenting existential risks and nobody taking taking those risks seriously for social reasons.
Lmao, I’m the author of the top level comment, and I honestly came to the same conclusion about this place being a cult (or at the very least, cult-like with a TON of red flags). I recently started lurking here again after not thinking about this group for months because I was curious how the rationalists were rationalizing the FTX collapse and also wanted to see their reaction to that openai chat thing.