I have saved this post on the internet archive[1].
If in 5-15 years, the prediction does not come true, i would like it to be saved as evidence of one of the many serious claims that world-ending AI will be with us in very short timelines. I think the author has given more than enough detail on what they mean by AGI, and has given more than enough detail on what it might look like, so it should be obvious whether or not the prediction comes true. In other words, no rationalising past this or taking it back. If this is what the author truly believes, they should have a permanent record of their abilities to make predictions.
I encourage everyone to save posts similar to this one in the internet archive. The AI community, if there is one, is quite divided on issues like these, and even among groups that are in broad agreement there are disagreements on details. It will be very useful to have a public archive of who made what claims so we know who to avoid and who to take seriously.
There are three kinds of people. Those who in the past made predictions which turned out to be false, those who didn’t make predictions, and those who in the past made predictions which turned out to be true. Obviously the third kind is the best & should be trusted the most. But what about the first and second kinds?
I get the impression from your comment that you think the second kind is better than the first kind; that the first kind should be avoided and the second kind taken seriously (provided they are making plausible arguments etc.) If so, I disagree; I’m not sure which kind is better, I could see it being the case that generally speaking the first kind is better (again provided they are making plausible arguments etc.)
If the Author believes what they’ve written then they clearly think that it would be more dangerous to ignore this than to be wrong about it, so I can’t really argue that they shouldn’t be person number 1. It’s a comfortable moral position you can force yourself into though. “If I’m wrong, at least we avoided total annihilation, so in a way I still feel good about myself”.
I see this particular kind of prediction as a kind of ethical posturing and can’t in good conscience let people make them without some kind of accountability. People have been paid millions to work on predictions similar to these. If they are wrong, they should be held accountable in proportion to whatever cost they have have incurred on society, big or small, financial or behavioural.
If wrong, I don’t want anyone brushing these predictions off as silly mistakes, simple errors in models, or rationalising them away. “That’s not actually what they meant by AGI”, or “It was better to be wrong than say nothing, please keep taking me seriously”. Sometimes mistakes are made because of huge fundamental errors in understanding across the entire subject and we do need a record of that for reasons more important than fun and games, so definitely be the first kind of person but, you know, people are watching is all.
Hmm. Apparently you meant something a little more extreme than I first thought. It kind of sounds like you think the content of my post is hazardous.
I see this particular kind of prediction as a kind of ethical posturing and can’t in good conscience let people make them without some kind of accountability.
Not sure what you mean by ethical posturing here. It’s generally useful for people to put their reasoning and thoughts out in public so that other people can take from the reasoning what they find valuable, and making a bunch of predictions ahead of time makes the reasoning testable.
For example, I’d really, really like it if a bunch of people who think long timelines are more likely wrote up detailed descriptions of their models and made lots of predictions. Who knows, they might know things I don’t, and I might change my mind! I’d like to!
People have been paid millions to work on predictions similar to these.
I, um, haven’t. Maybe the FTX Future Fund will decide to throw money at me later if they think the information was worth it to them, but that’s their decision to make.
If they are wrong, they should be held accountable in proportion to whatever cost they have have incurred on society, big or small, financial or behavioural.
If I am to owe a debt to Society if I am wrong, will Society pay me if I am right? Have I established a bet with Society? No. I just spent some time writing up why I changed my mind.
Going through the effort to provide testable reasoning is a service. That’s what FTX would be giving me money for, if they give me any money at all.
You may make the valid argument that I should consider possible downstream uses of the information I post- which I do! Not providing the information also has consequences. I weighed them to the best of my ability, but I just don’t see much predictable harm from providing testable reasoning to an audience of people who understand reasoning under uncertainty. (Incidentally, I don’t plan to go on cable news to be a talking head about ~impending doom~.)
I’m perfectly fine with taking a reputational hit for being wrong about something I should have known, or paying up in a bet when I lose. I worry what you’re proposing here is something closer to “stop talking about things in public because they might be wrong and being wrong might have costs.” That line of reasoning, taken to the limit, yields arresting seismologists.
I did say I think making wrong predictions can be dangerous, but i would have told you explicitly to stop if I thought yours was particularly dangerous (moreso just a bit ridiculous, if I’m being honest). I think you should see the value in keeping a record of what people say, without equating it to anti-science mobbing.
If I am to owe a debt to Society if I am wrong, will Society pay me if I am right?
Sure, you will be paid in respect and being taken seriously, because it wasn’t a bet like you said. That’s why I’m also not asking you to pay anything if you are wrong, you’re not one of the surprisingly many people asking for millions to work on this problem. I don’t expect them to pay anything either, but it would be nice. I’m not going to hold Nuremberg trials for AGI doomers or anything ridiculous like that.
The author explicitly states that their probability of the entire human race going extinct or some equivalent disaster will be 80% if AGI is developed by 2025. They also gave the probability of developing AGI by <2025 less than 5%. or so. Since AGI was, according to you and no one else, developed right here in 2023, this would make Porby’s estimate of extinction chance even higher than 80% and they would very wrong about when AGI would be developed. So tell me, do we give it to Porby even though the human race has not gone extinct and they were obviously way off on other estimates? No of course we don’t, because like I said in post one, Porby has clearly defined AGI in their own way, and whatever ambiguous model existing today that you think of as AGI is not a match with their definition of strong AGI.
Porby has clearly defined AGI in their own way, and whatever ambiguous model existing today that you think of as AGI is not a match with their definition of strong AGI.
Yup.
For what its worth, I haven’t noticeably updated my timelines since the post; GPT-4 and whatnot are pretty much what I expected, and I’d be pretty surprised if GPT-5 eats us.
(Edit: my P(doom by date) has actually gone down a bit for technical reasons as I’ve continued research! I’ll probably add a more complete update to this post as a comment when I get around to editing it for the openphil version of the contest.)
I have saved this post on the internet archive[1].
If in 5-15 years, the prediction does not come true, i would like it to be saved as evidence of one of the many serious claims that world-ending AI will be with us in very short timelines. I think the author has given more than enough detail on what they mean by AGI, and has given more than enough detail on what it might look like, so it should be obvious whether or not the prediction comes true. In other words, no rationalising past this or taking it back. If this is what the author truly believes, they should have a permanent record of their abilities to make predictions.
I encourage everyone to save posts similar to this one in the internet archive. The AI community, if there is one, is quite divided on issues like these, and even among groups that are in broad agreement there are disagreements on details. It will be very useful to have a public archive of who made what claims so we know who to avoid and who to take seriously.
[1] https://web.archive.org/web/20221020151610/https://www.lesswrong.com/posts/K4urTDkBbtNuLivJx/why-i-think-strong-general-ai-is-coming-soon
May the forces of the cosmos intervene to make me look silly.
There are three kinds of people. Those who in the past made predictions which turned out to be false, those who didn’t make predictions, and those who in the past made predictions which turned out to be true. Obviously the third kind is the best & should be trusted the most. But what about the first and second kinds?
I get the impression from your comment that you think the second kind is better than the first kind; that the first kind should be avoided and the second kind taken seriously (provided they are making plausible arguments etc.) If so, I disagree; I’m not sure which kind is better, I could see it being the case that generally speaking the first kind is better (again provided they are making plausible arguments etc.)
If the Author believes what they’ve written then they clearly think that it would be more dangerous to ignore this than to be wrong about it, so I can’t really argue that they shouldn’t be person number 1. It’s a comfortable moral position you can force yourself into though. “If I’m wrong, at least we avoided total annihilation, so in a way I still feel good about myself”.
I see this particular kind of prediction as a kind of ethical posturing and can’t in good conscience let people make them without some kind of accountability. People have been paid millions to work on predictions similar to these. If they are wrong, they should be held accountable in proportion to whatever cost they have have incurred on society, big or small, financial or behavioural.
If wrong, I don’t want anyone brushing these predictions off as silly mistakes, simple errors in models, or rationalising them away. “That’s not actually what they meant by AGI”, or “It was better to be wrong than say nothing, please keep taking me seriously”. Sometimes mistakes are made because of huge fundamental errors in understanding across the entire subject and we do need a record of that for reasons more important than fun and games, so definitely be the first kind of person but, you know, people are watching is all.
Hmm. Apparently you meant something a little more extreme than I first thought. It kind of sounds like you think the content of my post is hazardous.
Not sure what you mean by ethical posturing here. It’s generally useful for people to put their reasoning and thoughts out in public so that other people can take from the reasoning what they find valuable, and making a bunch of predictions ahead of time makes the reasoning testable.
For example, I’d really, really like it if a bunch of people who think long timelines are more likely wrote up detailed descriptions of their models and made lots of predictions. Who knows, they might know things I don’t, and I might change my mind! I’d like to!
I, um, haven’t. Maybe the FTX Future Fund will decide to throw money at me later if they think the information was worth it to them, but that’s their decision to make.
If I am to owe a debt to Society if I am wrong, will Society pay me if I am right? Have I established a bet with Society? No. I just spent some time writing up why I changed my mind.
Going through the effort to provide testable reasoning is a service. That’s what FTX would be giving me money for, if they give me any money at all.
You may make the valid argument that I should consider possible downstream uses of the information I post- which I do! Not providing the information also has consequences. I weighed them to the best of my ability, but I just don’t see much predictable harm from providing testable reasoning to an audience of people who understand reasoning under uncertainty. (Incidentally, I don’t plan to go on cable news to be a talking head about ~impending doom~.)
I’m perfectly fine with taking a reputational hit for being wrong about something I should have known, or paying up in a bet when I lose. I worry what you’re proposing here is something closer to “stop talking about things in public because they might be wrong and being wrong might have costs.” That line of reasoning, taken to the limit, yields arresting seismologists.
I did say I think making wrong predictions can be dangerous, but i would have told you explicitly to stop if I thought yours was particularly dangerous (moreso just a bit ridiculous, if I’m being honest). I think you should see the value in keeping a record of what people say, without equating it to anti-science mobbing.
Sure, you will be paid in respect and being taken seriously, because it wasn’t a bet like you said. That’s why I’m also not asking you to pay anything if you are wrong, you’re not one of the surprisingly many people asking for millions to work on this problem. I don’t expect them to pay anything either, but it would be nice. I’m not going to hold Nuremberg trials for AGI doomers or anything ridiculous like that.
So do we call it in favor of porby, or wait a bit longer for the ambiguity over whether we’ve truly crossed the AGI threshold to resolve?
The author explicitly states that their probability of the entire human race going extinct or some equivalent disaster will be 80% if AGI is developed by 2025. They also gave the probability of developing AGI by <2025 less than 5%. or so. Since AGI was, according to you and no one else, developed right here in 2023, this would make Porby’s estimate of extinction chance even higher than 80% and they would very wrong about when AGI would be developed. So tell me, do we give it to Porby even though the human race has not gone extinct and they were obviously way off on other estimates? No of course we don’t, because like I said in post one, Porby has clearly defined AGI in their own way, and whatever ambiguous model existing today that you think of as AGI is not a match with their definition of strong AGI.
Yup.
For what its worth, I haven’t noticeably updated my timelines since the post; GPT-4 and whatnot are pretty much what I expected, and I’d be pretty surprised if GPT-5 eats us.
(Edit: my P(doom by date) has actually gone down a bit for technical reasons as I’ve continued research! I’ll probably add a more complete update to this post as a comment when I get around to editing it for the openphil version of the contest.)