If the Author believes what they’ve written then they clearly think that it would be more dangerous to ignore this than to be wrong about it, so I can’t really argue that they shouldn’t be person number 1. It’s a comfortable moral position you can force yourself into though. “If I’m wrong, at least we avoided total annihilation, so in a way I still feel good about myself”.
I see this particular kind of prediction as a kind of ethical posturing and can’t in good conscience let people make them without some kind of accountability. People have been paid millions to work on predictions similar to these. If they are wrong, they should be held accountable in proportion to whatever cost they have have incurred on society, big or small, financial or behavioural.
If wrong, I don’t want anyone brushing these predictions off as silly mistakes, simple errors in models, or rationalising them away. “That’s not actually what they meant by AGI”, or “It was better to be wrong than say nothing, please keep taking me seriously”. Sometimes mistakes are made because of huge fundamental errors in understanding across the entire subject and we do need a record of that for reasons more important than fun and games, so definitely be the first kind of person but, you know, people are watching is all.
Hmm. Apparently you meant something a little more extreme than I first thought. It kind of sounds like you think the content of my post is hazardous.
I see this particular kind of prediction as a kind of ethical posturing and can’t in good conscience let people make them without some kind of accountability.
Not sure what you mean by ethical posturing here. It’s generally useful for people to put their reasoning and thoughts out in public so that other people can take from the reasoning what they find valuable, and making a bunch of predictions ahead of time makes the reasoning testable.
For example, I’d really, really like it if a bunch of people who think long timelines are more likely wrote up detailed descriptions of their models and made lots of predictions. Who knows, they might know things I don’t, and I might change my mind! I’d like to!
People have been paid millions to work on predictions similar to these.
I, um, haven’t. Maybe the FTX Future Fund will decide to throw money at me later if they think the information was worth it to them, but that’s their decision to make.
If they are wrong, they should be held accountable in proportion to whatever cost they have have incurred on society, big or small, financial or behavioural.
If I am to owe a debt to Society if I am wrong, will Society pay me if I am right? Have I established a bet with Society? No. I just spent some time writing up why I changed my mind.
Going through the effort to provide testable reasoning is a service. That’s what FTX would be giving me money for, if they give me any money at all.
You may make the valid argument that I should consider possible downstream uses of the information I post- which I do! Not providing the information also has consequences. I weighed them to the best of my ability, but I just don’t see much predictable harm from providing testable reasoning to an audience of people who understand reasoning under uncertainty. (Incidentally, I don’t plan to go on cable news to be a talking head about ~impending doom~.)
I’m perfectly fine with taking a reputational hit for being wrong about something I should have known, or paying up in a bet when I lose. I worry what you’re proposing here is something closer to “stop talking about things in public because they might be wrong and being wrong might have costs.” That line of reasoning, taken to the limit, yields arresting seismologists.
I did say I think making wrong predictions can be dangerous, but i would have told you explicitly to stop if I thought yours was particularly dangerous (moreso just a bit ridiculous, if I’m being honest). I think you should see the value in keeping a record of what people say, without equating it to anti-science mobbing.
If I am to owe a debt to Society if I am wrong, will Society pay me if I am right?
Sure, you will be paid in respect and being taken seriously, because it wasn’t a bet like you said. That’s why I’m also not asking you to pay anything if you are wrong, you’re not one of the surprisingly many people asking for millions to work on this problem. I don’t expect them to pay anything either, but it would be nice. I’m not going to hold Nuremberg trials for AGI doomers or anything ridiculous like that.
If the Author believes what they’ve written then they clearly think that it would be more dangerous to ignore this than to be wrong about it, so I can’t really argue that they shouldn’t be person number 1. It’s a comfortable moral position you can force yourself into though. “If I’m wrong, at least we avoided total annihilation, so in a way I still feel good about myself”.
I see this particular kind of prediction as a kind of ethical posturing and can’t in good conscience let people make them without some kind of accountability. People have been paid millions to work on predictions similar to these. If they are wrong, they should be held accountable in proportion to whatever cost they have have incurred on society, big or small, financial or behavioural.
If wrong, I don’t want anyone brushing these predictions off as silly mistakes, simple errors in models, or rationalising them away. “That’s not actually what they meant by AGI”, or “It was better to be wrong than say nothing, please keep taking me seriously”. Sometimes mistakes are made because of huge fundamental errors in understanding across the entire subject and we do need a record of that for reasons more important than fun and games, so definitely be the first kind of person but, you know, people are watching is all.
Hmm. Apparently you meant something a little more extreme than I first thought. It kind of sounds like you think the content of my post is hazardous.
Not sure what you mean by ethical posturing here. It’s generally useful for people to put their reasoning and thoughts out in public so that other people can take from the reasoning what they find valuable, and making a bunch of predictions ahead of time makes the reasoning testable.
For example, I’d really, really like it if a bunch of people who think long timelines are more likely wrote up detailed descriptions of their models and made lots of predictions. Who knows, they might know things I don’t, and I might change my mind! I’d like to!
I, um, haven’t. Maybe the FTX Future Fund will decide to throw money at me later if they think the information was worth it to them, but that’s their decision to make.
If I am to owe a debt to Society if I am wrong, will Society pay me if I am right? Have I established a bet with Society? No. I just spent some time writing up why I changed my mind.
Going through the effort to provide testable reasoning is a service. That’s what FTX would be giving me money for, if they give me any money at all.
You may make the valid argument that I should consider possible downstream uses of the information I post- which I do! Not providing the information also has consequences. I weighed them to the best of my ability, but I just don’t see much predictable harm from providing testable reasoning to an audience of people who understand reasoning under uncertainty. (Incidentally, I don’t plan to go on cable news to be a talking head about ~impending doom~.)
I’m perfectly fine with taking a reputational hit for being wrong about something I should have known, or paying up in a bet when I lose. I worry what you’re proposing here is something closer to “stop talking about things in public because they might be wrong and being wrong might have costs.” That line of reasoning, taken to the limit, yields arresting seismologists.
I did say I think making wrong predictions can be dangerous, but i would have told you explicitly to stop if I thought yours was particularly dangerous (moreso just a bit ridiculous, if I’m being honest). I think you should see the value in keeping a record of what people say, without equating it to anti-science mobbing.
Sure, you will be paid in respect and being taken seriously, because it wasn’t a bet like you said. That’s why I’m also not asking you to pay anything if you are wrong, you’re not one of the surprisingly many people asking for millions to work on this problem. I don’t expect them to pay anything either, but it would be nice. I’m not going to hold Nuremberg trials for AGI doomers or anything ridiculous like that.