To start off, I don’t see much point in formally betting $20 on an event conditioned on something I assign <<50% probability of happening within the next 30 years (powerful AI is launched and failed catastrophically and we’re both still alive to settle the bet and there was an unambiguous attribution of the failure to the AI). I mean sure, I can accept the bet, but largely because I don’t believe it matters one way or another, so I don’t think it counts from the epistemological virtue standpoint.
But I can state what I’d disagree with in your terms if I were to take it seriously, just to clarify my argument:
Sounds good.
Mostly sounds good, but I’d push back that “not actually running anything close to the dangerous limit” sounds like a win to me, even if theoretical research continues. One pretty straightforward Schelling point for a ban/moratorium on AGI research is “never train or run anything > X parameters”, with X << dangerous level at then-current paradigm. It may be easier explain to the public and politicians than many other potential limits, and this is important. It’s much easier to control too—checking that nobody collects and uses a gigashitton of GPUs [without supervision] is easier than to check every researcher’s laptop. Additionally, we’ll have nuclear weapons tests as a precedent.
That’s the core of my argument, really. If the consortium of 200 world experts says “this happened because your AI wasn’t aligned, let’s stop all AI research”, then Facebook AI or China can tell the consortium to go fuck themselves, and I agree with your skepticism that it’d make all labs pause for even a month (see: gain of function research, covid). But if it becomes public knowledge that a catastrophe of 1mln casualties happened because of AI, then it can trigger a panic which will make both the world leaders and the public to really honestly want to restrict this AI stuff, and it will both justify and enable the draconian measures required to make every lab to actually stop the research. Similar to how panics about nuclear energy, terrorism and covid worked. I propose defining “public agreement” as “leaders of the relevant countries (defined as the countries housing the labs from p.1, so US, China, maybe UK and a couple of others) each issue a clear public statement saying that the catastrophe happened because of an unaligned AI”. This is not an unreasonable ask, they were this unanimous about quite a few things, including vaccines.
To start off, I don’t see much point in formally betting $20 on an event conditioned on something I assign <<50% probability of happening within the next 30 years (powerful AI is launched and failed catastrophically and we’re both still alive to settle the bet and there was an unambiguous attribution of the failure to the AI). I mean sure, I can accept the bet, but largely because I don’t believe it matters one way or another, so I don’t think it counts from the epistemological virtue standpoint.
But I can state what I’d disagree with in your terms if I were to take it seriously, just to clarify my argument:
Sounds good.
Mostly sounds good, but I’d push back that “not actually running anything close to the dangerous limit” sounds like a win to me, even if theoretical research continues. One pretty straightforward Schelling point for a ban/moratorium on AGI research is “never train or run anything > X parameters”, with X << dangerous level at then-current paradigm. It may be easier explain to the public and politicians than many other potential limits, and this is important. It’s much easier to control too—checking that nobody collects and uses a gigashitton of GPUs [without supervision] is easier than to check every researcher’s laptop. Additionally, we’ll have nuclear weapons tests as a precedent.
That’s the core of my argument, really. If the consortium of 200 world experts says “this happened because your AI wasn’t aligned, let’s stop all AI research”, then Facebook AI or China can tell the consortium to go fuck themselves, and I agree with your skepticism that it’d make all labs pause for even a month (see: gain of function research, covid). But if it becomes public knowledge that a catastrophe of 1mln casualties happened because of AI, then it can trigger a panic which will make both the world leaders and the public to really honestly want to restrict this AI stuff, and it will both justify and enable the draconian measures required to make every lab to actually stop the research. Similar to how panics about nuclear energy, terrorism and covid worked. I propose defining “public agreement” as “leaders of the relevant countries (defined as the countries housing the labs from p.1, so US, China, maybe UK and a couple of others) each issue a clear public statement saying that the catastrophe happened because of an unaligned AI”. This is not an unreasonable ask, they were this unanimous about quite a few things, including vaccines.