He starts by emphasizing
The issue is so hotly debated because the stakes are major: According to some estimates, quadrillions of dollars of net present value are up for grabs, not to mention political power great enough to significantly disrupt the current world order.
[...]
The most important thing to realize, through all the noise of discussions and debates, is a very simple and indisputable fact: while we are racing towards AGI or even ASI, nobody currently knows how such an AGI or ASI could be made to behave morally, or at least behave as intended by its developers and not turn against humans.
And goes on to do a pretty great job addressing “those who think
AGI and ASI are impossible or are centuries in the future
AGI is possible but only in many decades
that we may reach AGI but not ASI, those who think that AGI and ASI will be kind to us
that corporations will only design well-behaving AIs and existing laws are sufficient
who think that we should accelerate AI capabilities research and not delay benefits of AGI
talking about catastrophic risks will hurt efforts to mitigate short-term human-rights issues with AI
concerned with the US-China cold war
that international treaties will not work
the genie is out of the bottle and we should just let go and avoid regulation
that open-source AGI code and weights are the solution
worrying about AGI is falling for Pascal’s wager”
In the section “For those who think that open-source AGI code and weights are the solution”
The National Institute of Health’s answer is “yes”. Here’s variola major (smallpox) for example. So those arguing that it’s a bad idea to share ML algorithms and artifacts should either make the case that the NIH is wrong to share the smallpox genome or make the case that sharing some subset of ML algorithms and artifacts is more dangerous than sharing the smallpox genome.
In fairness, some people have in fact made decent cracks at the argument that sharing some types of ML-related information is more dangerous than sharing the smallpox genome. Still, I think the people arguing that the spread of knowledge is the thing we want to target, rather than the spread of materials, could do a better job of making that argument. But the common-sense “you wouldn’t share the genome of a dangerous virus” argument doesn’t work because we would, in fact, share the genome of a dangerous virus (and I personally think that it’s actively good that we share the genomes of dangerous viruses, because it allows for stuff like this).
Personally, my take is that this is a dangerous mistake on the part of NIH and the worldwide academic community generally, that so much dangerous information has been released in regards to virology and bioweapons. Unfortunately, I’m clearly currently in the minority among scientists who think this. I wish I had a good idea of how to change people’s minds on this issue, and the connected one of publishing AI capabilities implementation details.
I initially thought there must be some simple reason that publishing the DNA sequence is not a dangerous thing to do, like “ok, but given that you would need a world class lab and maybe even some techniques which haven’t even been invented yet to get it to work, it’s not a dangerous thing to publish”.
According to this article from 2002, synthesising smallpox would be tricky, but within the reach of a terrorist organisation. Other viruses may be easier.
Apparently, large organisations like the NIH are foolhardy enough to publish dangerous data like this. I wonder if there’s some other justification, like “the data was already public, in such a way that it could not be removed”