The treaty can constrain the largest projects like DeepMind. Identifying them can be done by an international court. We don’t need to defend against the smartest humans, only the crackpots that still think we’re in an arms race instead of a bomb defusal operation. Imagine that the Manhattan project mathematicians had computed that a nuke had a 90% chance of igniting the atmosphere. Would their generals still have been table thumping that they should develop the nukes before the Nazis do? I think the more pressing concern is to make every potential researcher, including the Nazis, aware of the 90% chance. Legislation helping to slow is all your top-level comment requires.
The treaty can constrain the largest projects like DeepMind. Identifying them can be done by an international court. We don’t need to defend against the smartest humans, only the crackpots that still think we’re in an arms race instead of a bomb defusal operation. Imagine that the Manhattan project mathematicians had computed that a nuke had a 90% chance of igniting the atmosphere. Would their generals still have been table thumping that they should develop the nukes before the Nazis do? I think the more pressing concern is to make every potential researcher, including the Nazis, aware of the 90% chance. Legislation helping to slow is all your top-level comment requires.