What I keep seeing is that it seems overwhelmingly based on firsthand experience with open, transparent organizations, even though most of the relevant decisions are made by closed, nontransparent orgs like tech companies and military-minded, nationalistic governments.
It’s quite natural that you can’t see the nontransparent engagements of the people with whom you are talking. Eliezer did speak about his experiences with being invited to military planning exercises in the past.
Organisations like MIRI and other EA organizations that get a good portion of their budget from billionaires are well connected to speak to influential people behind closed doors.
Meanwhile, the emergence of intense cybersecurity competition has made chipmaking and computer software more opaque than ever, and there is plenty of room for it to be strangled further, merely by the threat of individual hackers (let alone individual Manhattan-projects).
While the threat is high it still hasn’t made the NSA privot from being more interested in defense and getting companies to fix vulnerabilities then it’s interested in exploiting those vulnerabilities.
Also, the publicly known history of nuclear weapons isn’t all that terrible. According to Thomas Schelling’s foundational book Arms and Influence (1966), nuclear escalations are effectively two nuclear powers playing a game of chicken, ramping up the odds of nuclear war until one of the sides folds.
Thomas Schelling didn’t know that if Arkhipov wouldn’t have stopped the submarine commander from launching nuclear weapons, that those would have actually be used in the cuban missle crisis.
When it comes to understand our current capability regarding nuclear weapons Dominic Cummings who did have inside insight summaries the current state in the UK in his substack:
If you think I’m probably too pessimistic, then ponder this comment by Professor Allison who has spent half a century in these circles: ‘Over the past decade, I have yet to meet a senior member of the US national security team who had so much as read the official national security strategies’ (emphasis added). NB. he is referring to reading the official strategies, not the explanations of why they are partly flawed!
Even if you think we weren’t just lucky with nuclear safety it’s quite different then AGI risk.
Biosafety seems more similar. We are two years into a pandemic that likely exists because of people screweing up biosafety and there a lot of publish pressure against stating it as a basic fact. In a world where it’s hard to admit basic facts, there’s we lack the political will to shut down other gain-of-function research.
Shutting down gain-of-function research after a catastroph that made us all suffer looks like a problem that would be comparably easier than regulating AGI but powerful forces are not coordinating to make that happen.
There’s big billionaires and little billionaires. And then there’s military elites and big billionaires. Inequality is prevalent among elites too, and insulation from ambitious and well-connected outsiders is a prerequisite to having any sort of stability in national security decisionmaking. However, personal networks abound and all sorts of things can happen due to chance.
I think that nuclear accidents are very real but they are also overemphasized on lesswrong, and far too few people here know the basics of nuclear deterrence and coercion, which are one of the biggest prerequisites to understanding nuclear standoffs and major conflicts like Ukraine. Deliberate action can be depicted as accidents, dramatically decreasing risks and costs of the deliberate action.
Gain-of-function in the current era is understandable and sane, even if it’s unfortunate. The programs aren’t gathering dust anymore, which they’ve generally appeared to do since WW2. It’s terrible news, obviously, but everyone’s thinking about it, which means everyone’s thinking about everyone thinking about it. Also, in terms of offense-defense, deterrence and MAD can be outmaneuvered if the enemy has many more options than you, e.g. they can do something that’s a little bit insane and your only option for retaliation is to retaliate with something that’s extremely insane.
It’s quite natural that you can’t see the nontransparent engagements of the people with whom you are talking. Eliezer did speak about his experiences with being invited to military planning exercises in the past.
Organisations like MIRI and other EA organizations that get a good portion of their budget from billionaires are well connected to speak to influential people behind closed doors.
While the threat is high it still hasn’t made the NSA privot from being more interested in defense and getting companies to fix vulnerabilities then it’s interested in exploiting those vulnerabilities.
Thomas Schelling didn’t know that if Arkhipov wouldn’t have stopped the submarine commander from launching nuclear weapons, that those would have actually be used in the cuban missle crisis.
When it comes to understand our current capability regarding nuclear weapons Dominic Cummings who did have inside insight summaries the current state in the UK in his substack:
Even if you think we weren’t just lucky with nuclear safety it’s quite different then AGI risk.
Biosafety seems more similar. We are two years into a pandemic that likely exists because of people screweing up biosafety and there a lot of publish pressure against stating it as a basic fact. In a world where it’s hard to admit basic facts, there’s we lack the political will to shut down other gain-of-function research.
Shutting down gain-of-function research after a catastroph that made us all suffer looks like a problem that would be comparably easier than regulating AGI but powerful forces are not coordinating to make that happen.
There’s big billionaires and little billionaires. And then there’s military elites and big billionaires. Inequality is prevalent among elites too, and insulation from ambitious and well-connected outsiders is a prerequisite to having any sort of stability in national security decisionmaking. However, personal networks abound and all sorts of things can happen due to chance.
I think that nuclear accidents are very real but they are also overemphasized on lesswrong, and far too few people here know the basics of nuclear deterrence and coercion, which are one of the biggest prerequisites to understanding nuclear standoffs and major conflicts like Ukraine. Deliberate action can be depicted as accidents, dramatically decreasing risks and costs of the deliberate action.
Gain-of-function in the current era is understandable and sane, even if it’s unfortunate. The programs aren’t gathering dust anymore, which they’ve generally appeared to do since WW2. It’s terrible news, obviously, but everyone’s thinking about it, which means everyone’s thinking about everyone thinking about it. Also, in terms of offense-defense, deterrence and MAD can be outmaneuvered if the enemy has many more options than you, e.g. they can do something that’s a little bit insane and your only option for retaliation is to retaliate with something that’s extremely insane.