Also, Google is already training big models and keeping them under wraps very securely.
This seems straightforwardly false. The current status quo at Google as of earlier this year is <SL3 level security (not robust to normal insiders or to terrorists groups) based on what is said in their frontier model framework.
By contrast, Google published The Interview (a parody of the monster at the top of the Un Dynasty) on its Google Play Store after the North Koreans hacked Sony for making it and threatened anyone who published it with reprisal. Everyone else wimped out. It wasn’t going to be published at all, but Google said “bring it!”… and then North Korea presumably threw their script kiddies at Gog Ma Gog’s datacenters and made no meaningful dent whatsoever (because there were no additional stories about it after that (which NK would have yelled about if they had anything to show for their attacks)).
I don’t think this is much evidence for the claim you say. Many parts of the US government are robust to North Korea and being robust to some North Korean operation is much easier than being robust to all Chinese operations attacking a diverse range of targets.
Basically, Google is already outperforming state actors here.
I’m quite skeptical that Google is uniformly better than the goverment, though I agree parts of google are better than parts of government. It’s hard to assign a net security competence rating.
I’m reporting the “thonk!” in my brain like a proper scholar and autist, but I’m not expecting my words to fully justify what happened in my brain.
I believe what I believe, and can unpack some of the reasons for it in text that is easy and ethical for me to produce, but if you’re not convinced then that’s OK in my book. Update as you will <3
I worked at Google for ~4 years starting in 2014 and was impressed by the security posture.
When I ^f for [SL3] in that link and again in the PDF it links to, there are no hits (and [terror] doesn’t occur in either source either) so I’m not updating much from what you said.
My general prior is that the older any government subagency (or heck, even any institution) is, the more likely it is to survive for even longer into the future, and the more likely it is to be incompetent-unto-evil-in-practice.
Google is relatively young. Younger than the NSA or NIST. Deepmind started outside of Google and is even younger.
When I ^f for [SL3] in that link and again in the PDF it links to, there are no hits (and [terror] doesn’t occur in either source either) so I’m not updating much from what you said.
The frontier model framework says:
0: Status quo
Industry standard development and enterprise controls. E.g., multi-factor authentication, basic access control mechanisms, secure software development standards, red-team tests.
And the next level (1: Controlled access) says “Approximately RAND L3” implying that status quo is <L3 (this is presumably SL3 which is the term used in the RAND report, I don’t know why they used a different term).
I think you’re overindexing on the phrase “status quo”, underindexing on “industry standard”, and missing a lot of practical microstructure.
Lots of firms or teams across industry have attempted to “EG” implement multi-factor authentication or basic access control mechanisms or secure software development standards or red-team tests. Sony probably had some of that in some of its practices in some of its departments when North Korea 0wned them.
Google does not just “OR them together” and half-ass some of these things. It “ANDs together” reasonably high quality versions of everything. Then every year they anneal the culture a little bit more around small controlled probes of global adequacy.
..
Also, in reading that RAND document, I would like to report another“thonk!” sound!
..
Rand’s author(s) seem to have entirely (like at a conceptual level) left out the possibility that AGI (during a training run or during QA with humans or whatever) would itself “become the attacker” and need to be defended against.
You don’t just have to keep bad guys OUT, you have to keep “the possible bad guy that was just created by a poorly understood daemon summoning process” IN, and that perspective doesn’t appear anywhere in any of the RAND document that I can see.
No results when I ^f for [demon], [summon], [hypno], [subvert], [pervert], [escape].
((“Subvert” was used once, but it was in a basic bitch paragraph like this (bold in original):
Most access control systems are either software systems or have significant software components. In addition to the more specialized ways of undermining such systems described above, an attacker could undermine them by finding code vulnerabilities and then subverting their behavior (without actually dealing with their cryptographic or core functionality at all). A major category of code vulnerabilities that undermine access control systems on a regular basis are privilege escalation vulnerabilities.
Duh.
))
The best thing in that RAND document is [infected] which occurs twice <3
Page 54 (bold not in original):
PyTorch’s prerelease Linux packages, PyTorch-nightly, were compromised via a malicious binary on PyPi.122 The torchtriton dependency in the PyTorch-nightly build was replaced with a malicious package, leading to the extraction of sensitive data from infected systems.
Page 61 (bold not in original, footnote not a link in the original):
Agent.btz, the malware that infected the U.S. Department of Defense in 2008 and led to the creation of U.S. Cyber Command, came from a USB device that was found in a parking lot on a base in the Middle East.164
“Infection” is a really good metaphor and word for bringing extreme rigor to the tracing of actual causal processes, because it gets at the idea of “programs” (viral or microbial or fungal or whatever genomes (or genome products) spreading via arbitrary physical transmission methods). This is a prett darn correct frame! :-)
Ever since covid I’ve been beating the drum for BSL5 to handle scary diseases by assuming that the real danger is NOT that “a known disease brought into a lab to be studied there merely infects a scientist (or everyone inside the facility for that matter)”, but that someone inside the BSL5 gets infected with something new that would never naturally evolve and then uses creativity to escape the BSL5 while infected and then that causes a pandemic.
Currently the BSL system only goes up to 4, and all incremental increases in caution and safety are related to the protection of the lab worker from what they are working with more and more and more, while assuming good faith by the site administrator, and the security guards at the exit gate (which may or may not be distinct from the entry gate) who might also get infected, and so on.
I want a Bio-SL5 standard to exist in biology, and I want all GoF research restricted by an international treaty that ensures that the GoF only happens inside BSL5 facilities.
Similarly, I also think humanity also needs an analogous “CompSci-SL5 standard” that imagines the entire computing facility (data center plus programmers plus janitors who have physical access and so on plus all the managers of everyone working inside the CSSL5 campus) is subverted by a nascent superintelligence and then used by that nascent superintelligence to try to subvert all of Earth.
There is no hint of anything at all like this as part of the threat modeling in the RAND report.
Also, if Google had such a thing back when I worked there, I didn’t hear about it. (Then again, maybe the existence of it would have been kept secret?)
“In March 2021, Slaoui was fired from the board of GSK subsidiary Galvani Bioelectronics over what GSK called “substantiated” sexual harassment allegations stemming from his time at the parent company.[4] Slaoui issued an apology statement and stepped down from positions at other companies at the same time.[5]”
Yeah. I know. I’m relatively cynical about such things. Imagine how bad humans are in general if that is what an unusually good and competent and heroic human is like!
This seems straightforwardly false. The current status quo at Google as of earlier this year is <SL3 level security (not robust to normal insiders or to terrorists groups) based on what is said in their frontier model framework.
I don’t think this is much evidence for the claim you say. Many parts of the US government are robust to North Korea and being robust to some North Korean operation is much easier than being robust to all Chinese operations attacking a diverse range of targets.
I’m quite skeptical that Google is uniformly better than the goverment, though I agree parts of google are better than parts of government. It’s hard to assign a net security competence rating.
I’m reporting the “thonk!” in my brain like a proper scholar and autist, but I’m not expecting my words to fully justify what happened in my brain.
I believe what I believe, and can unpack some of the reasons for it in text that is easy and ethical for me to produce, but if you’re not convinced then that’s OK in my book. Update as you will <3
I worked at Google for ~4 years starting in 2014 and was impressed by the security posture.
When I ^f for [SL3] in that link and again in the PDF it links to, there are no hits (and [terror] doesn’t occur in either source either) so I’m not updating much from what you said.
I remember how the FDA handled covid, but I also remember Operation Warp Speed.
One of those teams was dismantled right afterwards. The good team (that plausibly saved millions of lives) was dismantled, not the bad one (that killed on the order of a million people whose deaths could have been prevented by quickly deployed covid tests in December in airports). The leader of the good team left government service almost instantly after he succeeded and has never been given many awards or honors.
My general prior is that the older any government subagency (or heck, even any institution) is, the more likely it is to survive for even longer into the future, and the more likely it is to be incompetent-unto-evil-in-practice.
Google is relatively young. Younger than the NSA or NIST. Deepmind started outside of Google and is even younger.
The frontier model framework says:
And the next level (1: Controlled access) says “Approximately RAND L3” implying that status quo is <L3 (this is presumably SL3 which is the term used in the RAND report, I don’t know why they used a different term).
I think you’re overindexing on the phrase “status quo”, underindexing on “industry standard”, and missing a lot of practical microstructure.
Lots of firms or teams across industry have attempted to “EG” implement multi-factor authentication or basic access control mechanisms or secure software development standards or red-team tests. Sony probably had some of that in some of its practices in some of its departments when North Korea 0wned them.
Google does not just “OR them together” and half-ass some of these things. It “ANDs together” reasonably high quality versions of everything. Then every year they anneal the culture a little bit more around small controlled probes of global adequacy.
..
Also, in reading that RAND document, I would like to report another “thonk!” sound!
..
Rand’s author(s) seem to have entirely (like at a conceptual level) left out the possibility that AGI (during a training run or during QA with humans or whatever) would itself “become the attacker” and need to be defended against.
It is like they haven’t even seen Ex Machina, or read A Fire Upon The Deep or Daemon.
You don’t just have to keep bad guys OUT, you have to keep “the possible bad guy that was just created by a poorly understood daemon summoning process” IN, and that perspective doesn’t appear anywhere in any of the RAND document that I can see.
No results when I ^f for [demon], [summon], [hypno], [subvert], [pervert], [escape].
((“Subvert” was used once, but it was in a basic bitch paragraph like this (bold in original):
Duh.
))
The best thing in that RAND document is [infected] which occurs twice <3
Page 54 (bold not in original):
Page 61 (bold not in original, footnote not a link in the original):
“Infection” is a really good metaphor and word for bringing extreme rigor to the tracing of actual causal processes, because it gets at the idea of “programs” (viral or microbial or fungal or whatever genomes (or genome products) spreading via arbitrary physical transmission methods). This is a prett darn correct frame! :-)
Ever since covid I’ve been beating the drum for BSL5 to handle scary diseases by assuming that the real danger is NOT that “a known disease brought into a lab to be studied there merely infects a scientist (or everyone inside the facility for that matter)”, but that someone inside the BSL5 gets infected with something new that would never naturally evolve and then uses creativity to escape the BSL5 while infected and then that causes a pandemic.
Currently the BSL system only goes up to 4, and all incremental increases in caution and safety are related to the protection of the lab worker from what they are working with more and more and more, while assuming good faith by the site administrator, and the security guards at the exit gate (which may or may not be distinct from the entry gate) who might also get infected, and so on.
I want a Bio-SL5 standard to exist in biology, and I want all GoF research restricted by an international treaty that ensures that the GoF only happens inside BSL5 facilities.
Similarly, I also think humanity also needs an analogous “CompSci-SL5 standard” that imagines the entire computing facility (data center plus programmers plus janitors who have physical access and so on plus all the managers of everyone working inside the CSSL5 campus) is subverted by a nascent superintelligence and then used by that nascent superintelligence to try to subvert all of Earth.
There is no hint of anything at all like this as part of the threat modeling in the RAND report.
Also, if Google had such a thing back when I worked there, I didn’t hear about it. (Then again, maybe the existence of it would have been kept secret?)
From the wiki of the good team guy
“In March 2021, Slaoui was fired from the board of GSK subsidiary Galvani Bioelectronics over what GSK called “substantiated” sexual harassment allegations stemming from his time at the parent company.[4] Slaoui issued an apology statement and stepped down from positions at other companies at the same time.[5]”
Yeah. I know. I’m relatively cynical about such things. Imagine how bad humans are in general if that is what an unusually good and competent and heroic human is like!