I listened to the book Hacking the Hacker by Roger Grimes. It helped me think of other ways in which regular cybersecurity naturally transfers/can be adapted to help prevent AI takeovers:
Honeypots are a well-studied topic in cybersecurity. If you have infiltrated a network, you must be careful with the data you collect and the vulnerabilities you exploit because some of them may be traps! This kind of exercise feels close to the kind of attitude a misaligned AI has to have when taking subtle bad actions. The book is light on details about how honeypots are built and how they fail, and this is a topic I want to learn more about.
Humans are weak in somewhat LLM-like ways: you can’t trust them with passwords, they will break your rules if that feels appropriate to them, … The kind of things you might use to make trusted monitoring work even in worlds where adversarial robustness isn’t solved might look a lot like the cursed stack of incentives and warnings people use in cyber when they want to make sure humans don’t mindlessly follow the instructions of an attacker. (Though LLMs are also different/weaker than humans in a bunch of ways.)
(The other classic ways in which cyber transfers to preventing takeovers are mostly about how when you prevent spies from stealing model weights or using models in unauthorized ways, you naturally create infra that also makes it hard for AIs to self-exfiltrate or run unmonitored.)
It was also fun to see arguments and drama that are close in kind to the kind of arguments about what are the best ways to prevent AI catastrophes: people in cybersecurity argue about whether focusing on vulnerabilities is right or whether it’s better to focus on the human element, whether the focus on strong passwords is right, whether some solutions are too bothersome/costly to be used in practice, whether imposing specific cybersecurity standards is a good idea, … It made me realize how niche AI most safety arguments must look like to people outside the AI safety community.
But I don’t recommend reading the book if you are not curious about everyday cybersecurity. Most of the book is about more regular everyday cyberattacks (social engineering, common vulns, …) cybersecurity (patching, passwords, …), and advice for people who want to do cyber professionally. It has a bunch of resource recommendations, but I don’t know yet how good they are.
I listened to the book Hacking the Hacker by Roger Grimes. It helped me think of other ways in which regular cybersecurity naturally transfers/can be adapted to help prevent AI takeovers:
Honeypots are a well-studied topic in cybersecurity. If you have infiltrated a network, you must be careful with the data you collect and the vulnerabilities you exploit because some of them may be traps! This kind of exercise feels close to the kind of attitude a misaligned AI has to have when taking subtle bad actions. The book is light on details about how honeypots are built and how they fail, and this is a topic I want to learn more about.
Humans are weak in somewhat LLM-like ways: you can’t trust them with passwords, they will break your rules if that feels appropriate to them, … The kind of things you might use to make trusted monitoring work even in worlds where adversarial robustness isn’t solved might look a lot like the cursed stack of incentives and warnings people use in cyber when they want to make sure humans don’t mindlessly follow the instructions of an attacker. (Though LLMs are also different/weaker than humans in a bunch of ways.)
(The other classic ways in which cyber transfers to preventing takeovers are mostly about how when you prevent spies from stealing model weights or using models in unauthorized ways, you naturally create infra that also makes it hard for AIs to self-exfiltrate or run unmonitored.)
It was also fun to see arguments and drama that are close in kind to the kind of arguments about what are the best ways to prevent AI catastrophes: people in cybersecurity argue about whether focusing on vulnerabilities is right or whether it’s better to focus on the human element, whether the focus on strong passwords is right, whether some solutions are too bothersome/costly to be used in practice, whether imposing specific cybersecurity standards is a good idea, … It made me realize how niche AI most safety arguments must look like to people outside the AI safety community.
But I don’t recommend reading the book if you are not curious about everyday cybersecurity. Most of the book is about more regular everyday cyberattacks (social engineering, common vulns, …) cybersecurity (patching, passwords, …), and advice for people who want to do cyber professionally. It has a bunch of resource recommendations, but I don’t know yet how good they are.