Fabien Roger

• Fabien Roger 20 Nov 2024 21:53 UTC

  LW: 4 AF: 3

  1

  AF

  in reply to: Dakara’s comment on: By De­fault, GPTs Think In Plain Sight

  Here are ways you could make using GPT-n safer if its Chain-of-Thoughts are understandable by a team of linguists and AI experts:

  • On a small random subset of deployment Chain-of-Thought, and when running adversarial evaluations /​ honeypots aimed at tricking the model into revealing a potential misalignment, actually call your team of linguists and AI experts to analyze what GPT-n is generating (it probably isn’t that much text anyway). Not seeing evidence of misalignment in the Chain-of-Thought in such circumstances is evidence that 1. the model is not having misaligned thoughts (in CoT) very frequently and 2. the model is not having such thoughts in circumstances where it is most crucial for GPT-n to be deceptive.

  • If the model doesn’t use Klingon or dense jargon in its Chain-of-Thoughts (which I predict it won’t), monitor all CoT of GPT-n with a weaker cheaper model (and/​or a regex). If something looks suspicious to the weaker monitor, you can:

    • call your team of linguists and AI experts to analyze it further

    • or resample the CoT (this likely was just a fluke, since most CoT don’t have worrying signs)

    • or give up on this particular query (e.g. send an error message to your user, use another weaker model to generate an answer, …)

    • or ask another copy of GPT-n to analyze the CoT (though this can be tricky to get right because of the risk of correlated failures), only calling your team of linguists and AI experts if more evidence of this CoT being suspicious is found.

  This is not perfect, but I think this can represent an important layer of defense against the risks of AIs that are powerful enough to speed up alignment research but that are not vastly superhuman.

• Fabien Roger 18 Nov 2024 18:16 UTC

  LW: 2 AF: 2

  0

  AF

  in reply to: Buck’s comment on: Buck’s Shortform

  Technical measures to prevent users from using the AI for particular tasks don’t help against the threat of the lab CEO trying to use the AI for those harmful tasks

  Actually, it is not that clear to me. I think adversarial robustness is helpful (in conjunction with other things) to prevent CEOs from misusing models.

  If at some point in a CEO trying to take over wants to use HHH to help them with the takeover, that model will likely refuse to do egregiously bad things. So the CEO might need to use helpful-only models. But there might be processes in place to access helpful-only models—which might make it harder for the CEO to take over. So while I agree that you need good security and governance to prevent a CEO from using helpful-only models to take over, I think that without good adversarial robustness, it is much harder to build adequate security/​governance measures without destroying an AI-assisted-CEO’s productivity.
  
  There is a lot of power concentration risk that just comes from people in power doing normal people-in-power things, such as increasing surveillance on dissidents—for which I agree that adversarial robustness is ~useless. But security against insider threats is quite useless too.

• Fabien Roger 17 Nov 2024 13:47 UTC

  5 points

  1

  in reply to: Satron’s comment on: AI Con­trol: Im­prov­ing Safety De­spite In­ten­tional Subversion

  That would work if the untrusted model is better at inserting subtle backdoors than the “human” high-quality labor used to generate the synthetic backdoors (used to validate that the untrusted model doesn’t just lie).

  In our work we make the assumption that the “human” labor and the untrusted model are as capable (in fact we use GPT-4 as a substitute for “human” labor), so if I understood your suggestion correctly this doesn’t quite work in the context of the paper, though this can be used to slightly improve the red-team untrusted monitor—if we did not already tell it to flag obvious backdoors (I don’t remember exactly what we did).

  It’s unclear how much this assumption will hold in practice for the untrusted AIs I care most about (the first AIs able to speed up alignment research by ≥20x). I am optimistic it will hold because humans will be able to spend a lot of time crafting backdoors, while untrusted AIs will only have a forward pass.

• Fabien Roger 30 Oct 2024 21:43 UTC

  2 points

  0

  in reply to: Julian Stastny’s comment on: The case for un­learn­ing that re­moves in­for­ma­tion from LLM weights

  I am not sure that it is over-conservative. If you have an HP-shaped that can easily be transformed in HP-data using fine-tuning, does it give you a high level of confidence that people misusing the model won’t be able to extract the information from the HP-shaped hole or that a misaligned model won’t be able to notice to HP-shaped hole and use that to answer to question to HP when it really wants to?

  I think that it depends on the specifics of how you built the HP-shaped hole (without scrambling the information). I don’t have a good intuition for what a good technique like that could look like. A naive thing that comes to mind would be something like “replace all facts in HP by their opposite” (if you had a magic fact-editing tool), but I feel like in this situation it would be pretty easy for an attacker (human misuse or misaligned model) to notice “wow all HP knowledge has been replaced by anti-HP knowledge” and then extract all the HP information by just swapping the answers.

• Fabien Roger 30 Oct 2024 21:37 UTC

  2 points

  0

  in reply to: Julian Stastny’s comment on: The case for un­learn­ing that re­moves in­for­ma­tion from LLM weights

  That’s right.

  I think it’s not that devastating, since I expect that their method can be adapted to counter classic LoRA tuning (the algorithm takes as input some set finetuning methods it “trains against”). But yeah, it’s not reassuring that it doesn’t generalize between full-weight FT and LoRA.

• Fabien Roger 28 Oct 2024 17:09 UTC

  4 points

  0

  on: Over­com­ing Bias Anthology

  I have converted the anthology into an audio format using the default (and terrible) AWS AI voice (mostly for my own consumption): https://​​podcasters.spotify.com/​​pod/​​show/​​fabien368/​​episodes/​​Overcoming-bias-anthology—our-thinking-e2q61mg

• Fabien Roger 24 Oct 2024 11:14 UTC

  LW: 6 AF: 5

  0

  AF

  on: the case for CoT un­faith­ful­ness is overstated

  I like this post. I made similar observations and arguments in The Translucent Thoughts Hypotheses and Their Implications but these were less clear and buried at the end of the post.

• Fabien Roger 23 Oct 2024 14:24 UTC

  12 points

  0

  on: [Paper] Hid­den in Plain Text: Emer­gence and Miti­ga­tion of Stegano­graphic Col­lu­sion in LLMs

  Very cool work!

  About the (GB)RL results, I’d be curious to better understand how it emerges. Given how human understandable these seem, I guess that what happens is that there is an already existing subtle link by default between the payload and the generated text at least some of the time, and that training

  1. teaches the receiver to recognize this signal

  2. reinforces that behavior

  3. makes the behavior gradually more subtle

  Do you share this interpretation? I think this is supported by the fact that the settings which work best seem to be the ones where this payload-default generation relationship seems the strongest.

  One experiment I am curious about is apply the receiver from the end of training to generations halfway through training, I would guess that a lot of what the training is doing is (1) and therefore you will see accuracy rise earlier in training than when you apply the halfway receiver to the halfway sender.

  I’ve added to the Other experiments of my negative results post the following paragraph:

  Hidden in Plain Text has some positive results on some similar setups! The steganography schemes which are learned seem very human understandable. Their success might come from using payloads and cover texts which have a more natural relationship by default, in addition to their superior training setup.

• Fabien Roger 21 Oct 2024 12:59 UTC

  2 points

  0

  in reply to: Bogdan Ionut Cirstea’s comment on: The case for un­learn­ing that re­moves in­for­ma­tion from LLM weights

  These are mostly unlearning techniques, right? I am not familiar enough with diffusion models to make an informed guess about how good the unlearning of the meta-learning paper is (but in general, I am skeptical, in LLMs all attempts at meta-unlearning have failed, providing some robustness only against specific parameterizations, so absent evaluations of reparametrizations, I wouldn’t put much weights on the results).

  I think the inter-based unlearning techniques look cool! I would be cautious about using them for evaluation though, especially when they are being optimized against like in the ConceptVector paper. I put more faith in SGD making loss go down when the knowledge is present.

• Fabien Roger 21 Oct 2024 12:50 UTC

  2 points

  0

  in reply to: Roger Dearnaley’s comment on: The case for un­learn­ing that re­moves in­for­ma­tion from LLM weights

  It would be great if this was solvable with security, but I think it’s not for most kinds of deployments. For external deployments you almost can never know if a google account corresponds to a student or a professional hacker. For internal deployments you might have misuse from insiders. Therefore, it is tempting to try to address that with AI (e.g. monitoring, refusals, …), and in practice many AI labs try to use this as the main layer of defense (e.g. see OpenAI’s implementation of model spec, which relies almost entirely on monitoring account activity and models refusing clearly ill-intentioned queries). But I agree this is far from being bulletproof.

• Fabien Roger 17 Oct 2024 8:43 UTC

  LW: 7 AF: 7

  0

  AF

  on: The case for un­learn­ing that re­moves in­for­ma­tion from LLM weights

  Good fine-tuning robustness (i.e. creating models which attackers have a hard time fine-tuning to do a target task) could make the framework much harder to apply. The existence of such technique is a main motivation for describing it as an adversarial framework rather than just saying “just do fine-tuning”. All existing tamper resistant technique can be broken (Repnoise fails if you choose the learning rate right, Tamper-resistant fine-tuning method fails if you use LoRA …), and if you use unlearning techniques which look like that, you should really do the basic fine-tuning variations that break Repnoise and Tamper-resistant fine-tuning when evaluating your technique.

  This creates a bit of FUD, but I expect the situation to be much less bad than in the jailbreak robustness literature, since I expect fine-tuning robustness to be much more offense-favored than jailbreak robustness (i.e. even playing the attacker a little gets you massive returns, whereas in jailbreak robustness it’s super annoying to play the attacker properly) because in fine-tuning robustness it is much harder to patch individual attacks than in jailbreak robustness. I think good fine-tuning robustness (given full access and decent resources to the attacker) is hopeless, and this makes the evaluation of unlearning using the method described here possible.

  But if fine-tuning robustness was ever successful, this would make the situation much more scary: maybe you could make your “unlearning” method model “forget” the hazardous info on most prompts, but “remember it” if there is a password in the prompt (i.e. people can’t extract the information if they don’t know what the password is, even with white-box access to the weights). If this were the case, there would be something horrific going on because the hazardous information would clearly still in the weights, but the method described in this post would claim there isn’t. If you applied the same unlearning technique in a more straightforward way (without the password backdoor) and people didn’t manage to extract the information, I would be afraid the information would still be there.

The case for un­learn­ing that re­moves in­for­ma­tion from LLM weights

• Fabien Roger 11 Oct 2024 12:39 UTC

  LW: 7 AF: 7

  0

  AF

  on: Be­hav­ioral red-team­ing is un­likely to pro­duce clear, strong ev­i­dence that mod­els aren’t scheming

  Overall, making a safety case from these techniques seems really rough. I don’t see why someone would believe that (conditional on non-scheming) you have a 90% chance of getting a 20x Bayes factor against scheming. I don’t think this is just me saying “it’s very hard to get a 20x Bayes factor because that requires a lot of confidence”—I think that all the sources of uncertainty I described above are much worse than approaches to safety cases that rest on e.g. either inability or on control

  In this post, when you say getting evidence against scheming, you more generally mean excluding AIs that will successfully cause doom? In particular:

  • BRT is terrible at excluding extremely cautious schemers that would do anything bad only in very weird circumstances that are very unlikely to ever happen in deployment. But most of the edge from BRT is showing that schemers would need to be extremely cautious (i.e. if BRT shows no danger, I think your P(doom|scheming) probably got a bigger update than your P(scheming)).

  • Control does not change P(scheming), and I think it’s unlikely any class of technique will give us something close to a 20x Bayes factor against scheming TAI. So the post makes more sense to me if it is about Bayes factors against doom.

  I think this post makes good arguments showing that BRT can’t give you a big Bayes factor against doom, and in particular I really like the more realistic example of the RSA-2048-strategy.

  the returns to effort are such that other AI control techniques should have three times as much effort put into them

  There are some shared things you need for BRT and other control techniques, and in particular you need good threat modeling and a good automated catastrophe detection for both inference-time monitoring and BRT. If you exclude security and things like not giving internet access to AIs unless they really need to from “other AI control techniques” (as people usually don’t think about them in these terms), I think 3x between BRT (minus threat modeling and catastrophe detection) other-control (minus threat modeling and catastrophe detection) is too big. Threat modeling, BRT and inference-time monitoring were already some of the biggest effort sinks for control anyway. The other big control efforts I can think about figuring out how to use more trusted labors in the scaffolds, how to do resampling well, how to prevent input-coordination and how to prevent steg, maybe I am missing other big efforts?

• Fabien Roger 5 Oct 2024 15:10 UTC

  LW: 19 AF: 13

  4

  AF

  on: Fa­bien’s Shortform

  I listened to the book Hacking the Hacker by Roger Grimes. It helped me think of other ways in which regular cybersecurity naturally transfers/​can be adapted to help prevent AI takeovers:

  • Honeypots are a well-studied topic in cybersecurity. If you have infiltrated a network, you must be careful with the data you collect and the vulnerabilities you exploit because some of them may be traps! This kind of exercise feels close to the kind of attitude a misaligned AI has to have when taking subtle bad actions. The book is light on details about how honeypots are built and how they fail, and this is a topic I want to learn more about.

  • Humans are weak in somewhat LLM-like ways: you can’t trust them with passwords, they will break your rules if that feels appropriate to them, … The kind of things you might use to make trusted monitoring work even in worlds where adversarial robustness isn’t solved might look a lot like the cursed stack of incentives and warnings people use in cyber when they want to make sure humans don’t mindlessly follow the instructions of an attacker. (Though LLMs are also different/​weaker than humans in a bunch of ways.)

  (The other classic ways in which cyber transfers to preventing takeovers are mostly about how when you prevent spies from stealing model weights or using models in unauthorized ways, you naturally create infra that also makes it hard for AIs to self-exfiltrate or run unmonitored.)

  It was also fun to see arguments and drama that are close in kind to the kind of arguments about what are the best ways to prevent AI catastrophes: people in cybersecurity argue about whether focusing on vulnerabilities is right or whether it’s better to focus on the human element, whether the focus on strong passwords is right, whether some solutions are too bothersome/​costly to be used in practice, whether imposing specific cybersecurity standards is a good idea, … It made me realize how niche AI most safety arguments must look like to people outside the AI safety community.

  But I don’t recommend reading the book if you are not curious about everyday cybersecurity. Most of the book is about more regular everyday cyberattacks (social engineering, common vulns, …) cybersecurity (patching, passwords, …), and advice for people who want to do cyber professionally. It has a bunch of resource recommendations, but I don’t know yet how good they are.

• Fabien Roger 5 Oct 2024 13:39 UTC

  4 points

  0

  in reply to: LucaRighetti’s comment on: Is cy­ber­crime re­ally cost­ing trillions per year?

  This is great, thank you very much!

[Question] Is cy­ber­crime re­ally cost­ing trillions per year?

• Fabien Roger 17 Sep 2024 12:45 UTC

  LW: 2 AF: 2

  0

  AF

  in reply to: Mark Xu’s comment on: Es­ti­mat­ing Tail Risk in Neu­ral Networks

  Thanks for your explanations!

  Monitoring lets you do online training when you find catastrophe, but e.g. there might be no safe fallback action that allows you to do monitoring safely.

  Good point. I agree that if you don’t have time or resources to train the policy against the monitor on-the-fly, doing the catastrophe-suppression upfront is very nice.

  It is worth noting that if “training against the monitor” didn’t work because the monitor was not robust enough, then my understanding is that the same issue would plague the tail-risk estimation approach: for optimization against tail-risk to not get gamed, you not only need the tail-risk estimation to be robust, but you also need the catastrophe detector to be robust. Is that right?

  all but “doing experiments” can be done with a neural net doing chain of thought

  I would have thought there is a big gap between “doing tail-risk estimation for an NN forward pass” and “doing tail-risk estimation for an NN CoT”. Argmax (the one in T=0 sampling) is not a nice non-linearity, and my guess is that none of the hopes for NN tail-risk estimation present in the post allow you to make claims about weak model CoT. If you do layer-by-layer modeling (going through n_layers x n_sampled_tokens steps), I would be surprised if you ended up being able to show sth like “after careful consideration, and despite the naive warning signs (which you need to have, otherwise the CoT is not very useful), the CoT always converges to saying that there is no catastrophe”—it’s not like in single-forward-pass land where there is a clear sense in which the activations should not go in certain regions. I haven’t thought about it much, so maybe I am too pessimistic about dealing with the argmax?

• Fabien Roger 16 Sep 2024 8:35 UTC

  LW: 4 AF: 2

  0

  AF

  on: Es­ti­mat­ing Tail Risk in Neu­ral Networks

  Do you think that catastrophe detectors will be large neural networks?

  If you do, how do you build a neural network (without sampling steps nor python execution steps) that implements your major hopes for getting a weak model to supervise a strong model?

  • We could give \(M_{weak}\) more time to think than \(M\), so it could examine each portion of the action in detail.

  • We could allow \(M_{weak}\) to interrogate \(M\) about the function of various parts of its action, and do detailed checks for inconsistencies in \(M\)’s answers.

  • We could allow \(M_{weak}\) to perform separate experiments to verify \(M\)’s claims about its actions, and to evaluate whether or not the actions have any side effects.

  • \(M_{weak}\) could be conservative when evaluating \(M\)’s action, flagging part of the action it did not understand as potentially catastrophic.

  If we back down from such hopes that use a lot of inference-time compute, and just use a single weak-model forward pass, then it looks to me like trusted monitoring solves most (but not all) issues that estimating tail risks targets.

  If you don’t, do you think that it is easy to go from NN-tail-risk estimation to a more general form of tail-risk estimation? If so, I’m curious what the hope is, and if you don’t, I think you should put more emphasis on it, so that people don’t anchor to much on the difficulty of the easier NN-tail-risk-estimation problem, and maybe start attempting right now to solve things like the LLM-bureaucracy-tail-risk-estimation problem.

• Fabien Roger 31 Aug 2024 12:47 UTC

  2 points

  0

  in reply to: M. Y. Zuo’s comment on: Fa­bien’s Shortform

  My bad, I should have said “a decade or two”, which I think is more plausible. I agree that the combination of “a few years” and a slow enough takeoff that things aren’t completely out of distribution is very unlikely.

• Fabien Roger 30 Aug 2024 13:12 UTC

  4 points

  2

  in reply to: M. Y. Zuo’s comment on: Fa­bien’s Shortform

  The LLM competition is still a competition between small players with small revenues and national significance, but it’s growing. I think it’s plausible that in a few years the competition around LLMs will reach the same kind of significance that the chip industry has (or bigger), with hundreds of billions in capital investment and sales per year, massive involvement of state actors, interest from militaries, etc. and may also go through similar dynamics (e.g. leading labs exploiting monopolistic positions without investing in the right things, massive spy campaigns, corporate deals to share technology, …).

  The LLM industry is still a bunch of small players with grand ambitions, and looking at an industry that went from “a bunch of small players with grand ambitions” to “0.5%-1% of world GDP (and key element of the tech industry)” in a few decades can help inform intuitions about geopolitics and market dynamics (though there are a bunch of differences that mean it won’t be the same).