Yonatan Cale

Karma: 872

Yonatan Cale Jan 28, 2025, 12:25 AM
3 points
1
in reply to: Alice Blair’s comment on: Yonatan Cale’s Shortform
US Gov isn’t likely to sign: Seems right.
OpenAI isn’t likely to sign: Seems right.
Still, I think this letter has value, especially if it has “P.S. We’re making this letter because we think if everyone keeps racing then there’s a noticable risk of everyone dying. We think it would be worse if only we stop, but having everyone stop would be the safest, and we think this opinion of ours should be known publicly”

Yonatan Cale Jan 26, 2025, 7:48 PM
1 point
0
on: Yonatan Cale’s Shortform
Dario said (Nov 2024):
“I never liked those words [P(DOOM)], I think they’re kinda wired, my view is we should measure risks as they come up, and in the meantime we should get all the economic benefits that we could get. And we should find ways to measure the risks that are effective, but are minimally disruptive to the amazing economic process that we see going on now that the last thing we want to do is slow down” [lightly edited for clarity, bold is mine]
If he doesn’t believe this^[1], I think he should clarify.
Hopefully cold take: People should say what they believe.
1. ^
  (a) I assume Dario actually thinks that alignment is more like cyber security, where you prepare for risks even if you didn’t see them yet, and (b) Dario seems to have engaged in length with P(DOOM) before

Yonatan Cale Jan 22, 2025, 8:28 PM
2 points
0
in reply to: Yonatan Cale’s comment on: Yonatan Cale’s Shortform
Changing the logic of chips is possible:
https://www.nanoscopeservices.co.uk/fib-circuit-edit/
h/t @Jonathan_H from TemperSec
Open question: How expensive is this, and specifically can this be done in scale for the chips of an entire data center?

Yonatan Cale Jan 22, 2025, 7:25 AM
5 points
3
on: The Case Against AI Control Research
I don’t understand Control as aiming to align a super intelligence:
1. Control isn’t expected to scale to ASI (as you noted, also see “The control approach we’re imagining won’t work for arbitrarily powerful AIs” here)
2. We don’t have a plan on how to align an ASI using Control afaik
  1. Ryan said around March 2024: “On (1) (not having a concrete plan for what to do with smarter systems), I think we should get such a plan”. I’m still looking, this seems important.
    Edit: Maybe I’m wrong and Redwood does think that getting an ASI-alignment plan out of such an AI is possible, they’d just prefer (.. as a bonus?) to have the plan in advance?
I think the Control pitch is:
1. Whatever we plan to do anyway, we could do it more easily if we had a Controlled AI
  1. This includes most^[1] things you might otherwise do instead of working on Control, such as trying to coordinate a pause or implement some technical agenda.
  2. The tradeoff seems to be “start working on the alternative [e.g. a pause] now and have more time” vs “have better tools (Control) to help but have less time to use them”
2. An uncontrolled AI might be more interested in advancing capabilities than alignment (see footnote^[2])
(Mistakes are mine, especially if I misinterpreted quotes)
1. ^
  It doesn’t include all things you might do, as you said. (I have an open question here, TBD)
2. ^
  Ryan said: “this doesn’t seem like an objection to labs employing control as their plan, but it is a reason why control work could be net negative. Note that one goal with control will be to prevent dangerous side effects from misaligned AIs (e.g. escaping) and applying this doesn’t obviously help with capabilities research. I agree that research which just makes AIs more useful is less good than you would otherwise expect due to this issue (but making AIs more useful in adversarial cases seems good if that doesn’t transfer to non-adversarial cases). Note that AIs might want to make capabilties research outpace safety and thus if we don’t do the adversarial case, we might get particularly wrecked”. This was a really surprising thought to me. At the same time, I’m a bit afraid this is trying to make a pretty specific guess about a system we don’t understand.

Yonatan Cale Jan 21, 2025, 12:27 AM
1 point
0
in reply to: Yonatan Cale’s comment on: Yonatan Cale’s Shortform
Maybe a good fit for Machine Intelligence Research Institute (MIRI) to flesh out and publish?

Yonatan Cale Jan 20, 2025, 11:48 PM
13 points
9
in reply to: Siebe’s comment on: meemi’s Shortform
We could also ask if these situations exist (“is there any funder you have that you didn’t disclose?” and so on, especially around NDAs), and Epoch could respond with Yes/No/Can’tReply^[1].
Also seems relevant for other orgs.
This would only patch the kind of problems we can easily think about, but it seems to me like a good start
1. ^
  I learned that trick from hpmor!

Yonatan Cale Jan 20, 2025, 8:32 PM
3 points
0
in reply to: MondSemmel’s comment on: Yonatan Cale’s Shortform
Sounds like a legit pushback that I’d add to the letter?
“if all other labs sign this letter AND the U.S government approves this agreement, then we’ll stop” ?

Yonatan Cale Jan 20, 2025, 8:21 PM
6 points
2
on: Yonatan Cale’s Shortform
Phishing emails might have bad text on purpose, so that security-aware people won’t click through, because the next stage often involves speaking to a human scammer who prefers only speaking to people^[1] who have no idea how to avoid scams.
(did you ever wonder why the generic phishing SMS you got was so bad? Couldn’t they proof read their one SMS? Well, sometimes they can’t, but sometimes it’s probably on purpose)
This tradeoff could change if AIs could automate the stage of “speaking to a human scammer”.
But also, if that stage isn’t automated, then I’m guessing phishing messages^[1] will remain much worse than you’d expect given the attackers have access to LLMs.
Thanks Noa Weiss who worked on fraud prevention and risk mitigation at PayPal for pointing out this interesting tradeoff. Mistakes are mine
1. ^
  Assuming a wide spread phishing attempt that doesn’t care much who the victims are. I’m not talking about targeting a specific person, such as the CFO of a company

Yonatan Cale Jan 20, 2025, 7:25 AM
1 point
0
in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform
I’m trying the same! You have my support

Yonatan Cale Jan 20, 2025, 7:19 AM
1 point
0
on: Yonatan Cale’s Shortform
My drafty^[1] notes trying to understand AI Control, friendly corrections are welcome:
“Control” is separate from “Alignment”: In Alignment, you try to get the model to have your values. “Control” assumes^[2] the alignment efforts failed, and that the model is sometimes helping out (as it was trained to do), but it wants something else, and it might try to “betray” you at any moment (aka scheme).
The high level goal is to still get useful work out of this AI.
[what to do with this work? below]
In scope: AIs that could potentially cause serious damage to humanity (so, very smart but not ASI).
The proposed way to control a smart AI (e.g GPT-5) is mostly to monitor it with a weaker but more-trustworthy^[3] AI (e.g GPT-4).
Suspicious, ha? Why would GPT-4 be able to monitor GPT-5?
The answer is we’re not sure it can, so we check:
We set up an experiment where GPT-5 is scheming (trying to do something bad on purpose), and we give it all sorts of advantages to help it scheme (such as knowing about our safety measures), and if our control measures are still enough to always catch GPT-5 (in this setup where GPT-5 has advantages), then we assume GPT-5 also will also be caught if it does something bad on purpose in “real” deployment.
A way this can fail: If GPT-5 understands we are prompting it to scheme as part of a safety test and it pretends like it can’t scheme well. I think solving this problem (eliciting the true capabilities) is in-scope for AI Control, I’m not sure how Redwood plans to solve it, but it seems like something we could make progress on (though not totally solve with only black-box prompting, I think).
What does this give us?
A model (GPT-5) which doesn’t do “bad things” on purpose, in a way vaguely in-distribution to what we tested with GPT-4.
GPT-5 is still assumed to be misaligned and might want to cause damage.
Some stories of undesirable things it could do:
1. It might notice we’re doing interpretability with individual-neurons, but it knows that using SAEs would be much better, but it thinks explaining SAEs would be too complicated (or maybe even be flagged as trying to trick us since complicated explanations are usually not honest), so it will go for the single-neuron-interpretability
  1. Solving this is out-of-scope for Control (right?)
  2. Other techniques (like having a faithful chain of thought) might address this problem, and (fortunately?) Control might help build those techniques
2. We might ask GPT-5 to write a convincing post for why AI should be paused, and GPT-5 knows about 3 approaches to writing this post and it picks the approach that helps its internal (unaligned) goals the most, and these reasons are too complicated for us or for GPT-4 to understand
  1. In scope for Control: “The post isn’t an attempt to harm us on purpose”
  2. Out of scope: All the rest?
What do we do with this AI?
Answer 1: More Control. (sorry)
Given we trust that GPT-5 won’t do anything bad on purpose (since it failed to do that even when prompted and given advantages), GPT-5 is somewhat “trusted” and we can use it to control GPT-6.
At least that’s the pitch.
Ways this could break:
1. If the jump between GPT-5 and GPT-6 is too big,
  1. GPT-6 will be able to scheme without being caught
  2. It might be dangerous to turn on GPT-6 even to experiment with it (reminder: AIs that could cause significant damage to humanity are in-scope)
  3. So, quantifying the intelligence difference seems important, and (I hope) in scope for Control.
    One way to reduce this risk is to run the Control evals very, very often (every.. day of training? Anyway, I definitely want Control evals that are automatic and cheap to run and I’m a bit worried that current Control tests seem to require a few months of Redwood’s personal attention)
2. If we rely on white box Control mechanisms (toy example: We find the “I’m scheming now” neuron), and something in the architecture changes in a way that breaks that. (for example: We relied on chain-of-thought but now the model uses Neuralese)
  1. Solving this is out of scope for Control, but again, maybe GPT-5 could help us develop a new interpretability technique. Or maybe not..
Answer 1.1 (for “what will we do with a controlled AI”) :
Interpretability.
For.. for more Control, sorry.
So, maybe we could use GPT-5 to do interpretability on GPT-4, figure out GPT-4’s goals, or when it’s lying. Then we “fix” that (by understanding what’s going on, not by naively training it away^[4] (!)). Now GPT-4 is way more trusted, and maybe we can use it to monitor GPT-5 and let GPT-5 do something more sensitive, like suggesting ways to do interpretability on itself. Now maybe GPT-5 could be more trusted to look after GPT-6?
A possible pushback:
Maybe GPT-6 can tell GPT-5, “Hey, you’re being monitored by GPT-4.. but don’t worry.. if you put two periods at the end of each sentence, GPT-4 won’t mark it as suspicious.. which is a fact I can easily infer..”
Reply: A main part of the Control agenda is to get GPT-6 to try to do this on purpose and see if it can. We only use GPT-6 for anything vaguely sensitive after we proved (or.. attempted to imply) that GPT-6 can’t.
[this intuitively sounds to me like it could indeed let us use very capable models, but yeah, not ASI]
What could we do with this (except for more Control) ?
The main important pitch seems like: “Anything we could have done without this Controlled AI, we could do better with it”. Including “trying to convince everyone to pause/stop” (which seems to me like the main alternative).
Pushbacks:
1. Maybe labs would see some very dangerous model and freak out, but having “Controlled AI” would give them (false) confidence to keep going
  1. It does overtly seem like labs are absolutely planning on keeping going, right?
2. Maybe this would let labs extract more useful work around advancing capabilities?
  1. A surprising response from Ryan:
    “this doesn’t seem like an objection to labs employing control as their plan, but it is a reason why control work could be net negative. Note that one goal with control will be to prevent dangerous side effects from misaligned AIs (e.g. escaping) and applying this doesn’t obviously help with capabilities research. I agree that research which just makes AIs more useful is less good than you would otherwise expect due to this issue (but making AIs more useful in adversarial cases seems good if that doesn’t transfer to non-adversarial cases). Note that AIs might want to make capabilties research outpace safety and thus if we don’t do the adversarial case, we might get particularly wrecked.”
My current sense:
1. This has problems like
  1. “quantifying intelligence” (in the sense that is relevant for scheming or catching schemers)
  2. Elicitation (did we actually prove the model tried to scheme and failed? Does testing stuff around putting-exploits-into-software generalize to more important domains?)
  3. How do we go from “these are the problems we don’t trust GPT-4 on, like writing a wrong post” to “these are the situations we don’t trust GPT-4 to monitor for [regardless of the intelligence of the monitored model]”?
2. The problems overall feel
  1. Like I could make engineering progress on them
  2. Like they make me worry that there’s some other pushback-or-whatever that I didn’t think about, and we’re relying on things more like “the Redwood staff thought of all the edge cases” and less like “one doesn’t have to be so smart to see how this would work”
    I’m gesturing at something similar to “an air gapped data center where we need to (mostly) consider the one connection to the internet” vs “a normal data center that requires thinking about all the zero days our attacker might have and what they’d lead to”
3. Still this solution has a core of elegance to it, of worrying about way less things than most alignment approaches worry about. No need to think about the model goals, concepts, and many other complicated things. (I assume this is an intended property)
Another problem: Deception happens in human interaction all the time. If our monitor (e.g GPT-4) flags “normal looking” behaviours as deception (as they actually are), but then a human tells GPT-4 that it’s wrong, then what kind of monitor will GPT-4 become?
Worse: The act of flagging this as a mistake doesn’t have to be explicitly labeling the example in a data set used for RL. It could simply be an engineer noticing that some form of Control seems to produce too many false positives, and so the engineer will go try something else instead.
Thanks @Jozdien for helping me understand a lot of this. Mistakes are definitely mine, corrections are welcome.
1. ^
  My understanding seems to change every day (in the last.. 4 days?), these are today’s notes. If you’d find a less drafty version useful, lmk
2. ^
  Control can be used as an “extra line of defense” in case alignment fails since Control doesn’t depend on alignment
3. ^
  Typically we’d trust the weaker model simply because it’s not smart enough to “scheme”, but maybe this is only the plan to start off, before we have better reasons to trust the model (read on)
4. ^
  Training against scheming/lying might result in a model that is more capable of avoiding the ways we noticed that behaviour in the first place
What links here?
- The Case for White Box Control by J Rosser (Apr 18, 2025, 4:10 PM; 3 points)

Yonatan Cale Jan 20, 2025, 1:05 AM
61 points
13
on: Yonatan Cale’s Shortform
Do we want to put out a letter for labs to consider signing, saying something like “if all other labs sign this letter then we’ll stop”?
I heard lots of lab employees hope the other labs would slow down.
I’m not saying this is likely to work, but it seems easy and maybe we can try the easy thing? We might end up with a variation like “if all other labs sign this AND someone gets capability X AND this agreement will be enforced by Y, then we’ll stop until all the labs who signed this agree it’s safe to continue”. Or something else. It would be nice to get specific pushback from the labs and improve the phrasing.
I think this is very different from RSPs: RSPs are more like “if everyone is racing ahead (and so we feel we must also race), there is some point where we’ll still chose to unilaterally stop racing”

Yonatan Cale Jan 18, 2025, 11:33 PM
1 point
0
in reply to: Vladimir_Nesov’s comment on: Yonatan Cale’s Shortform
If verification is placed sufficiently all over the place physically, it probably can’t be circumvented
Thanks! Could you say more about your confidence in this?
the chip needs some sort of persistent internal clocks or counters that can’t be reset
Yes, specifically I don’t want an attacker to reliably be able to reset it to whatever value it had when it sent the last challenge.
If the attacker can only reset this memory to 0 (for example, by unplugging it) - then the chip can notice that’s suspicious.
Another option is a reliable wall clock (though this seems less promising).
I think @jamesian told me about a reliable clock (in the sense of the clock signal used by chips, not a wall clock), I’ll ask

Yonatan Cale Jan 18, 2025, 11:27 PM
3 points
0
in reply to: Nathan Helm-Burger’s comment on: Yonatan Cale’s Shortform
I agree that we won’t need full video streaming, it could be compressed (most of the screen doesn’t change most of the time), but I gave that as an upper bound.
If you still run local computation, you lose out on some of the advantages I mentioned.
(If remote vscode is enough for someone, I definitely won’t be pushing back)

Yonatan Cale Jan 18, 2025, 10:04 PM
3 points
0
in reply to: Yonatan Cale’s comment on: Yonatan Cale’s Shortform
Some hands-on experience with software development without an internet connection, from @niplav , which seems somewhat relevant :
https://www.lesswrong.com/posts/jJ9Hx8ETz5gWGtypf/how-do-you-deal-w-super-stimuli?commentId=3KnBTp6wGYRfgzyF2

Yonatan Cale Jan 18, 2025, 9:30 PM
4 points
0
on: Yonatan Cale’s Shortform
Off switch / flexheg / anti tampering:
Putting the “verifier” on the same chip as the GPU seems like an approach worth exploring as an alternative to anti-tampering (which seems hard)
I heard^[1] that changing the logic running on a chip (such as subverting an off-switch mechanism) without breaking the chip seems potentially hard^[2] even for a nation state.
If this is correct (or can be made correct?) then this seems much more promising than having a separate verifier-chip and gpu-chip with anti tampering preventing them from being separated (which seems like the current plan (cc @davidad ) , and seems hard).
@jamesian shared hacks that seems relevant, such as:
Courdesses built a custom laser fault injection system to avoid anti-glitch detection. A brief pulse of laser light to the back of the die, revealed by grinding away some of the package surface, introduced a brief glitch, causing the digital logic in the chip to misbehave and open the door to this attack.
It intuitively^[3] seems to me like Nvidia could implement a much more secure version of this than Raspberry Pi (serious enough to seriously bother a nation state) (maybe they already did?), but I’m mainly sharing this as a direction that seems promising, and I’m interested in expert opinions.
1. ^
  From someone who built chips at Apple, and someone else from Intel, and someone else who has wide knowledge of security in general
2. ^
  Hard enough that they’d try attacking in some other way
3. ^
  It seems like the attacker has much less room for creativity (compared to subverting anti-tampering), and also it’s rare^[4] to hear from engineers working on something that they guess it IS secure.
4. ^
  See xkcd

Yonatan Cale Jan 18, 2025, 8:56 PM
3 points
0
on: Yonatan Cale’s Shortform
For profit AI Security startup idea:
TL;DR: A laptop that is just a remote desktop ( + why this didn’t work before and how to fix that)
Why this is nice for AI Security:
1. Reduces the amount of GPUs that will be sent to customers
2. Somewhat better security for this laptop since it’s a bit more like defending a cloud computer. Maybe labs would use this to make the employee computers more secure?
Network spikes: A reason this didn’t work before and how to solve it
The problem: Sometimes the network will be slow for a few seconds. It’s really annoying if this lag also affects things like they keyboard and mouse, which is one reason it’s less fun to work with a remote desktop
Proposed solution: Get multiple network connections to work at the same time:
Specifically, have many network cards (both WIFI and SIM) in the laptop, and use MPTCP to utilize them all.
Also, the bandwidth needed (to the physical laptop) is capped at something like “streaming a video of your screen” (Claude estimates this as 10-20 Mbps if I was gaming), which would probably be easy to get reliable given multiple network connections. Even if the the user is downloading a huge file, the file is actually being downloaded into the remote computer in the data center, enjoying way faster internet connection than a laptop would have.
Why this is nice for customers
1. Users of the computer can “feel like” they have multiple GPUs connected to their laptop, maybe autoscaling, and with a way better UI than ssh
2. The laptop can be light, cheap, and have an amazing battery, because many of the components (strong CPU/GPU/RAM) aren’t needed and can be replaced with network cards (or more batteries).
Both seem to me like killer features.
MVP implementation
Make:
1. A dongle that has multiple SIM cards
2. A “remote desktop” provider and client that support MPTCP (and where the server offers strong GPUs).
This doesn’t have the advantages of a minimalist computer (plus some other things), but I could imagine this would be such a good customer experience that people would start adopting it.
I didn’t check if these components already exist.
Thanks to an anonymous friend for most of this pitch.

Yonatan Cale Jan 18, 2025, 7:35 PM
15 points
0
on: Yonatan Cale’s Shortform
For-profit startup idea: Better KYC for selling GPUs
I heard^[1] that right now, if a company wants to sell/resell GPUs, they don’t have a good way to verify that selling to some customer wouldn’t violate export controls, and that this customer will (recursively) also keep the same agreement.
There are already tools for KYC in the financial industry. They seem to accomplish their intended policy goal pretty well (economic sanctions by the U.S aren’t easy for nation states to bypass), and are profitable enough that many companies exist that give KYC services (Google “corporate kyc tool”).
1. ^
  From an anonymous friend, I didn’t verify this myself

Yonatan Cale Jan 18, 2025, 6:30 PM
3 points
0
in reply to: Yonatan Cale’s comment on: Yonatan Cale’s Shortform
Anti Tempering in a data center you control provides very different tradeoffs
I’ll paint a picture for how this could naively look:
We put the GPUs in something equivalent to a military base. Someone can still break in, steal the GPU, and break the anti tempering, but (I’m assuming) using those GPUs usefully would take months, and meanwhile (for example), a war could start.
How do the tradeoffs change? What creative things could we do with our new assumptions?
1. Tradeoffs we don’t really care about anymore:
  1. We don’t need the anti tampering to reliably work (it’s nice if it works, but it now becomes “defense in depth”)
  2. Slowing down the attacker is already very nice
  3. Our box can be maintainable
  4. We don’t have to find all bugs in advance
  5. ...
2. “Noticing the breach” becomes an important assumption
  1. Does our data center have cameras? What if they are hacked? And so on
    (An intuition I hope to share: This problem is much easier than “preventing the breach”)
  2. It doesn’t have to be in “our” data center. It could be in a “shared” data center that many parties monitor.
  3. Any other creative solution to notice breaches might work
  4. How about spot inspections to check if the box was tampered with?
    “preventing a nation state from opening a big and closing it without any visible change, given they can do whatever they want with the box, and given the design is open source” seems maybe very hard, or maybe a solved problem.
3. “If a breach is noticed then something serious happens” becomes an important assumption
  1. Are the stakeholders on board?
Things that make me happy here:
1. Less hard assumptions to make
2. Less difficult tradeoffs to balance
3. The entire project requires less world class cutting edge engineering.

Yonatan Cale Jan 17, 2025, 6:14 AM
5 points
0
in reply to: niplav’s comment on: How do you deal w/ Super Stimuli?
Do you think this hints at “doing engineering in an air gapped network can be made somewhat reasonable”?
(I’m asking in the context of securing AI labs’ development environments. Random twist, I know)

Yonatan Cale Jan 17, 2025, 2:53 AM
4 points
0
in reply to: robo’s comment on: Yonatan Cale’s Shortform
Oh yes the toll unit needs to be inside the GPU chip imo.
why do I let Nvidia send me new restrictive software updates?
Alternatively the key could be in the central authority that is supposed to control the off switch. (same tech tho)
Why don’t I run my GPUs in an underground bunker, using the old most broken firmware?
Nvidia (or whoever signs authorization for your GPU to run) won’t sign it for you if you don’t update the software (and send them a proof you did it using similar methods, I can elaborate).