I think people did point out that CICERO lies, and that was a useful update about how shallow attempts to prevent AI deception can fail. I think it could be referenced, and has been referenced, in relevant discussions. I don’t think CICERO provides much or any evidence that we’ll get the kind of scheming that could lead to AI takeover, so it’s not at all surprising that the empirical ML community hasn’t done a massive update. I think the situation will be very different if we do find an AI system that is systematically scheming enough to pose non-negligible takeover risk and ‘catch it red handed’.
I think people did point out that CICERO lies, and that was a useful update about how shallow attempts to prevent AI deception can fail. I think it could be referenced, and has been referenced, in relevant discussions
None of which comes anywhere close to your claims about what labs would do if they caught systematic scheming to deceive and conquer humans in systems trained normally. CICERO schemes very systematically, in a way which depends crucially on the LLM which was not trained to deceive or scheme. It does stuff that would have been considered a while ago a redline. And what analysis does it get? Some cursory ‘pointing out’. Some ‘referencing in relevant discussions’. (Hasn’t even been replicated AFAIK.)
any evidence that we’ll get the kind of scheming that could lead to AI takeover,
See, that’s exactly the problem with this argument—the goalposts will keep moving. The red line will always be a little further beyond. You’re making the ‘warning shot’ argument. CICERO presents every element except immediate blatant risk of AI takeover, which makes it a good place to start squeezing that scientific juice, and yet, it’s still not enough. Because your argument is circular. You can only be convinced of ‘systematic scheming to pose non-negligible takeover risk’ if you’ve already been convinced that it’s ‘systematic scheming to pose non-negligible takeover risk’. You present it as if there were some clear, objective brightline, but there is not and will not be, because each time it’ll be like Sydney or CICERO or …: “oh, it didn’t take over, and therefore doesn’t present a takeover risk” and therefore no update happens. So all your assertion boils down to is the tautology that labs will deeply examine the risky agents they choose to deeply examine.
It seems like you think CICERO and Sydney are bigger updates than I do. Yes, there’s a continuum of cases of catching deception where it’s reasonable for the ML community to update on the plausibility of AI takeover. Yes, it’s important that the ML community updates before AI systems pose significant risk, and there’s a chance that they won’t do so. But I don’t see the lack of strong update towards p(doom) from CICERO as good evidence that the ML community won’t update if we get evidence of systematic scheming (including trying to break out of the lab when there was never any training signal incentivising that behaviour). I think that kind of evidence would be much more relevant to AI takeover risk than CICERO.
To clarify my position in case i’ve been misunderstood. I’m not saying the ML community will definitely update in time. I’m saying that if there is systematic scheming and we catch it red-handed (as I took Buck to be describing) then there will likely be a very significant update. And CICERO seems like a weak counter example (but not zero evidence)
I think people did point out that CICERO lies, and that was a useful update about how shallow attempts to prevent AI deception can fail. I think it could be referenced, and has been referenced, in relevant discussions. I don’t think CICERO provides much or any evidence that we’ll get the kind of scheming that could lead to AI takeover, so it’s not at all surprising that the empirical ML community hasn’t done a massive update. I think the situation will be very different if we do find an AI system that is systematically scheming enough to pose non-negligible takeover risk and ‘catch it red handed’.
None of which comes anywhere close to your claims about what labs would do if they caught systematic scheming to deceive and conquer humans in systems trained normally. CICERO schemes very systematically, in a way which depends crucially on the LLM which was not trained to deceive or scheme. It does stuff that would have been considered a while ago a redline. And what analysis does it get? Some cursory ‘pointing out’. Some ‘referencing in relevant discussions’. (Hasn’t even been replicated AFAIK.)
See, that’s exactly the problem with this argument—the goalposts will keep moving. The red line will always be a little further beyond. You’re making the ‘warning shot’ argument. CICERO presents every element except immediate blatant risk of AI takeover, which makes it a good place to start squeezing that scientific juice, and yet, it’s still not enough. Because your argument is circular. You can only be convinced of ‘systematic scheming to pose non-negligible takeover risk’ if you’ve already been convinced that it’s ‘systematic scheming to pose non-negligible takeover risk’. You present it as if there were some clear, objective brightline, but there is not and will not be, because each time it’ll be like Sydney or CICERO or …: “oh, it didn’t take over, and therefore doesn’t present a takeover risk” and therefore no update happens. So all your assertion boils down to is the tautology that labs will deeply examine the risky agents they choose to deeply examine.
It seems like you think CICERO and Sydney are bigger updates than I do. Yes, there’s a continuum of cases of catching deception where it’s reasonable for the ML community to update on the plausibility of AI takeover. Yes, it’s important that the ML community updates before AI systems pose significant risk, and there’s a chance that they won’t do so. But I don’t see the lack of strong update towards p(doom) from CICERO as good evidence that the ML community won’t update if we get evidence of systematic scheming (including trying to break out of the lab when there was never any training signal incentivising that behaviour). I think that kind of evidence would be much more relevant to AI takeover risk than CICERO.
To clarify my position in case i’ve been misunderstood. I’m not saying the ML community will definitely update in time. I’m saying that if there is systematic scheming and we catch it red-handed (as I took Buck to be describing) then there will likely be a very significant update. And CICERO seems like a weak counter example (but not zero evidence)