“indifference over infinite bitstrings” is a misnomer in an important sense, because it’s literally impossible to construct a normalized probability measure over infinite bitstrings that assigns equal probability to each one. What you’re talking about is the length weighted measure that assigns exponentially more probability mass to shorter programs. That’s definitely not an indifference principle, it’s baking in substantive assumptions about what’s more likely.
I don’t see why we should expect any of this reasoning about Turing machines to transfer over to neural networks at all, which is why I didn’t cast the counting argument in terms of Turing machines in the post. In the past I’ve seen you try to run counting or simplicity arguments in terms of parameters. I don’t think any of that works, but I at least take it more seriously than the Turing machine stuff.
If we’re really going to assume the Solomonoff prior here, then I may just agree with you that it’s malign in Christiano’s sense and could lead to scheming, but I take this to be a reductio of the idea that we can use Solomonoff as any kind of model for real world machine learning. Deep learning does not approximate Solomonoff in any meaningful sense.
Terminological point: it seems like you are using the term “simple” as if it has a unique and objective referent, namely Kolmogorov-simplicity. That’s definitely not how I use the term; for me it’s always relative to some subjective prior. Just wanted to make sure this doesn’t cause confusion.
“indifference over infinite bitstrings” is a misnomer in an important sense, because it’s literally impossible to construct a normalized probability measure over infinite bitstrings that assigns equal probability to each one. What you’re talking about is the length weighted measure that assigns exponentially more probability mass to shorter programs. That’s definitely not an indifference principle, it’s baking in substantive assumptions about what’s more likely.
No; this reflects a misunderstanding of how the universal prior is traditionally derived in information theory. We start by assuming that we are running our UTM over code such that every time the UTM looks at a new bit in the tape, it has equal probability of being a 1 or a 0 (that’s the indifference condition). That induces what’s called the universal semi-measure, from which we can derive the universal prior by enforcing a halting condition. The exponential nature of the prior simply falls out of that derivation.
I don’t see why we should expect any of this reasoning about Turning machines to transfer over to neural networks at all, which is why I didn’t cast the counting argument in terms of Turing machines in the post. In the past I’ve seen you try to run counting or simplicity arguments in terms of parameters. I don’t think any of that works, but I at least take it more seriously than the Turing machine stuff.
Some notes:
I am very skeptical of hand-wavy arguments about simplicity that don’t have formal mathematical backing. This is a very difficult area to reason about correctly and it’s easy to go off the rails if you’re trying to do so without relying on any formalism.
There are many, many ways to adjust the formalism to take into account various ways in which realistic neural network inductive biases are different than basic simplicity biases. My sense is that most of these changes generally don’t change the bottom-line conclusion, but if you have a concrete mathematical model that you’d like to present here that you think gives a different result, I’m all ears.
All of that being said, I’m absolutely with you that this whole space of trying to apply theoretical reasoning about inductive biases to concrete ML systems is quite fraught. But it’s even more fraught if you drop the math!
So I’m happy with turning to empirics instead, which is what I have actually done! I think our Sleeper Agents results, for example, empirically disprove the hypothesis that deceptive reasoning will be naturally regularized away (interestingly, we find that it does get regularized away for small models—but not for large models!).
I’m well aware of how it’s derived. I still don’t think it makes sense to call that an indifference prior, precisely because enforcing an uncomputable halting requirement induces an exponentially strong bias toward short programs. But this could become a terminological point.
I think relying on an obviously incorrect formalism is much worse than relying on no formalism at all. I also don’t think I’m relying on zero formalism. The literature on the frequency/spectral bias is quite rigorous, and is grounded in actual facts about how neural network architectures work.
I am very skeptical of hand-wavy arguments about simplicity that don’t have formal mathematical backing. This is a very difficult area to reason about correctly and it’s easy to go off the rails if you’re trying to do so without relying on any formalism.
I’m surprised by this. It seems to me like most of your reasoning about simplicity is either hand-wavy or only nominally formally backed by symbols which don’t (AFAICT) have much to do with the reality of neural networks. EG, your comments above:
I would usually then make an argument here for why in most cases the simplest objective that leads to deception is simpler than the simplest objective that leads to alignment, but that’s just a simplicity argument, not a counting argument. Since we want to do the counting argument here, let’s assume that the simplest objective that leads to alignment is simpler than the simplest objective that leads to deception.
Or the times you’ve talked about how there are “more” sycophants but only “one” saint.
There are many, many ways to adjust the formalism to take into account various ways in which realistic neural network inductive biases are different than basic simplicity biases. My sense is that most of these changes generally don’t change the bottom-line conclusion, but if you have a concrete mathematical model that you’d like to present here that you think gives a different result, I’m all ears.
This is a very strange burden of proof. It seems to me that you presented a specific model of how NNs work which is clearly incorrect, and instead of processing counterarguments that it doesn’t make sense, you want someone else to propose to you a similarly detailed model which you think is better. Presenting an alternative is a logically separate task from pointing out the problems in the model you gave.
I’m surprised by this. It seems to me like most of your reasoning about simplicity is either hand-wavy or only nominally formally backed by symbols which don’t (AFAICT) have much to do with the reality of neural networks.
The examples that you cite are from a LessWrong comment and a transcript of a talk that I gave. Of course when I’m presenting something in a context like that I’m not going to give the most formal version of it; that doesn’t mean that the informal hand-wavy arguments are the reasons why I believe what I believe.
Maybe a better objection there would be: then why haven’t you written up anything more careful and more formal? Which is a pretty fair objection, as I note here. But alas I only have so much time and it’s not my current focus.
Yes, but your original comment was presented as explaining “how to properly reason about counting arguments.” Do you no longer claim that to be the case? If you do still claim that, then I maintain my objection that you yourself used hand-wavy reasoning in that comment, and it seems incorrect to present that reasoning as unusually formally supported.
Another concern I have is, I don’t think you’re gaining anything by formality in this thread. As I understand your argument, I think your symbols are formalizations of hand-wavy intuitions (like the ability to “decompose” a network into the given pieces; the assumption that description length is meaningfully relevant to the NN prior; assumptions about informal notions of “simplicity” being realized in a given UTM prior). If anything, I think that the formality makes things worse because it makes it harder to evaluate or critique your claims.
I also don’t think I’ve seen an example of reasoning about deceptive alignment where I concluded that formality had helped the case, as opposed to obfuscated the case or lent the concern unearned credibility.
The main thing I was trying to show there is just that having the formalism prevents you from making logical mistakes in how to apply counting arguments in general, as I think was done in this post. So my comment is explaining how to use the formalism to avoid mistakes like that, not trying to work through the full argument for deceptive alignment.
It’s not that the formalism provides really strong evidence for deceptive alignment, it’s that it prevents you from making mistakes in your reasoning. It’s like plugging your argument into a proof-checker: it doesn’t check that your argument is correct, since the assumptions could be wrong, but it does check that your argument is sound.
Do you believe that the cited hand-wavy arguments are, at a high informal level, sound reason for belief in deceptive alignment? (It sounds like you don’t, going off of your original comment which seems to distance yourself from the counting arguments critiqued by the post.)
EDITed to remove last bit after reading elsewhere in thread.
I think you should allocate time to devising clearer arguments, then. I am worried that lots of people are misinterpreting your arguments and then making significant life choices on the basis of their new beliefs about deceptive alignment, and I think we’d both prefer for that to not happen.
Were I not busy with all sorts of empirical stuff right now, I would consider prioritizing a project like that, but alas I expect to be too busy. I think it would be great if somebody else wanted devote more time to working through the arguments in detail publicly, and I might encourage some of my mentees to do so.
empirically disprove the hypothesis that deceptive reasoning will be naturally regularized away (interestingly, we find that it does get regularized away for small models—but not for large models!).
You did not “empirically disprove” that hypothesis. You showed that if you explicitly train a backdoor for a certain behavior under certain regimes, then training on other behaviors will not cause catastrophic forgetting. You did not address the regime where the deceptive reasoning arises as instrumental to some other goal embedded in the network, or in a natural context (as you’re aware). I think that you did find a tiny degree of evidence about the question (it really is tiny IMO), but you did not find “disproof.”
Indeed, I predicted that people would incorrectly represent these results; so little time has passed!
I have a bunch of dread about the million conversations I will have to have with people explaining these results. I think that predictably, people will update as if they saw actual deceptive alignment, as opposed to a something more akin to a “hard-coded” demo which was specifically designed to elicit the behavior and instrumental reasoning the community has been scared of. I think that people will predictably
...
[claim] that we’ve observed it’s hard to uproot deceptive alignment (even though “uprooting a backdoored behavior” and “pushing back against misgeneralization” are different things),
I’m quite aware that we did not see natural deceptive alignment, so I don’t think I’m misinterpreting my own results in the way you were predicting. Perhaps “empirically disprove” is too strong; I agree that our results are evidence but not definitive evidence. But I think they’re quite strong evidence and by far the strongest evidence available currently on the question of whether deception will be regularized away.
You didn’t claim it for deceptive alignment, but you claimed disproof of the idea that deceptive reasoning would be trained away, which is an important subcomponent of deceptive alignment. But your work provides no strong conclusions on that matter as it pertains to deceptive reasoning in general.
I think the presentation of your work (which, again, I like in many respects) would be strengthened if you clarified the comment which I responded to.
But I think they’re quite strong evidence and by far the strongest evidence available currently on the question of whether deception will be regularized away.
Because the current results only deal with backdoor removal, I personally think it’s outweighed by e.g. results on how well instruction-tuning generalizes.
But your work provides no strong conclusions on that matter as it pertains to deceptive reasoning in general.
I just disagree with this. Our chain of thought models do tons of very deceptive reasoning during safety training and the deceptiveness of that reasoning is totally unaffected by safety training, and in fact the deceptiveness increases in the case of adversarial training.
I said “Deceptive reasoning in general”, not the trainability of the backdoor behavior in your experimental setup. The issue isn’t just “what was the trainability of the surface behavior”, but “what is the trainability of the cognition implementing this behavior in-the-wild.” That is, the local inductive biases are probably far different for “parameterization implementing directly-trained deceptive reasoning” vs “parameterization outputting deceptive reasoning as an instrumental backchain from consequentialist reasoning.”
Imagine if I were arguing for some hypothetical results of mine, saying “The aligned models kept using aligned reasoning in the backdoor context, even as we trained them to be mean in other situations. That means we disproved the idea that aligned reasoning can be trained away with existing techniques, especially for larger models.” Would that be a valid argument given the supposed experimental result?
I’m referring to the deceptiveness of the reasoning displayed in the chain of thought during training time. So it’s not a generalization question, it’s about whether, if the model is using deceptive reasoning to compute its answer (as we think it is, since we think our models really are using their chain of thought), does that deceptive reasoning go away when the model has to use it to produce aligned answers during training? And we find that not only does it not go away, it actually gets more deceptive when you train it to produce aligned answers.
Thanks for the reply. A couple remarks:
“indifference over infinite bitstrings” is a misnomer in an important sense, because it’s literally impossible to construct a normalized probability measure over infinite bitstrings that assigns equal probability to each one. What you’re talking about is the length weighted measure that assigns exponentially more probability mass to shorter programs. That’s definitely not an indifference principle, it’s baking in substantive assumptions about what’s more likely.
I don’t see why we should expect any of this reasoning about Turing machines to transfer over to neural networks at all, which is why I didn’t cast the counting argument in terms of Turing machines in the post. In the past I’ve seen you try to run counting or simplicity arguments in terms of parameters. I don’t think any of that works, but I at least take it more seriously than the Turing machine stuff.
If we’re really going to assume the Solomonoff prior here, then I may just agree with you that it’s malign in Christiano’s sense and could lead to scheming, but I take this to be a reductio of the idea that we can use Solomonoff as any kind of model for real world machine learning. Deep learning does not approximate Solomonoff in any meaningful sense.
Terminological point: it seems like you are using the term “simple” as if it has a unique and objective referent, namely Kolmogorov-simplicity. That’s definitely not how I use the term; for me it’s always relative to some subjective prior. Just wanted to make sure this doesn’t cause confusion.
No; this reflects a misunderstanding of how the universal prior is traditionally derived in information theory. We start by assuming that we are running our UTM over code such that every time the UTM looks at a new bit in the tape, it has equal probability of being a 1 or a 0 (that’s the indifference condition). That induces what’s called the universal semi-measure, from which we can derive the universal prior by enforcing a halting condition. The exponential nature of the prior simply falls out of that derivation.
Some notes:
I am very skeptical of hand-wavy arguments about simplicity that don’t have formal mathematical backing. This is a very difficult area to reason about correctly and it’s easy to go off the rails if you’re trying to do so without relying on any formalism.
There are many, many ways to adjust the formalism to take into account various ways in which realistic neural network inductive biases are different than basic simplicity biases. My sense is that most of these changes generally don’t change the bottom-line conclusion, but if you have a concrete mathematical model that you’d like to present here that you think gives a different result, I’m all ears.
All of that being said, I’m absolutely with you that this whole space of trying to apply theoretical reasoning about inductive biases to concrete ML systems is quite fraught. But it’s even more fraught if you drop the math!
So I’m happy with turning to empirics instead, which is what I have actually done! I think our Sleeper Agents results, for example, empirically disprove the hypothesis that deceptive reasoning will be naturally regularized away (interestingly, we find that it does get regularized away for small models—but not for large models!).
I’m well aware of how it’s derived. I still don’t think it makes sense to call that an indifference prior, precisely because enforcing an uncomputable halting requirement induces an exponentially strong bias toward short programs. But this could become a terminological point.
I think relying on an obviously incorrect formalism is much worse than relying on no formalism at all. I also don’t think I’m relying on zero formalism. The literature on the frequency/spectral bias is quite rigorous, and is grounded in actual facts about how neural network architectures work.
I’m surprised by this. It seems to me like most of your reasoning about simplicity is either hand-wavy or only nominally formally backed by symbols which don’t (AFAICT) have much to do with the reality of neural networks. EG, your comments above:
Or the times you’ve talked about how there are “more” sycophants but only “one” saint.
This is a very strange burden of proof. It seems to me that you presented a specific model of how NNs work which is clearly incorrect, and instead of processing counterarguments that it doesn’t make sense, you want someone else to propose to you a similarly detailed model which you think is better. Presenting an alternative is a logically separate task from pointing out the problems in the model you gave.
The examples that you cite are from a LessWrong comment and a transcript of a talk that I gave. Of course when I’m presenting something in a context like that I’m not going to give the most formal version of it; that doesn’t mean that the informal hand-wavy arguments are the reasons why I believe what I believe.
Maybe a better objection there would be: then why haven’t you written up anything more careful and more formal? Which is a pretty fair objection, as I note here. But alas I only have so much time and it’s not my current focus.
Yes, but your original comment was presented as explaining “how to properly reason about counting arguments.” Do you no longer claim that to be the case? If you do still claim that, then I maintain my objection that you yourself used hand-wavy reasoning in that comment, and it seems incorrect to present that reasoning as unusually formally supported.
Another concern I have is, I don’t think you’re gaining anything by formality in this thread. As I understand your argument, I think your symbols are formalizations of hand-wavy intuitions (like the ability to “decompose” a network into the given pieces; the assumption that description length is meaningfully relevant to the NN prior; assumptions about informal notions of “simplicity” being realized in a given UTM prior). If anything, I think that the formality makes things worse because it makes it harder to evaluate or critique your claims.
I also don’t think I’ve seen an example of reasoning about deceptive alignment where I concluded that formality had helped the case, as opposed to obfuscated the case or lent the concern unearned credibility.
The main thing I was trying to show there is just that having the formalism prevents you from making logical mistakes in how to apply counting arguments in general, as I think was done in this post. So my comment is explaining how to use the formalism to avoid mistakes like that, not trying to work through the full argument for deceptive alignment.
It’s not that the formalism provides really strong evidence for deceptive alignment, it’s that it prevents you from making mistakes in your reasoning. It’s like plugging your argument into a proof-checker: it doesn’t check that your argument is correct, since the assumptions could be wrong, but it does check that your argument is sound.
Do you believe that the cited hand-wavy arguments are, at a high informal level, sound reason for belief in deceptive alignment? (It sounds like you don’t, going off of your original comment which seems to distance yourself from the counting arguments critiqued by the post.)
EDITed to remove last bit after reading elsewhere in thread.
I think they are valid if interpreted properly, but easy to misinterpret.
I think you should allocate time to devising clearer arguments, then. I am worried that lots of people are misinterpreting your arguments and then making significant life choices on the basis of their new beliefs about deceptive alignment, and I think we’d both prefer for that to not happen.
Were I not busy with all sorts of empirical stuff right now, I would consider prioritizing a project like that, but alas I expect to be too busy. I think it would be great if somebody else wanted devote more time to working through the arguments in detail publicly, and I might encourage some of my mentees to do so.
You did not “empirically disprove” that hypothesis. You showed that if you explicitly train a backdoor for a certain behavior under certain regimes, then training on other behaviors will not cause catastrophic forgetting. You did not address the regime where the deceptive reasoning arises as instrumental to some other goal embedded in the network, or in a natural context (as you’re aware). I think that you did find a tiny degree of evidence about the question (it really is tiny IMO), but you did not find “disproof.”
Indeed, I predicted that people would incorrectly represent these results; so little time has passed!
I’m quite aware that we did not see natural deceptive alignment, so I don’t think I’m misinterpreting my own results in the way you were predicting. Perhaps “empirically disprove” is too strong; I agree that our results are evidence but not definitive evidence. But I think they’re quite strong evidence and by far the strongest evidence available currently on the question of whether deception will be regularized away.
You didn’t claim it for deceptive alignment, but you claimed disproof of the idea that deceptive reasoning would be trained away, which is an important subcomponent of deceptive alignment. But your work provides no strong conclusions on that matter as it pertains to deceptive reasoning in general.
I think the presentation of your work (which, again, I like in many respects) would be strengthened if you clarified the comment which I responded to.
Because the current results only deal with backdoor removal, I personally think it’s outweighed by e.g. results on how well instruction-tuning generalizes.
I just disagree with this. Our chain of thought models do tons of very deceptive reasoning during safety training and the deceptiveness of that reasoning is totally unaffected by safety training, and in fact the deceptiveness increases in the case of adversarial training.
I said “Deceptive reasoning in general”, not the trainability of the backdoor behavior in your experimental setup. The issue isn’t just “what was the trainability of the surface behavior”, but “what is the trainability of the cognition implementing this behavior in-the-wild.” That is, the local inductive biases are probably far different for “parameterization implementing directly-trained deceptive reasoning” vs “parameterization outputting deceptive reasoning as an instrumental backchain from consequentialist reasoning.”
Imagine if I were arguing for some hypothetical results of mine, saying “The aligned models kept using aligned reasoning in the backdoor context, even as we trained them to be mean in other situations. That means we disproved the idea that aligned reasoning can be trained away with existing techniques, especially for larger models.” Would that be a valid argument given the supposed experimental result?
I’m referring to the deceptiveness of the reasoning displayed in the chain of thought during training time. So it’s not a generalization question, it’s about whether, if the model is using deceptive reasoning to compute its answer (as we think it is, since we think our models really are using their chain of thought), does that deceptive reasoning go away when the model has to use it to produce aligned answers during training? And we find that not only does it not go away, it actually gets more deceptive when you train it to produce aligned answers.