I’ll think more about the outer shell stuff, it’s possible that my objection actually arises with the consequentialist assumption, but I’m not sure.
It’s on my todo list to write a comment responding to some of the specifics of Redwood’s control post.
I would be excited to read this / help with a draft.
Yes, approximately, as I believe you and I are capable of doing.
The analogy is strained due to not being able to gradient update my brain with arbitrary training data. It’s pretty unclear to me if I could pull off deception like this, it seems pretty hard imo.
Seems easy enough to predict given roughly human-scientist-team level of capabilities.
One situation I’m imagining here is that we’ve (explicitly) trained our AI on scientific output from January to March, and we’re now telling it to do June-level research. It does not know if we have labeled data for just up to March, for April, or all the way to June. I think if you took a human research team, stuck them on an island, and had them get up to speed in some subfield they’re not part of, they would be unable to distinguish between research that had just been done on the mainland or is just past the frontier. This just seems pretty hard. I think this example is closer to what we’ll get with AIs rather than the bar being at “human level” it’s more like “what has been accomplished by the humans+AIs in the outside world, for which I’ve received untrustworthy information”. And every time the AIs do useful work we can add this to our supervised training corpus.
> The data need not be human generated, it could be generated by other AIs, as long as we trust it.
?? This seems to be assuming a solution to the problem.
One version of this is: you have GPT-5 (potentially untrustworthy) do some research, you have humans carefully check this research using the best available tools (e.g., AI debate / other ‘scalable oversight’), and then you use this research as supervised / labeled data for training GPT-6 to do good work.
Latent adversarial training seems like it could make an agent robust to a few extra categories of internal perturbation, but doesn’t seem like it would help with any problem that we were focused on in this post.
Here’s a specific thing I mean: we can look at activations and use these to aid in the training process, even without a full understanding of what activations mean. Take this recent paper (which I think is super confused but helps get at the thing), “We find a strong positive relationship between the LLM’s attention to constraint tokens and the factual accuracy of generations”. My interpretation is that they found a probably-causal link between ‘the model is attending to relevant tokens’ and ‘the model gets the answer correct’, because duh, it’s like how when you give somebody a riddle you can tell whether they’re on track based on what parts of the question they’re focused on. While it’s probably non-trivial, I expect you could use this internal activation to train models which do a better job paying attention to the tokens you want them to. Similarly, I think it’s probable (but far from guaranteed) that deceptively aligned models are representing training/deployment in their activations, or that sandbagging models are representing human-level / super-human-level difficulty for tasks. If so, it it possible in principle to do all kinds of things with these activations which may help us. e.g., get the model to solve tasks it otherwise would have intentionally failed at, or giving us hints about the accuracy of a research result. I expect this is difficult implementation wise, but still a reason for hope. IMO activation engineering is off to a good start.
“leaning more on generalization” ≈ “leaning more on agency”,
That’s not why I meant to say. By leaning more on generalization, I mean leaning more on the data efficiency thing, and also weak-to-strong generalization ideas. I think this is different from “leaning more on agency”.
The problem is shaping the goal to be a specific thing, not changing it in unpredictable or not-fully-understood ways.
I’m not so sure of this. I could imagine starting with a deceptively aligned AI whose goal is “Make paperclips unless being supervised which is defined as X, Y, and Z, in which case look good to humans”. And if we could change this AI to have the goal “Make paperclips unless being supervised which is defined as X, Y, and Q, in which case look good to humans”, that might be highly desirable. In particular, it seems like adversarial training here allows us to expand the definition of ‘supervision’, thus making it easier to elicit good work from AIs (ideally not just ‘looks good’).
I think that there is a lot of room for the evidence to be ambiguous and controversial
I agree that this is a major potential problem which I think is pretty likely to occur; maybe I’ll think/write more about it soon. I think I brought this up in response to some of the Redwood stuff and got a response mentioning that we can play things forward through our oversight scheme, and get clearer answers to this question. I think this remains an important concern, but it’s also true that we’ll have the ability to actually run some outputs through to e.g., see if they trigger oversight AIs to mess up, see if a vulnerability would have been successfully inserted in production, etc. It might also end up being very expensive to do this for everything that seems like a possible coup attempt, idk.
I buy this for many environments and levels of power, but it’s obviously wrong for AIs that have taking-over-the-universe in their easily-within-reach options. And that’s the sort of AI we get if it can undergo self-improvement.
I don’t feel like I’m talking about AIs which have “taking-over-the-universe in their easily-within-reach options”. I think this is not within reach of the current employees of AGI labs, and the AIs I’m thinking of are similar to those employees in terms of capabilities, but perhaps a bit smarter, much faster, and under some really weird/strict constraints (control schemes).
I would be excited to read this / help with a draft.
We can meet in person one afternoon and work out some cruxes and write them up?
Is the claim here that the AI performs well on ~all the human-checkable tasks and then reverts to being absolutely useless or sabotaging on the hard stuff?
Yes, approximately, as I believe you and I are capable of doing. [...PhD student crux]
The analogy is strained due to not being able to gradient update my brain with arbitrary training data. It’s pretty unclear to me if I could pull off deception like this, it seems pretty hard imo.
Are you saying you buy my side of the PhD student crux, but that it isn’t closely analogous because you think we have more fine-grained control over the thoughts or goals of the AI? And in your second sentence my guess is that you are saying a second reason you disagree is that it looks difficult for a research AI to know when to perform well and when to perform less well? I think we might be miscommunicating about this because it doesn’t look like the sort of situation where the “deception” is a relatively difficult task.
I think if you took a human research team, stuck them on an island, and had them get up to speed in some subfield they’re not part of, they would be unable to distinguish between research that had just been done on the mainland or is just past the frontier. This just seems pretty hard.
I agree with this. But the research we need isn’t just past the frontier. At some point not that far past the frontier, you get to a situation where the research team can say “given our guesses about mainland researchers and their time constraints, we’d put <1% on them having made discovery X {or explored idea Y, or done experiment Z}. Let’s {delay giving it to them, not work on that direction, insert a subtle false assumption, instead share our 8th best idea, just say we’re stuck, communicate it poorly}”.
I think this example is closer to what we’ll get with AIs rather than the bar being at “human level” it’s more like “what has been accomplished by the humans+AIs in the outside world, for which I’ve received untrustworthy information”.
Sure, kinda, but the important part here is that a trapped agentic AI can selectively mislead you about the most important {knowledge, tasks, strategies} that you aren’t capable of doing with just humans. If you’re capable of doing it with humans, or if you’re close to being able to do it with humans, then why not just do that? Seems safer.
And every time the AIs do useful work we can add this to our supervised training corpus.
Only if you spend the human resources to verify it. Are you aiming for 30x research acceleration? Because this by itself might be enough limit you below that level. (and it also directly provides a way for the AI to tell “what time it is” by sending messages to future iterations, and plausibly also provide a bunch of information about human research capability and capacity).
One version of this is: you have GPT-5 (potentially untrustworthy) do some research, you have humans carefully check this research using the best available tools (e.g., AI debate / other ‘scalable oversight’), and then you use this research as supervised / labeled data for training GPT-6 to do good work.
Mostly my objection to this is above. On scalable oversight, I buy that it might be possible to get 20% improvements in the accuracy of your evaluations of (adversarialy generated!) research, compared to team of humans trying really hard to evaluate research, given a human level research AI. This is so far from being enough. Are you predicting much better scalable oversight that scales much further than I am predicting it will? (I’m assuming you’re using “GPT-5/6” to represent “some powerful future consequentialist AI that can do research”, because otherwise we should be having a different argument).
While it’s probably non-trivial, I expect you could use this internal activation to train models which do a better job paying attention to the tokens you want them to.
It’d be cool if there were a demonstration of manipulating internal activations during training, in a way that fixes non-trivial unwanted OOD behavior. If such techniques existed, and also worked on consequentialist agents (I expect would be harder), I would consider them very useful tools that would help with building an aligned agent.
While it’s probably non-trivial, I expect you could use this internal activation to train models which do a better job paying attention to the tokens you want them to.
This is very plausible, but I think you’re stretching the analogy between LMs and future research AIs way too far. “Paying attention to the tokens we want them to” doesn’t add much/any control over the properties we want control over (compared to e.g. changing the input string to remove information we don’t want it paying attention to).
Similarly, I think it’s probable (but far from guaranteed) that deceptively aligned models are representing training/deployment in their activations, or that sandbagging models are representing human-level / super-human-level difficulty for tasks. If so, it it possible in principle to do all kinds of things with these activations which may help us. e.g., get the model to solve tasks it otherwise would have intentionally failed at, or giving us hints about the accuracy of a research result. I expect this is difficult implementation wise, but still a reason for hope. IMO activation engineering is off to a good start.
I think you might be centrally thinking about an LLM completing a few paragraphs, where I’m thinking of an AI doing a large scale scientific project? I buy that for many concepts X, you can work out the “neural correlates of X” in your AI. But that kind of thing is noisy (and plausibly can be optimized against by the AI) unless you have a deep understanding of what you are measuring. And optimizing against such imperfect metrics obviously wouldn’t do much beyond destroying the metric. I do think research in this direction has a chance of being useful, but mainly by being upstream of much better understanding.
By leaning more on generalization, I mean leaning more on the data efficiency thing
Sorry for misinterpreting you, but this doesn’t clarify what you meant.
also weak-to-strong generalization ideas.
I think I don’t buy the analogy in that paper, and I don’t find the results surprising or relevant (by my current understanding, after skimming it). My understanding of the result is “if you have a great prior, you can use it to overcome some label noise and maybe also label bias”. But I don’t think this is very relevant to extracting useful work from a misaligned agent (which is what we are talking about here), and based on the assumptions they describe, I think they agree? (I just saw appendix G, I’m a fan of it, it’s really valuable that they explained their alignment plan concisely and listed their assumptions).
I could imagine starting with a deceptively aligned AI whose goal is “Make paperclips unless being supervised which is defined as X, Y, and Z, in which case look good to humans”. And if we could change this AI to have the goal “Make paperclips unless being supervised which is defined as X, Y, and Q, in which case look good to humans”, that might be highly desirable. In particular, it seems like adversarial training here allows us to expand the definition of ‘supervision’, thus making it easier to elicit good work from AIs (ideally not just ‘looks good’).
If we can tell we are have such an AI, and we can tell that our random modifications are affecting the goal, and also the change is roughly one that helps us rather than changing many things that might or might not be helpful, this would be a nice situation to be in.
I don’t feel like I’m talking about AIs which have “taking-over-the-universe in their easily-within-reach options”. I think this is not within reach of the current employees of AGI labs, and the AIs I’m thinking of are similar to those employees in terms of capabilities, but perhaps a bit smarter, much faster, and under some really weird/strict constraints (control schemes).
Section 6 assumes we have failed to control the AI, so it is free of weird/strict constraints, and free to scale itself up, improve itself, etc. So my comment is about an AI that no longer can be assumed to have human-ish capabilities.
Thanks for your response!
I’ll think more about the outer shell stuff, it’s possible that my objection actually arises with the consequentialist assumption, but I’m not sure.
I would be excited to read this / help with a draft.
The analogy is strained due to not being able to gradient update my brain with arbitrary training data. It’s pretty unclear to me if I could pull off deception like this, it seems pretty hard imo.
One situation I’m imagining here is that we’ve (explicitly) trained our AI on scientific output from January to March, and we’re now telling it to do June-level research. It does not know if we have labeled data for just up to March, for April, or all the way to June. I think if you took a human research team, stuck them on an island, and had them get up to speed in some subfield they’re not part of, they would be unable to distinguish between research that had just been done on the mainland or is just past the frontier. This just seems pretty hard. I think this example is closer to what we’ll get with AIs rather than the bar being at “human level” it’s more like “what has been accomplished by the humans+AIs in the outside world, for which I’ve received untrustworthy information”. And every time the AIs do useful work we can add this to our supervised training corpus.
One version of this is: you have GPT-5 (potentially untrustworthy) do some research, you have humans carefully check this research using the best available tools (e.g., AI debate / other ‘scalable oversight’), and then you use this research as supervised / labeled data for training GPT-6 to do good work.
Here’s a specific thing I mean: we can look at activations and use these to aid in the training process, even without a full understanding of what activations mean. Take this recent paper (which I think is super confused but helps get at the thing), “We find a strong positive relationship between the LLM’s attention to constraint tokens and the factual accuracy of generations”. My interpretation is that they found a probably-causal link between ‘the model is attending to relevant tokens’ and ‘the model gets the answer correct’, because duh, it’s like how when you give somebody a riddle you can tell whether they’re on track based on what parts of the question they’re focused on. While it’s probably non-trivial, I expect you could use this internal activation to train models which do a better job paying attention to the tokens you want them to. Similarly, I think it’s probable (but far from guaranteed) that deceptively aligned models are representing training/deployment in their activations, or that sandbagging models are representing human-level / super-human-level difficulty for tasks. If so, it it possible in principle to do all kinds of things with these activations which may help us. e.g., get the model to solve tasks it otherwise would have intentionally failed at, or giving us hints about the accuracy of a research result. I expect this is difficult implementation wise, but still a reason for hope. IMO activation engineering is off to a good start.
That’s not why I meant to say. By leaning more on generalization, I mean leaning more on the data efficiency thing, and also weak-to-strong generalization ideas. I think this is different from “leaning more on agency”.
I’m not so sure of this. I could imagine starting with a deceptively aligned AI whose goal is “Make paperclips unless being supervised which is defined as X, Y, and Z, in which case look good to humans”. And if we could change this AI to have the goal “Make paperclips unless being supervised which is defined as X, Y, and Q, in which case look good to humans”, that might be highly desirable. In particular, it seems like adversarial training here allows us to expand the definition of ‘supervision’, thus making it easier to elicit good work from AIs (ideally not just ‘looks good’).
I agree that this is a major potential problem which I think is pretty likely to occur; maybe I’ll think/write more about it soon. I think I brought this up in response to some of the Redwood stuff and got a response mentioning that we can play things forward through our oversight scheme, and get clearer answers to this question. I think this remains an important concern, but it’s also true that we’ll have the ability to actually run some outputs through to e.g., see if they trigger oversight AIs to mess up, see if a vulnerability would have been successfully inserted in production, etc. It might also end up being very expensive to do this for everything that seems like a possible coup attempt, idk.
I don’t feel like I’m talking about AIs which have “taking-over-the-universe in their easily-within-reach options”. I think this is not within reach of the current employees of AGI labs, and the AIs I’m thinking of are similar to those employees in terms of capabilities, but perhaps a bit smarter, much faster, and under some really weird/strict constraints (control schemes).
We can meet in person one afternoon and work out some cruxes and write them up?
Are you saying you buy my side of the PhD student crux, but that it isn’t closely analogous because you think we have more fine-grained control over the thoughts or goals of the AI? And in your second sentence my guess is that you are saying a second reason you disagree is that it looks difficult for a research AI to know when to perform well and when to perform less well? I think we might be miscommunicating about this because it doesn’t look like the sort of situation where the “deception” is a relatively difficult task.
I agree with this. But the research we need isn’t just past the frontier. At some point not that far past the frontier, you get to a situation where the research team can say “given our guesses about mainland researchers and their time constraints, we’d put <1% on them having made discovery X {or explored idea Y, or done experiment Z}. Let’s {delay giving it to them, not work on that direction, insert a subtle false assumption, instead share our 8th best idea, just say we’re stuck, communicate it poorly}”.
Sure, kinda, but the important part here is that a trapped agentic AI can selectively mislead you about the most important {knowledge, tasks, strategies} that you aren’t capable of doing with just humans. If you’re capable of doing it with humans, or if you’re close to being able to do it with humans, then why not just do that? Seems safer.
Only if you spend the human resources to verify it. Are you aiming for 30x research acceleration? Because this by itself might be enough limit you below that level. (and it also directly provides a way for the AI to tell “what time it is” by sending messages to future iterations, and plausibly also provide a bunch of information about human research capability and capacity).
Mostly my objection to this is above. On scalable oversight, I buy that it might be possible to get 20% improvements in the accuracy of your evaluations of (adversarialy generated!) research, compared to team of humans trying really hard to evaluate research, given a human level research AI. This is so far from being enough. Are you predicting much better scalable oversight that scales much further than I am predicting it will? (I’m assuming you’re using “GPT-5/6” to represent “some powerful future consequentialist AI that can do research”, because otherwise we should be having a different argument).
It’d be cool if there were a demonstration of manipulating internal activations during training, in a way that fixes non-trivial unwanted OOD behavior. If such techniques existed, and also worked on consequentialist agents (I expect would be harder), I would consider them very useful tools that would help with building an aligned agent.
This is very plausible, but I think you’re stretching the analogy between LMs and future research AIs way too far. “Paying attention to the tokens we want them to” doesn’t add much/any control over the properties we want control over (compared to e.g. changing the input string to remove information we don’t want it paying attention to).
I think you might be centrally thinking about an LLM completing a few paragraphs, where I’m thinking of an AI doing a large scale scientific project? I buy that for many concepts X, you can work out the “neural correlates of X” in your AI. But that kind of thing is noisy (and plausibly can be optimized against by the AI) unless you have a deep understanding of what you are measuring. And optimizing against such imperfect metrics obviously wouldn’t do much beyond destroying the metric. I do think research in this direction has a chance of being useful, but mainly by being upstream of much better understanding.
Sorry for misinterpreting you, but this doesn’t clarify what you meant.
I think I don’t buy the analogy in that paper, and I don’t find the results surprising or relevant (by my current understanding, after skimming it). My understanding of the result is “if you have a great prior, you can use it to overcome some label noise and maybe also label bias”. But I don’t think this is very relevant to extracting useful work from a misaligned agent (which is what we are talking about here), and based on the assumptions they describe, I think they agree? (I just saw appendix G, I’m a fan of it, it’s really valuable that they explained their alignment plan concisely and listed their assumptions).
If we can tell we are have such an AI, and we can tell that our random modifications are affecting the goal, and also the change is roughly one that helps us rather than changing many things that might or might not be helpful, this would be a nice situation to be in.
Section 6 assumes we have failed to control the AI, so it is free of weird/strict constraints, and free to scale itself up, improve itself, etc. So my comment is about an AI that no longer can be assumed to have human-ish capabilities.
There are enough open threads that I think we’re better off continuing this conversation in person. Thanks for your continued engagement.