This is Section 1.5 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.
Audio version of this section here, or search “Joe Carlsmith Audio” on your podcast app.
On “slack” in training
Before diving into an assessment of the arguments for expecting
scheming, I also want to flag a factor that will come up repeatedly in
what follows: namely, the degree of “slack” that we should expect
training to allow. By this I mean something like: how much is the
training process ruthlessly and relentlessly pressuring the model to
perform in a manner that yields maximum reward, vs. shaping the model in
a more “relaxed” way, that leaves more room for less-than-maximally
rewarded behavior. That is, in a low-slack regime, “but that sort of
model would be getting less reward than would be possible given its
capabilities” is a strong counterargument against training creating a
model of the relevant kind, whereas in a high-slack regime, it’s not (so
high slack regimes will generally involve greater uncertainty about the
type of model you end up with, since models that get less-than-maximal
reward are still in the running).
Or, in more human terms: a low-slack regime is more like a hyper-intense
financial firm that immediately fires any employees who fall behind in
generating profits (and where you’d therefore expect surviving employees
to be hyper-focused on generating profits – or perhaps, hyper-focused on
the profits that their supervisors think they’re generating), whereas
a high-slack regime is more like a firm where employees can freely goof
off, drink martinis at lunch, and pursue projects only vaguely related
to the company’s bottom line, and who only need to generate some
amount of profit for the firm sometimes.
(Or at least, that’s the broad distinction I’m trying to point at.
Unfortunately, I don’t have a great way of making it much more precise,
and I think it’s possible that thinking in these terms will ultimately
be misleading.)
Slack matters here partly because below, I’m going to be making various
arguments that appeal to possibly-quite-small differences in the amount
of reward that different models will get. And the force of these
arguments depends on how sensitive training is to these differences. But
I also think it can inform our sense of what models to expect more
generally.
For example, I think slack matters to the probability that training will
create models that pursue proxy goals imperfectly correlated with reward
on the training inputs. Thus, in a low-slack regime, it may be fairly
unlikely for a model trained to help humans with science to end up
pursuing a general “curiosity drive” (in a manner that doesn’t then
motivate instrumental training-gaming), because a model’s pursuing its
curiosity in training would sometimes deviate from maximally
helping-the-humans-with-science.
That said, note that the degree of slack is conceptually distinct from
the diversity and robustness of the efforts made in training to root out
goal misgeneralization. Thus, for example, if you’re rewarding a model
when it gets gold coins, but you only ever show your model environments
where the only gold things are coins, then a model that tries to get
gold-stuff-in-general will perform just as well a model that gets gold
coins in particular, regardless of how intensely training pressures the
model to get maximum reward on those environments. E.g., a low-slack
regime could in principle select either of these models, whereas a
high-slack regime would leave more room for models that just get fewer
gold coins period (for example, models that sometimes pursue red things
instead, or who waste lots of time thinking before they pursue their
gold coins).
In this sense, a low-slack regime doesn’t speak all that strongly
against mis-generalized non-training-gamers. Rather, it speaks against
models that aren’t pursuing what I’ll call a “max reward goal
target” – that is, a goal target very closely correlated with reward on
the inputs the model in fact receives in training. The specified goal,
by definition, is a max reward goal target (since it is the
“thing-being-rewarded”), as is the reward process itself (whether
optimized for terminally or instrumentally). But in principle,
misgeneralized goals (e.g., “get gold stuff in general”) could be “max
reward” as well, if you never show the model the inputs where the reward
process would punish them.
(The thing that speaks against mis-generalized
non-training-gamers – though, not decisively – is what I’ll call
“mundane adversarial training” – that is, showing the model a wide
diversity of training inputs designed to differentiate between the
specified goal and other mis-generalized goals.[1] Thus, for example,
if you show your “get-gold-stuff-in-general” model a situation where
it’s easier to get gold cupcakes than gold coins, then giving reward for
gold-coin-getting will punish the misgeneralized goal.)
Finally, I think slack may be a useful concept in understanding various
biological analogies for ML training.
Thus, for example, people sometimes analogize the human
dopamine/pleasure system to the reward process, and note that many
humans don’t end up pursuing “reward” in this sense directly – for
example, they would turn down the chance to
“wirehead”
in experience machines. I’ll leave the question of whether this is a
good analogy for another day (though note, at the least, that humans
who “wirehead” in this sense would’ve been selected against by
evolution). If we go with this analogy, though, then it seems worth
noting that this sort of reward process, at least, plausibly leaves
quite a bit slack – e.g., many humans plausibly could get quite a
bit more reward than they do (for example, by optimizing more
directly for their own pleasure), but the reward process doesn’t
pressure them very hard to do so.
Similarly, to the extent we analogize evolutionary selection to ML
training, it seems to have left humans quite a bit of “slack,” at
least thus far – that is, we could plausibly be performing much
better by the lights of inclusive genetic fitness (though if you
imagine evolutionary selection persisting for much longer, one can
imagine ending up with creatures that optimize for their inclusive
genetic fitness much more actively).
How much slack will there be in training? I’m not sure, and I think it
will plausibly vary according to the stage of the training process. For
example, naively, pre-training currently looks to me like a lower-slack
regime than RL fine-tuning. What’s more, to the extent that “slack” ends
up pointing at something real and important, I think it plausible that
it will be a parameter that we can control – for example, by training
longer and on more data.
Assuming we can control it, is it preferable to have less slack, or
more? Again, I’m not sure. Currently, though, I lean towards the view
that less slack is preferable, because less slack gives you higher
confidence about what sort of model you end up with.[2] Indeed,
counting on slack to ensure alignment in particular smacks, to me, of
wishful thinking – and in particular, of counting on greater
uncertainty about the model’s goals to speak in favor of the specific
goals you want it to have. Thus, for example, my sense is that some
people acknowledge that the goals specified by our training process will
be misaligned to some extent (for example, human raters will sometimes
reward dishonest, misleading, or manipulative responses), but they hope
that our models learn an aligned policy regardless. But even granted
that the slack in training allows this deviation from max-reward
behavior, why think that the deviation will land on an aligned policy in
particular? (There are possible answers here – for example, that an
aligned policy like “just be honest” is in some sense simpler or more
natural.[3] But I’m wary of wishful thinking here as well.)
My main point at present, though, is just that the degree of slack in
training may be an important factor shaping our expectations about what
sorts of models training will produce.
I’m calling this “mundane” adversarial training because it
investigates what the model does in situations we’re able to
actually put it in. This is in contrast with fancier and more
speculative forms of adversarial training, which attempt to get
information about what a model would do in situations we can’t
actually show it, at least not in a controlled setting – for
example, what it would do if it were no longer under our control, or
what it would do if the true date was ten years in the future, etc.
On “slack” in training (Section 1.5 of “Scheming AIs”)
This is Section 1.5 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.
Audio version of this section here, or search “Joe Carlsmith Audio” on your podcast app.
On “slack” in training
Before diving into an assessment of the arguments for expecting scheming, I also want to flag a factor that will come up repeatedly in what follows: namely, the degree of “slack” that we should expect training to allow. By this I mean something like: how much is the training process ruthlessly and relentlessly pressuring the model to perform in a manner that yields maximum reward, vs. shaping the model in a more “relaxed” way, that leaves more room for less-than-maximally rewarded behavior. That is, in a low-slack regime, “but that sort of model would be getting less reward than would be possible given its capabilities” is a strong counterargument against training creating a model of the relevant kind, whereas in a high-slack regime, it’s not (so high slack regimes will generally involve greater uncertainty about the type of model you end up with, since models that get less-than-maximal reward are still in the running).
Or, in more human terms: a low-slack regime is more like a hyper-intense financial firm that immediately fires any employees who fall behind in generating profits (and where you’d therefore expect surviving employees to be hyper-focused on generating profits – or perhaps, hyper-focused on the profits that their supervisors think they’re generating), whereas a high-slack regime is more like a firm where employees can freely goof off, drink martinis at lunch, and pursue projects only vaguely related to the company’s bottom line, and who only need to generate some amount of profit for the firm sometimes.
(Or at least, that’s the broad distinction I’m trying to point at. Unfortunately, I don’t have a great way of making it much more precise, and I think it’s possible that thinking in these terms will ultimately be misleading.)
Slack matters here partly because below, I’m going to be making various arguments that appeal to possibly-quite-small differences in the amount of reward that different models will get. And the force of these arguments depends on how sensitive training is to these differences. But I also think it can inform our sense of what models to expect more generally.
For example, I think slack matters to the probability that training will create models that pursue proxy goals imperfectly correlated with reward on the training inputs. Thus, in a low-slack regime, it may be fairly unlikely for a model trained to help humans with science to end up pursuing a general “curiosity drive” (in a manner that doesn’t then motivate instrumental training-gaming), because a model’s pursuing its curiosity in training would sometimes deviate from maximally helping-the-humans-with-science.
That said, note that the degree of slack is conceptually distinct from the diversity and robustness of the efforts made in training to root out goal misgeneralization. Thus, for example, if you’re rewarding a model when it gets gold coins, but you only ever show your model environments where the only gold things are coins, then a model that tries to get gold-stuff-in-general will perform just as well a model that gets gold coins in particular, regardless of how intensely training pressures the model to get maximum reward on those environments. E.g., a low-slack regime could in principle select either of these models, whereas a high-slack regime would leave more room for models that just get fewer gold coins period (for example, models that sometimes pursue red things instead, or who waste lots of time thinking before they pursue their gold coins).
In this sense, a low-slack regime doesn’t speak all that strongly against mis-generalized non-training-gamers. Rather, it speaks against models that aren’t pursuing what I’ll call a “max reward goal target” – that is, a goal target very closely correlated with reward on the inputs the model in fact receives in training. The specified goal, by definition, is a max reward goal target (since it is the “thing-being-rewarded”), as is the reward process itself (whether optimized for terminally or instrumentally). But in principle, misgeneralized goals (e.g., “get gold stuff in general”) could be “max reward” as well, if you never show the model the inputs where the reward process would punish them.
(The thing that speaks against mis-generalized non-training-gamers – though, not decisively – is what I’ll call “mundane adversarial training” – that is, showing the model a wide diversity of training inputs designed to differentiate between the specified goal and other mis-generalized goals.[1] Thus, for example, if you show your “get-gold-stuff-in-general” model a situation where it’s easier to get gold cupcakes than gold coins, then giving reward for gold-coin-getting will punish the misgeneralized goal.)
Finally, I think slack may be a useful concept in understanding various biological analogies for ML training.
Thus, for example, people sometimes analogize the human dopamine/pleasure system to the reward process, and note that many humans don’t end up pursuing “reward” in this sense directly – for example, they would turn down the chance to “wirehead” in experience machines. I’ll leave the question of whether this is a good analogy for another day (though note, at the least, that humans who “wirehead” in this sense would’ve been selected against by evolution). If we go with this analogy, though, then it seems worth noting that this sort of reward process, at least, plausibly leaves quite a bit slack – e.g., many humans plausibly could get quite a bit more reward than they do (for example, by optimizing more directly for their own pleasure), but the reward process doesn’t pressure them very hard to do so.
Similarly, to the extent we analogize evolutionary selection to ML training, it seems to have left humans quite a bit of “slack,” at least thus far – that is, we could plausibly be performing much better by the lights of inclusive genetic fitness (though if you imagine evolutionary selection persisting for much longer, one can imagine ending up with creatures that optimize for their inclusive genetic fitness much more actively).
How much slack will there be in training? I’m not sure, and I think it will plausibly vary according to the stage of the training process. For example, naively, pre-training currently looks to me like a lower-slack regime than RL fine-tuning. What’s more, to the extent that “slack” ends up pointing at something real and important, I think it plausible that it will be a parameter that we can control – for example, by training longer and on more data.
Assuming we can control it, is it preferable to have less slack, or more? Again, I’m not sure. Currently, though, I lean towards the view that less slack is preferable, because less slack gives you higher confidence about what sort of model you end up with.[2] Indeed, counting on slack to ensure alignment in particular smacks, to me, of wishful thinking – and in particular, of counting on greater uncertainty about the model’s goals to speak in favor of the specific goals you want it to have. Thus, for example, my sense is that some people acknowledge that the goals specified by our training process will be misaligned to some extent (for example, human raters will sometimes reward dishonest, misleading, or manipulative responses), but they hope that our models learn an aligned policy regardless. But even granted that the slack in training allows this deviation from max-reward behavior, why think that the deviation will land on an aligned policy in particular? (There are possible answers here – for example, that an aligned policy like “just be honest” is in some sense simpler or more natural.[3] But I’m wary of wishful thinking here as well.)
My main point at present, though, is just that the degree of slack in training may be an important factor shaping our expectations about what sorts of models training will produce.
I’m calling this “mundane” adversarial training because it investigates what the model does in situations we’re able to actually put it in. This is in contrast with fancier and more speculative forms of adversarial training, which attempt to get information about what a model would do in situations we can’t actually show it, at least not in a controlled setting – for example, what it would do if it were no longer under our control, or what it would do if the true date was ten years in the future, etc.
Thanks to Daniel Kokotajlo for discussion here.
Various humans, for example, plausibly settle on a policy like “just be honest” even though it doesn’t always get rewarded.