Note that in this example your model is unable to sample from the conditional you specified, since it is restricted to α+β+γ=1. In this regime truthfulness and persuasiveness are anticorrelated because of a capability constraint of your model, it just literally isn’t able to increase both at the same time, and conditioning can do better because you are generating lots of samples and picking the best.
(You point this out in your comment, but it seems worth emphasizing. As you say, if you do RL with a KL penalty, then the capability limit is the only way you can get this kind of mismatch. Without a KL penalty the exact behavior of RL vs conditioning will depend on details of gradient descent, though it seems quite similar in practice and I’m not sure which way this comparison goes.)
In terms of being able to sample from the conditional, I don’t think that the important constraint here is α+β+γ=1. Rather, it seems that the important constraint is that our architecture can only sample from distributions of the form αN(μA,σ2A)+βN(μB,σ2B)+γN(μC,σ2C); even allowing α,β,γ to be arbitrary real numbers, this will never be the same as either (a) the distribution produced by conditioning the base model on high persuasiveness, or (b) the distribution which maximizes expected persuasiveness—KL divergence from the base model.
I’m not sure the above point as an important one. I just wanted to disambiguate some different capabilities limitations which appeared in the example:
limitations on what sorts of distributions the architecture could approximate
limitations on the latent capabilities in the base model for producing true/persuasive outputs
limitations on how much steering each of the various latent capabilities gets to exert (α+β+γ=1).
On my understanding, your point was about limitation (1). But I don’t feel especially nervous about limitation (1) -- taking the output distribution of our pretrained model and weighting it by a Boltzman factor feels like it should produce a kinda crazy distribution, and my naive intuition is that we shouldn’t necessarily expect our model to be able to approximate this distribution that well after RL finetuning with a KL penalty.
I think I’m most nervous about the way we modeled limitation (3): I have no idea how to think about the extent to which models’ capabilities trade off against one another, and taking α,β,γ∈[0,1] without additional constraints would have resulted in outputs of mean truthiness α′μA+μB for some α′ which we can’t pin down without specifying additional details (e.g. is there weight decay?).
I’m also most nervous about this way of modeling limitation (2)/(3), since it seems like it leads directly to the conclusion “fine-tuning always trades off truthfulness and persuasion, but conditioning can improve both.”
Note that in this example your model is unable to sample from the conditional you specified, since it is restricted to α+β+γ=1. In this regime truthfulness and persuasiveness are anticorrelated because of a capability constraint of your model, it just literally isn’t able to increase both at the same time, and conditioning can do better because you are generating lots of samples and picking the best.
(You point this out in your comment, but it seems worth emphasizing. As you say, if you do RL with a KL penalty, then the capability limit is the only way you can get this kind of mismatch. Without a KL penalty the exact behavior of RL vs conditioning will depend on details of gradient descent, though it seems quite similar in practice and I’m not sure which way this comparison goes.)
In terms of being able to sample from the conditional, I don’t think that the important constraint here is α+β+γ=1. Rather, it seems that the important constraint is that our architecture can only sample from distributions of the form αN(μA,σ2A)+βN(μB,σ2B)+γN(μC,σ2C); even allowing α,β,γ to be arbitrary real numbers, this will never be the same as either (a) the distribution produced by conditioning the base model on high persuasiveness, or (b) the distribution which maximizes expected persuasiveness—KL divergence from the base model.
I’m not sure the above point as an important one. I just wanted to disambiguate some different capabilities limitations which appeared in the example:
limitations on what sorts of distributions the architecture could approximate
limitations on the latent capabilities in the base model for producing true/persuasive outputs
limitations on how much steering each of the various latent capabilities gets to exert (α+β+γ=1).
On my understanding, your point was about limitation (1). But I don’t feel especially nervous about limitation (1) -- taking the output distribution of our pretrained model and weighting it by a Boltzman factor feels like it should produce a kinda crazy distribution, and my naive intuition is that we shouldn’t necessarily expect our model to be able to approximate this distribution that well after RL finetuning with a KL penalty.
I think I’m most nervous about the way we modeled limitation (3): I have no idea how to think about the extent to which models’ capabilities trade off against one another, and taking α,β,γ∈[0,1] without additional constraints would have resulted in outputs of mean truthiness α′μA+μB for some α′ which we can’t pin down without specifying additional details (e.g. is there weight decay?).
I’m also most nervous about this way of modeling limitation (2)/(3), since it seems like it leads directly to the conclusion “fine-tuning always trades off truthfulness and persuasion, but conditioning can improve both.”