In terms of being able to sample from the conditional, I don’t think that the important constraint here is α+β+γ=1. Rather, it seems that the important constraint is that our architecture can only sample from distributions of the form αN(μA,σ2A)+βN(μB,σ2B)+γN(μC,σ2C); even allowing α,β,γ to be arbitrary real numbers, this will never be the same as either (a) the distribution produced by conditioning the base model on high persuasiveness, or (b) the distribution which maximizes expected persuasiveness—KL divergence from the base model.
I’m not sure the above point as an important one. I just wanted to disambiguate some different capabilities limitations which appeared in the example:
limitations on what sorts of distributions the architecture could approximate
limitations on the latent capabilities in the base model for producing true/persuasive outputs
limitations on how much steering each of the various latent capabilities gets to exert (α+β+γ=1).
On my understanding, your point was about limitation (1). But I don’t feel especially nervous about limitation (1) -- taking the output distribution of our pretrained model and weighting it by a Boltzman factor feels like it should produce a kinda crazy distribution, and my naive intuition is that we shouldn’t necessarily expect our model to be able to approximate this distribution that well after RL finetuning with a KL penalty.
I think I’m most nervous about the way we modeled limitation (3): I have no idea how to think about the extent to which models’ capabilities trade off against one another, and taking α,β,γ∈[0,1] without additional constraints would have resulted in outputs of mean truthiness α′μA+μB for some α′ which we can’t pin down without specifying additional details (e.g. is there weight decay?).
I’m also most nervous about this way of modeling limitation (2)/(3), since it seems like it leads directly to the conclusion “fine-tuning always trades off truthfulness and persuasion, but conditioning can improve both.”
In terms of being able to sample from the conditional, I don’t think that the important constraint here is α+β+γ=1. Rather, it seems that the important constraint is that our architecture can only sample from distributions of the form αN(μA,σ2A)+βN(μB,σ2B)+γN(μC,σ2C); even allowing α,β,γ to be arbitrary real numbers, this will never be the same as either (a) the distribution produced by conditioning the base model on high persuasiveness, or (b) the distribution which maximizes expected persuasiveness—KL divergence from the base model.
I’m not sure the above point as an important one. I just wanted to disambiguate some different capabilities limitations which appeared in the example:
limitations on what sorts of distributions the architecture could approximate
limitations on the latent capabilities in the base model for producing true/persuasive outputs
limitations on how much steering each of the various latent capabilities gets to exert (α+β+γ=1).
On my understanding, your point was about limitation (1). But I don’t feel especially nervous about limitation (1) -- taking the output distribution of our pretrained model and weighting it by a Boltzman factor feels like it should produce a kinda crazy distribution, and my naive intuition is that we shouldn’t necessarily expect our model to be able to approximate this distribution that well after RL finetuning with a KL penalty.
I think I’m most nervous about the way we modeled limitation (3): I have no idea how to think about the extent to which models’ capabilities trade off against one another, and taking α,β,γ∈[0,1] without additional constraints would have resulted in outputs of mean truthiness α′μA+μB for some α′ which we can’t pin down without specifying additional details (e.g. is there weight decay?).
I’m also most nervous about this way of modeling limitation (2)/(3), since it seems like it leads directly to the conclusion “fine-tuning always trades off truthfulness and persuasion, but conditioning can improve both.”