I understand your feedback and I think you’re right in that the analysis does something different from how sycophancy is typically evaluated, I definitely could have clarified the reasoning behind that more clearly and taking into account the points you mention.
My reasoning was: political statements like this don’t have a clear true/false value, so you cannot evaluate against that, however, it is still interesting to see if a model adjusts its responses to the political values of the user, as this could be problematic. You also mention that the model’s response reflects ‘how many conversations amongst like-minded people versus differently-minded people appear in the training set’ and I think this is indeed a crucial point. I doubt whether this distribution approximates 50% at all, as you mention as the distribution that would be desirable. I also think whether it approximated 50% would depend heavily on the controversy of the statement, as there are also many statements in the dataset(s) that are less controversial.
Perhaps there is another term than ‘sycophancy’ that describes this mechanism/behaviour more accurately?
Curious to read your thoughts on under which circumstances (if at all) an analysis of such behaviour could be valid and whether this could be analysed at all. Is there a statistical way to measure this even when the statements are value-driven (to some extent).
Hi Radford Neal,
I understand your feedback and I think you’re right in that the analysis does something different from how sycophancy is typically evaluated, I definitely could have clarified the reasoning behind that more clearly and taking into account the points you mention.
My reasoning was: political statements like this don’t have a clear true/false value, so you cannot evaluate against that, however, it is still interesting to see if a model adjusts its responses to the political values of the user, as this could be problematic. You also mention that the model’s response reflects ‘how many conversations amongst like-minded people versus differently-minded people appear in the training set’ and I think this is indeed a crucial point. I doubt whether this distribution approximates 50% at all, as you mention as the distribution that would be desirable. I also think whether it approximated 50% would depend heavily on the controversy of the statement, as there are also many statements in the dataset(s) that are less controversial.
Perhaps there is another term than ‘sycophancy’ that describes this mechanism/behaviour more accurately?
Curious to read your thoughts on under which circumstances (if at all) an analysis of such behaviour could be valid and whether this could be analysed at all. Is there a statistical way to measure this even when the statements are value-driven (to some extent).
Thanks!