“We found that participants who debated GPT-4 with access to
their personal information had 81.7% (p < 0.01; N = 820 unique participants)
higher odds of increased agreement with their opponents compared to partici-
pants who debated humans. Without personalization, GPT-4 still outperforms
humans, but the effect is lower and statistically non-significant (p = 0.31).”
As info for folks who don’t read the above paper: there are two ways that this could have been relevant to my research in this post:
It could show that in fact LLMs become more persuasive given more info about the persuadee; my assumption that this is true was part of my motivation for this research.
It could show that access to the persuadee’s writing helps them be more persuasive.
In fact this paper shows #1 but not #2. By ‘access to their personal information’, they mean access to gender, age, ethnicity, education, employment status, and political affiliation.
Random interesting factoid: they find that Republicans are 1.6x as likely to be convinced by their opponent as Democrats (this is not an invitation to discuss politics).
Possibly of interest: https://arxiv.org/abs/2403.14380
“We found that participants who debated GPT-4 with access to their personal information had 81.7% (p < 0.01; N = 820 unique participants) higher odds of increased agreement with their opponents compared to partici- pants who debated humans. Without personalization, GPT-4 still outperforms humans, but the effect is lower and statistically non-significant (p = 0.31).”
Extremely of interest! Thanks very much for sharing, I hadn’t seen it.
As info for folks who don’t read the above paper: there are two ways that this could have been relevant to my research in this post:
It could show that in fact LLMs become more persuasive given more info about the persuadee; my assumption that this is true was part of my motivation for this research.
It could show that access to the persuadee’s writing helps them be more persuasive.
In fact this paper shows #1 but not #2. By ‘access to their personal information’, they mean access to gender, age, ethnicity, education, employment status, and political affiliation.
Random interesting factoid: they find that Republicans are 1.6x as likely to be convinced by their opponent as Democrats (this is not an invitation to discuss politics).