Thank you for doing this research! There’s a lot that I love about this piece, besides the obvious thing which is that it is seriously investigating a very important and neglected topic. For example, I love the definitions of various things, I love the little vignettes of ways the near future could go, and I love the core argument about why these possibilities are both plausible and really bad.
I am pretty much on the same page as you, but I have a few minor disagreements (mostly about emphasis). TL;DR is that you focus on more extreme, exotic possibilities (which I think are real and deserve attention) and neglect more mundane, moderate possibilities that are still bad enough to constitute significant existential risk factors.
Chatbots / AI personas could turn out to be a really big deal but even if they don’t the overall argument/threat model still goes through. And the some of the non-chatbot kinds of persuasion tool are much more plausible because they already exist and already are having big effects. (E.g. feeders and analysts). Also, even your “moderately competent persuasion” benchmark is more extreme than I expect to happen (at least in the next 5 years):
Create an AI assistant tailored to a particular individual. Allow them to use it as much as they want.
The AI assistant is highly competent at providing the person with correct and relevant information for their daily life where the person knows the ground truth, and generally sounds knowledgeable and wise. Due to this, with p~0.5 the person feels inclined to turn to it for advice, and expect it to be more knowledgeable/reliable than their human friends, on questions where they don’t know the ground truth. They allow it to strongly filter what information they receive (e.g. they read personalized summaries of the news generated by the assistant). They become locked in to this particular product.
Consider by contrast a more “mundane” scenario, as follows:
Advances in AI drop the cost of censorship by two orders of magnitude; one human censor + big AI assistants can do what would have taken hundreds or thousands of human censors before. Meanwhile, advances in AI improve the effectiveness of propaganda significantly; when the people that produce content pushing an ideology or political agenda (journalists, editors, speechwriters, tech companies creating recommendation algorithms, official government propagandists, academics, etc.) have access to the latest AI tools, they are able to craft and recommend content that is more customized and persuasive to the recepient. Quantitatively, whatever the effect size of media and recommendation was in 2020 (say, moving 2 percentile points on the ideological spectrum towards ideology X for every hour per week spent consuming content pushing X) similar-genre content produced using the latest AI tools in 2025 has twice the effect size on average. And all of this happens by 2026.
The technology in this world doesn’t look qualitatively different from the stuff deployed today; arguably it’s just a more aggressive deployment of language model techniques we already have. No exciting chatbots or AI personas. Yet I think that this world is really dangerous.
I don’t have a well-fleshed out model exactly but I do have a simple one: Human populations are territory, fought over by ideologies; censorship and propaganda are significant factors in how ideologies spread, intensify, and resist decay/drift; therefore if censorship and propaganda get significantly cheaper and more effective for some ideologies (the ones that have access to the latest tech) those ideologies will spread significantly (gobbling up territory controlled by ideologies that don’t have the tech, as well as neutral, previously-unideological territory) and intensify significantly and become noticeably more dogmatic/stable.
I would love to see research that collects data on metrics like cost of censorship and effectiveness of propaganda, and plots them over time to see if there is any general trend and how much it has varied over the course of human history, and whether the latest AI techniques are changing these metrics significantly. I would also love to see research on how important censorship and propaganda (broadly construed, to include pretty much any content filtering or producing designed at least in part to push an ideology or political agenda) is compared to other factors such as face-to-face conversations, random drift, and truth/feedback-from-reality.
I would love to see research that collects data on metrics like cost of censorship and effectiveness of propaganda, and plots them over time to see if there is any general trend and how much it has varied over the course of human history, and whether the latest AI techniques are changing these metrics significantly.
If you have not already seen it, this report from CSET discusses the extent to which something as capable as GPT-3 changes the cost and effectiveness of disinformation and propaganda.
Thank you for doing this research! There’s a lot that I love about this piece, besides the obvious thing which is that it is seriously investigating a very important and neglected topic. For example, I love the definitions of various things, I love the little vignettes of ways the near future could go, and I love the core argument about why these possibilities are both plausible and really bad.
I am pretty much on the same page as you, but I have a few minor disagreements (mostly about emphasis). TL;DR is that you focus on more extreme, exotic possibilities (which I think are real and deserve attention) and neglect more mundane, moderate possibilities that are still bad enough to constitute significant existential risk factors.
Chatbots / AI personas could turn out to be a really big deal but even if they don’t the overall argument/threat model still goes through. And the some of the non-chatbot kinds of persuasion tool are much more plausible because they already exist and already are having big effects. (E.g. feeders and analysts). Also, even your “moderately competent persuasion” benchmark is more extreme than I expect to happen (at least in the next 5 years):
Consider by contrast a more “mundane” scenario, as follows:
Advances in AI drop the cost of censorship by two orders of magnitude; one human censor + big AI assistants can do what would have taken hundreds or thousands of human censors before. Meanwhile, advances in AI improve the effectiveness of propaganda significantly; when the people that produce content pushing an ideology or political agenda (journalists, editors, speechwriters, tech companies creating recommendation algorithms, official government propagandists, academics, etc.) have access to the latest AI tools, they are able to craft and recommend content that is more customized and persuasive to the recepient. Quantitatively, whatever the effect size of media and recommendation was in 2020 (say, moving 2 percentile points on the ideological spectrum towards ideology X for every hour per week spent consuming content pushing X) similar-genre content produced using the latest AI tools in 2025 has twice the effect size on average. And all of this happens by 2026.
The technology in this world doesn’t look qualitatively different from the stuff deployed today; arguably it’s just a more aggressive deployment of language model techniques we already have. No exciting chatbots or AI personas. Yet I think that this world is really dangerous.
I don’t have a well-fleshed out model exactly but I do have a simple one: Human populations are territory, fought over by ideologies; censorship and propaganda are significant factors in how ideologies spread, intensify, and resist decay/drift; therefore if censorship and propaganda get significantly cheaper and more effective for some ideologies (the ones that have access to the latest tech) those ideologies will spread significantly (gobbling up territory controlled by ideologies that don’t have the tech, as well as neutral, previously-unideological territory) and intensify significantly and become noticeably more dogmatic/stable.
I would love to see research that collects data on metrics like cost of censorship and effectiveness of propaganda, and plots them over time to see if there is any general trend and how much it has varied over the course of human history, and whether the latest AI techniques are changing these metrics significantly. I would also love to see research on how important censorship and propaganda (broadly construed, to include pretty much any content filtering or producing designed at least in part to push an ideology or political agenda) is compared to other factors such as face-to-face conversations, random drift, and truth/feedback-from-reality.
If you have not already seen it, this report from CSET discusses the extent to which something as capable as GPT-3 changes the cost and effectiveness of disinformation and propaganda.
There was also a recorded discussion/seminar of the same topics with the authors of the report.
I don’t think it’s exactly what you’re looking for, but it seemed adjacent enough to be worth mentioning.