Connor Leahy (QT of OP): While I expect I think the gap between the two is shorter than Sam thinks, I still think people are vastly underestimating how much of a problem this will be and already is. The Semantic Apocalypse is in sight.
While it’s great to see people taking the threat of AI persuasion seriously, semantic hijacking is the wrong approach.
LLMs finding combinations of words to manipulate people is probably years away, whereas conventional AI-powered research and hacking of the impression-formation process probably went online years ago, since large tech companies and intelligence agencies have access to billions of cases of people forming impressions in measurable directions e.g. social media user data. The use of AI for persuasion revolves around the data processing capabilities of 2010s AI which facilitates human behavior research when combined with large amounts of user data, not recent trends with LLMs.
The AI safety community has catastrophically misguided world models about the incentives driving the AI industry, and this will result in bad decisions and misguided policies e.g. AI pause forecasting or relating to US-China affairs.
While it’s great to see people taking the threat of AI persuasion seriously, semantic hijacking is the wrong approach.
LLMs finding combinations of words to manipulate people is probably years away, whereas conventional AI-powered research and hacking of the impression-formation process probably went online years ago, since large tech companies and intelligence agencies have access to billions of cases of people forming impressions in measurable directions e.g. social media user data. The use of AI for persuasion revolves around the data processing capabilities of 2010s AI which facilitates human behavior research when combined with large amounts of user data, not recent trends with LLMs.
The AI safety community has catastrophically misguided world models about the incentives driving the AI industry, and this will result in bad decisions and misguided policies e.g. AI pause forecasting or relating to US-China affairs.