Scott Alexander wrote a really good article on this; lots of people out there right now are falsely concluding that nudges don’t exist. Furthermore, the replication crisis is an issue from academic psychology, which I argued in this post is obsolete and should stop anchoring our understanding of how effective human manipulation can become in the 2020s. I cover some of this in the “alien” paragraph.
Consider targeted advertising. Despite the amount of data social media companies collect on their users, ad targeting still sucks. Even in the area of attempted behavior manipulation that’s subject to more optimization pressure than any other, companies still can’t predict, let alone control, their users’ purchasing decisions with anything close to consistency. Their data simply isn’t sufficient.
I agree that the vast majority of people attempting to do targeting advertising do not have sufficient data. But that doesn’t tell us much about whether the big 5 tech companies, or intelligence agencies, have sufficient data to do that, and aren’t being really loud about it. My argument is that, due to the prevalence of data theft and data poisoning, it’s entirely plausible that sufficient data is being monopolized by extremely powerful people, as the profit motive alone is enough for that, let alone a new era of controlling people. I’ve met several people who tried to do predictive analytics for campaign staff, and each one of them had grossly insufficient data compared to the big 5 tech companies, and yet they were confident that the big 5 tech companies couldn’t do it either. It really shouldn’t be surprising that this mentality would be prevalent in both scenarios where the human steering tech was and wasn’t acute.
The human brain isn’t a predictable computer program for which a hacker can discover “zero days.” It’s a noisy physical organ that’s subject to chaotic dynamics and frequently does things that would be impossible to predict even with an extremely extensive set of behavioral data.
I highly doubt that the signal to noise ratio of the human brain makes predictive analytics impossible, let alone with sample sizes in the millions or even billions. One of my main arguments is that there are in fact millions or billions of sensors pointed at millions of people (although of course telecom security might limit the ability for any actor anywhere to acquire enough unpoisoned data). Clown attacks is an example; the human though process is indeed sloppy, but it is sloppy in consistent, ways and these consistent ways can easily be discovered and exploited when you have such large amounts of interactive behavioral data. Gradient descent + social media algorithms alone can steer people’s thinking in measurable directions, insofar as those directions are measurable.
I think it is theoretically possible that humans are extremely resistant to manipulation of all kinds, but I’d argue that bar for proving or indicating that is much higher than you seem to think here, especially in the current experiment-based paradigm where only powerful people get access to enough useful data and the data centers required to process it (though it’s possible that this level of research is less centralized and more accessible than I thought, JP Morgan’s experiments on their employees surprised me, but I also wouldn’t be surprised to find that the NSA poisoned JP Morgan’s data sets because they took issue with major banks joining in on gaining these capabilities).
I agree that the vast majority of people attempting to do targeting advertising do not have sufficient data. But that doesn’t tell us much about whether the big 5 tech companies, or intelligence agencies, have sufficient data to do that, and aren’t being really loud about it.
If any of the big tech companies had the capability for actually-good targeted advertising, they’d use it. The profit motive would be very strong. The fact that targeted ads still “miss” so frequently is strong evidence that nobody has the highly advanced, scalable, personalized manipulation capabilities you describe.
Social media recommendation algorithms aren’t very powerful either. For instance, when I visit YouTube, it’s not unusual for it to completely fail to recommend anything I’m interested in watching. The algorithm doesn’t even seem to have figured out that I’ve never played Skyrim or that I’m not Christian. In the scenario in which social media companies have powerful manipulation capabilities that they hide from the public, the gap between the companies’ public-facing and hidden recommendation systems would be implausibly large.
As for chaotic dynamics, there’s strong experimentalevidence that they occur in the brain, and even if they didn’t, they would still occur in people’s surrounding environments. Even if it weren’t prohibitively expensive to point millions or billions of sensors at one person, that still wouldn’t be enough to predict everything. But tech companies and security agencies don’t have millions or billions of sensors pointed at each person. Compared to the entirety of what a person experiences and thinks, computer use patterns are a very sparse signal even for the most terminally online segment of the population (let alone your very offline grandma). Hence the YouTube algorithm flubbing something as basic as my religion—there’s just too much relevant information they don’t have access to.
Making people feel unsafe is a great way to lose users, and people during the mid 2010s did notice advertisements that predicted what they wanted before they gave any indication that they want it, and they did react in an extraordinary measurable way to that (quitting the platform). Even the most mundane systems you could possibly expect would notice a dynamic like that and default to playing conservatively and reducing risk e.g. randomly selecting ads. People who use LW or are unusually strongly involved with AI in other ways probably stopped getting truly targeted ads years ago (or even people with more than statistics 101), because with sample sizes in the millions you can anticipate that kind of problem from a mile away.
Likewise, if a social media platform makes you feel safe, that is basically not an indicator at all at how effective gradient descent is at creating user experiences that prevent quitting.
Can you go into more detail about your chaotic dynamics point? Chaotic dynamics don’t seem to prevent clown attacks from succeeding at a sufficiently high rate; and the whole point of the multi-armed bandit algorithms I’ve described is that any manipulation technique will have a decent failure rate; that redundancy is why using social media for an hour a day is dangerous but looking at a computer screen one time is safe. These systems require massive amounts of data on massive numbers of people in order to work, and that is what they’re getting (although I’m not saying that a gait camera couldn’t make a ton of probabilistic inferences about a person from only 10 seconds of footage).
If I had read your comment before writing the post, I wouldn’t have used the word “zero days” nearly so frequently in this post, or even at all, because you’re absolutely right that the exploits I’ve described here are very squishy and unreliable in a way that is very different from how zero days are generally understood.
Scott Alexander wrote a really good article on this; lots of people out there right now are falsely concluding that nudges don’t exist. Furthermore, the replication crisis is an issue from academic psychology, which I argued in this post is obsolete and should stop anchoring our understanding of how effective human manipulation can become in the 2020s. I cover some of this in the “alien” paragraph.
I agree that the vast majority of people attempting to do targeting advertising do not have sufficient data. But that doesn’t tell us much about whether the big 5 tech companies, or intelligence agencies, have sufficient data to do that, and aren’t being really loud about it. My argument is that, due to the prevalence of data theft and data poisoning, it’s entirely plausible that sufficient data is being monopolized by extremely powerful people, as the profit motive alone is enough for that, let alone a new era of controlling people. I’ve met several people who tried to do predictive analytics for campaign staff, and each one of them had grossly insufficient data compared to the big 5 tech companies, and yet they were confident that the big 5 tech companies couldn’t do it either. It really shouldn’t be surprising that this mentality would be prevalent in both scenarios where the human steering tech was and wasn’t acute.
I highly doubt that the signal to noise ratio of the human brain makes predictive analytics impossible, let alone with sample sizes in the millions or even billions. One of my main arguments is that there are in fact millions or billions of sensors pointed at millions of people (although of course telecom security might limit the ability for any actor anywhere to acquire enough unpoisoned data). Clown attacks is an example; the human though process is indeed sloppy, but it is sloppy in consistent, ways and these consistent ways can easily be discovered and exploited when you have such large amounts of interactive behavioral data. Gradient descent + social media algorithms alone can steer people’s thinking in measurable directions, insofar as those directions are measurable.
I think it is theoretically possible that humans are extremely resistant to manipulation of all kinds, but I’d argue that bar for proving or indicating that is much higher than you seem to think here, especially in the current experiment-based paradigm where only powerful people get access to enough useful data and the data centers required to process it (though it’s possible that this level of research is less centralized and more accessible than I thought, JP Morgan’s experiments on their employees surprised me, but I also wouldn’t be surprised to find that the NSA poisoned JP Morgan’s data sets because they took issue with major banks joining in on gaining these capabilities).
If any of the big tech companies had the capability for actually-good targeted advertising, they’d use it. The profit motive would be very strong. The fact that targeted ads still “miss” so frequently is strong evidence that nobody has the highly advanced, scalable, personalized manipulation capabilities you describe.
Social media recommendation algorithms aren’t very powerful either. For instance, when I visit YouTube, it’s not unusual for it to completely fail to recommend anything I’m interested in watching. The algorithm doesn’t even seem to have figured out that I’ve never played Skyrim or that I’m not Christian. In the scenario in which social media companies have powerful manipulation capabilities that they hide from the public, the gap between the companies’ public-facing and hidden recommendation systems would be implausibly large.
As for chaotic dynamics, there’s strong experimental evidence that they occur in the brain, and even if they didn’t, they would still occur in people’s surrounding environments. Even if it weren’t prohibitively expensive to point millions or billions of sensors at one person, that still wouldn’t be enough to predict everything. But tech companies and security agencies don’t have millions or billions of sensors pointed at each person. Compared to the entirety of what a person experiences and thinks, computer use patterns are a very sparse signal even for the most terminally online segment of the population (let alone your very offline grandma). Hence the YouTube algorithm flubbing something as basic as my religion—there’s just too much relevant information they don’t have access to.
Making people feel unsafe is a great way to lose users, and people during the mid 2010s did notice advertisements that predicted what they wanted before they gave any indication that they want it, and they did react in an extraordinary measurable way to that (quitting the platform). Even the most mundane systems you could possibly expect would notice a dynamic like that and default to playing conservatively and reducing risk e.g. randomly selecting ads. People who use LW or are unusually strongly involved with AI in other ways probably stopped getting truly targeted ads years ago (or even people with more than statistics 101), because with sample sizes in the millions you can anticipate that kind of problem from a mile away.
Likewise, if a social media platform makes you feel safe, that is basically not an indicator at all at how effective gradient descent is at creating user experiences that prevent quitting.
Can you go into more detail about your chaotic dynamics point? Chaotic dynamics don’t seem to prevent clown attacks from succeeding at a sufficiently high rate; and the whole point of the multi-armed bandit algorithms I’ve described is that any manipulation technique will have a decent failure rate; that redundancy is why using social media for an hour a day is dangerous but looking at a computer screen one time is safe. These systems require massive amounts of data on massive numbers of people in order to work, and that is what they’re getting (although I’m not saying that a gait camera couldn’t make a ton of probabilistic inferences about a person from only 10 seconds of footage).
If I had read your comment before writing the post, I wouldn’t have used the word “zero days” nearly so frequently in this post, or even at all, because you’re absolutely right that the exploits I’ve described here are very squishy and unreliable in a way that is very different from how zero days are generally understood.
There was that story about the girl that got ads for baby stuff before her parents knew about the pregnancy… ;)