It’s common for people to be worried about recommender systems being addictive or promoting filter bubbles etc, but as far as I can tell, they don’t have very good arguments for these worries. Whenever I talk to someone who seems to have actually studied the topic in depth, it seems they think that there are problems with recommender systems, but they are different from what people usually imagine.
I’ll go through the articles I’ve read that argue for worrying about recommender systems, and explain why I find them unconvincing. I’ve only looked at the ones that are widely read; there are probably significantly better arguments that are much less widely read.
A few sources say that it is bad + it has incredible scale + it should be super easy to solve. (I don’t trust the sources and suspect the authors didn’t check them; I agree there’s huge scale; I don’t see why it should be super easy to solve even if there is a problem, especially given that many of the supposed problems seem to have existed before recommender systems.)
Maybe working on recommender systems would have spillover effects on AI alignment. (This seems dominated by just working directly on AI alignment. Also the core feature of AI alignment is that the AI system deliberately and intentionally does things, and creates plans in new situations that you hadn’t seen before, which is not the case with recommender systems, so I don’t expect many spillover effects.)
I don’t know what the main claim was. Ostensibly it was meant to be “it is bad that companies have monetized human attention since this leads to lots of bad incentives and bad outcomes”. But then so many specific things mentioned have nothing to do with this claim and instead seem to be a vague general “tech companies are bad”. Most egregiously, in section Global effects [01:02:44], Rob argues “WhatsApp doesn’t have ads / recommender systems, so it acts as a control group, but it too has bad outcomes, doesn’t this mean the problem isn’t ads / recommender systems?” and Tristan says “That’s right, WhatsApp is terrible, it’s causing mass lynchings” as though that supports his point.
When Rob made some critique of the main argument, Tristan deflected with an example of tech doing bad things. But it’s always vaguely related, so you think he’s addressing the critique, even though he hasn’t actually. (I’m reminded of the Zootopia strategy for press conferences.) See sections “The messy real world vs. an imagined idealised world [00:38:20]” (Rob: weren’t negative things happening before social media? Tristan: it’s easy to fake credibility in text), “The persuasion apocalypse [00:47:46]” (Rob: can’t one-on-one conversations be persuasive too? Tristan: you can lie in political ads), “Revolt of the Public [00:56:48]” (Rob: doesn’t the internet allow ordinary people to challenge established institutions in good ways? Tristan: Alex Jones has been recommended 15 billion times.)
US politics [01:13:32] is a rare counterexample, where Rob says “why aren’t other countries getting polarized”, and Tristan replies “since it’s a positive feedback loop only countries with high initial polarization will see increasing polarization”. It’s not a particularly convincing response, but at least it’s a response.
Tristan seems to be very big on “the tech companies changed what they were doing, that proves we were right”. I think it is just as consistent to say “we yelled at the companies a lot and got the public to yell at them too, and that caused a change, regardless of whether the problem was serious or not, or whether the solution was net positive or not”.
The second half of the podcast focuses more on solutions. Given that I am unconvinced about the problem, I wasn’t all that interested, but it seemed generally reasonable.
(This post responds to the object level claims, which I have not done because I don’t know much about the object level.)
There’s also the documentary “The Social Dilemma”, but I expect it’s focused entirely on problems, probably doesn’t try to have good rigorous statistics, and surely will make no attempt at a cost-benefit analysis so I seriously doubt it would change my mind on anything. (And it is associated with Tristan Harris so I’d assume that most of the relevant details would have made it into the 80K podcast.)
Recommender systems are still influential, and you could want to work on them just because of their huge scale. I like Designing Recommender Systems to Depolarize as an example of what this might look like.
Thanks for this Rohin. I’ve been trying to raise awareness about the potential dangers persuasion/propaganda tools, but you are totally right that I haven’t actually done anything close to a rigorous analysis. I agree with what you say here that a lot of the typical claims being thrown around seem based more on armchair reasoning than hard data. I’d love to see someone really lay out the arguments and analyze them… My current take is that (some of) the armchair theories seem pretty plausible to me, such that I’d believe them unless the data contradicts. But I’m extremely uncertain about this.
I’ve been trying to raise awareness about the potential dangers persuasion/propaganda tools
I should note that there’s a big difference between “recommender systems cause polarization as a side effect of optimizing for engagement” and “we might design tools that explicitly aim at persuasion / propaganda”. I’m confident we could (eventually) do the latter if we tried to; the question is primarily whether we will try to and if we do what it’s effects will be.
My current take is that (some of) the armchair theories seem pretty plausible to me, such that I’d believe them unless the data contradicts.
Usually, for any sufficiently complicated question (which automatically includes questions about the impact of technologies used by billions of people, since people are so diverse), I think an armchair theory is only slightly better than a monkey throwing darts, so I’m more in the position of “yup, sounds plausible, but that doesn’t constrain my beliefs about what the data will show and medium quality data will trump the theory no matter how it comes out”.
I should note that there’s a big difference between “recommender systems cause polarization as a side effect of optimizing for engagement” and “we might design tools that explicitly aim at persuasion / propaganda”. I’m confident we could (eventually) do the latter if we tried to; the question is primarily whether we will try to and if we do what it’s effects will be.
Oh, then maybe we don’t actually disagree that much! I am not at all confident that optimizing for engagement has the side effect of increasing polarization. It seems plausible but it’s also totally plausible that polarization is going up for some other reason(s). My concern (as illustrated in the vignette I wrote) is that we seem to be on a slippery slope to a world where persuasion/propaganda is more effective and widespread than it has been historically, thanks to new AI and big data methods. My model is: Ideologies and other entities have always been using propaganda of various kinds, and there’s always been a race between improving propaganda tech and improving truth-finding tech, but we are currently in a big AI boom and in particular in a Big Data and Natural Language Processing boom, and this seems like it’ll be a big boost to propaganda tech, and unfortunately I can’t think of ways in which it will correspondingly boost truth-finding-ness across society, because while it can be used to make truth-finding tech maybe (e.g. prediction markets, fact-checkers, etc.) it seems like most people in practice just don’t want to adopt truth-finding tech. It’s true that we could design a different society/culture that used all this awesome new tech to be super truth-seeking and have a very epistemically healthy discourse, but it seems like we are not about to do that anytime soon, instead we are going in the opposite direction.
I think that story involves lots of assumptions I don’t immediately believe (but don’t disbelieve either):
People are very deliberately building persuasion / propaganda tech (as opposed to e.g. people like to loudly state opinions and the persuasive ones rise to the top)
Such people will quickly realize that AI will be very useful for this
They will actually try to build it (as opposed to e.g. raising a moral outcry and trying to get it banned)
The resulting AI system will in fact be very good at persuasion / propaganda
AI that fights persuasion / propaganda either won’t be built or will be ineffective (my unreliable armchair reasoning suggests the opposite; it seems to me like right now human fact-checking labor can’t keep up with human controversy-creating labor partly because humans enjoy the latter more than the former; this won’t be true with AI)
And probably there are a bunch of other assumptions I haven’t even thought to question.
I think it seems fine to raise the possibility and do more research (and for all I know CSET or GovAI has done this research) but at least under my beliefs the current action should not be “raise awareness”, it should be “figure out whether the assumptions are justified”.
I think it seems fine to raise the possibility and do more research (and for all I know CSET or GovAI has done this research) but at least under my beliefs the current action should not be “raise awareness”, it should be “figure out whether the assumptions are justified”.
That’s all I’m trying to do at this point, to be clear. Perhaps “raise awareness” was the wrong choice of phrase.
Re: the object-level points: For how I see this going, see my vignette, and my reply to steve. The bullet points you put here make it seem like you have a different story in mind. [EDIT: But I agree with you that it’s all super unclear and more research is needed to have confidence in any of this.]
That’s all I’m trying to do at this point, to be clear.
Excellent :)
For how I see this going, see my vignette, and my reply to steve.
(Link is broken, but I found the comment.) After reading that reply I still feel like it involves the assumptions I mentioned above.
Maybe your point is that your story involves “silos” of Internet-space within which particular ideologies / propaganda reign supreme. I don’t really see that as changing my object-level points very much but perhaps I’m missing something.
I was confusing, sorry—what I meant was, technically my story involves assumptions like the ones you list in the bullet points, but the way you phrase them is… loaded? Designed to make them seem implausible? idk, something like that, in a way that made me wonder if you had a different story in mind. Going through them one by one:
People are very deliberately building persuasion / propaganda tech (as opposed to e.g. people like to loudly state opinions and the persuasive ones rise to the top)
This is already happening in 2021 and previous, in my story it happens more.
Such people will quickly realize that AI will be very useful for this
Again, this is already happening.
They will actually try to build it (as opposed to e.g. raising a moral outcry and trying to get it banned)
Plenty of people are already raising a moral outcry. In my story these people don’t succeed in getting it banned, but I agree the story could be wrong. I hope it is!
The resulting AI system will in fact be very good at persuasion / propaganda
Yep. I don’t have hard evidence, but intuitively this feels like the sort of thing today’s AI techniques would be good at, or at least good-enough-to-improve-on-the-state-of-the-art.
AI that fights persuasion / propaganda either won’t be built or will be ineffective (my unreliable armchair reasoning suggests the opposite; it seems to me like right now human fact-checking labor can’t keep up with human controversy-creating labor partly because humans enjoy the latter more than the former; this won’t be true with AI)
I think it won’t be built & deployed in such a way that collective epistemology is overall improved. Instead, the propaganda-fighting AIs will themselves have blind spots, to allow in the propaganda of the “good guys.” The CCP will have their propaganda-fighting AIs, the Western Left will have theirs, the Western Right will have theirs, etc. (I think what happened with the internet is precedent for this. In theory, having all these facts available at all of our fingertips should have led to a massive improvement in collective epistemology and a massive improvement in truthfulness, accuracy, balance, etc. in the media. But in practice it didn’t.) It’s possible I’m being too cynical here of course!
technically my story involves assumptions like the ones you list in the bullet points, but the way you phrase them is… loaded? Designed to make them seem implausible?
I don’t think it’s designed to make them seem implausible? Maybe the first one? Idk, I could say that your story is designed to make them seem plausible (e.g. by not explicitly mentioning them as assumptions).
I think it’s fair to say it’s “loaded”, in the sense that I am trying to push towards questioning those assumptions, but I don’t think I’m doing anything epistemically unvirtuous.
This is already happening in 2021 and previous, in my story it happens more.
This does not seem obvious to me (but I also don’t pay much attention to this sort of stuff so I could be missing evidence that makes it very obvious).
The CCP will have their propaganda-fighting AIs, the Western Left will have theirs, the Western Right will have theirs, etc.
That seems correct. But plausibly the best way for these AIs to fight propaganda is to respond with truthful counterarguments.
I don’t really see “number of facts” as the relevant thing for epistemology. In my anecdotal experience, people disagree on values and standards of evidence, not on facts. AIs that can respond to anti-vaxxers in their own language seem way, way more impactful than what we have now.
(I just tried to find the best argument that GMOs aren’t going to cause long-term harms, and found nothing. We do at least have several arguments that COVID vaccines won’t cause long-term harms. I armchair-conclude that a thing has to get to the scale of COVID vaccine hesitancy before people bother trying to address the arguments from the other side.)
Perhaps I shouldn’t have mentioned any of this. I also don’t think you are doing anything epistemically unvirtuous. I think we are just bouncing off each other for some reason, despite seemingly being in broad agreement about things. I regret wasting your time.
That seems correct. But plausibly the best way for these AIs to fight propaganda is to respond with truthful counterarguments.
I don’t really see “number of facts” as the relevant thing for epistemology. In my anecdotal experience, people disagree on values and standards of evidence, not on facts. AIs that can respond to anti-vaxxers in their own language seem way, way more impactful than what we have now.
The first bit seems in tension with the second bit, no? At any rate, I also don’t see number of facts as the relevant thing for epistemology. I totally agree with your take here.
The first bit seems in tension with the second bit, no?
“Truthful counterarguments” is probably not the best phrase; I meant something more like “epistemically virtuous counterarguments”. Like, responding to “what if there are long-term harms from COVID vaccines” with “that’s possible but not very likely, and it is much worse to get COVID, so getting the vaccine is overall safer” rather than “there is no evidence of long-term harms”.
If you look at my posting history, you’ll see that all posts I’ve made on LW (two!) are negative toward social media and one calls out recommender systems explicitly. This post has made me reconsider some of my beliefs, thank you.
I realized that, while I have heard Tristan Harris, read The Attention Merchants, and perused other, similar sources, I haven’t looked for studies or data to back it all up. It makes sense on a gut level—that these systems can feed carefully curated information to softly steer a brain toward what the algorithm is optimizing for—but without more solid data, I found I can’t quite tell if this is real or if it’s just “old man yells at cloud.”
Subjectively, I’ve seen friends and family get sucked into social media and change into more toxic versions of themselves. Or maybe they were always assholes, and social media just lent them a specific, hivemind kind of flavor, which triggered my alarms? Hard to say.
Subjectively, I’ve seen friends and family get sucked into social media and change into more toxic versions of themselves. Or maybe they were always assholes, and social media just lent them a specific, hivemind kind of flavor, which triggered my alarms? Hard to say.
Fwiw, I am a lot more compelled by the general story “we are now seeing examples of bad behavior from the ‘other’ side that are selected across hundreds of millions of people, instead of thousands of people; our intuitions are not calibrated for this” (see e.g. here). That issue seems like a consequence of more global reach + more recording of bad stuff that happens. Though if I were planning to make it my career I would spend way more time figuring out whether that story is true as well.
This was a good post. I’d bookmark it, but unfortunately that functionality doesn’t exist yet.* (Though if you have any open source bookmark plugins to recommend, that’d be helpful.) I’m mostly responding to say this though:
While it wasn’t otherwise mentioned in the abstract of the paper (above), this was stated once:
This paper examines algorithmic depolarization interventions with the goal of conflict transformation: not suppressing or eliminating conflict but moving towards more constructive conflict.
I though this was worth calling out, although I am still in the process of reading that 10⁄14 page paper. (There are 4 pages of references.)
And some other commentary while I’m here:
It’s common for people to be worried about recommender systems being addictive
I imagine the recommender system is only as good as what it has to work with, content wise—and that’s before getting into ‘what does the recommender system have to go off of’, and ‘what does it do with what it has’.
Whenever I talk to someone who seems to have actually studied the topic in depth, it seems they think that there are problems with recommender systems, but they are different from what people usually imagine.
This part wasn’t elaborated on. To put it a different way:
It’s common for people to be worried about recommender systems being addictive or promoting filter bubbles etc, but as far as I can tell, they don’t have very good arguments for these worries.
Do the people ‘who know what’s going’ on (presumably) have better arguments? Do you?
*I also have a suspicion it’s not being used. I.e., past a certain number of bookmarks like 10, it’s not actually feasible to use the LW interface to access them.
Do the people ‘who know what’s going’ on (presumably) have better arguments?
Possibly, but if so, I haven’t seen them.
My current belief is “who knows if there’s a major problem with recommender systems or not”. I’m not willing to defer to them, i.e. say “there probably is a problem based on the fact that the people who’ve studied them think there’s a problem”, because as far as I can tell all of those people got interested in recommender systems because of the bad arguments and so it feels a bit suspicious / selection-effect-y that they still think there are problems. I would engage with arguments they provide and come to my own conclusions (whereas I probably would not engage with arguments from other sources).
Do you?
No. I just have anecdotal experience + armchair speculation, which I don’t expect to be much better at uncovering the truth than the arguments I’m critiquing.
This might still be good for generating ideas (if not far more accurate than brainstorming or trying to come up with a way to generate models via ‘brute force’).
But the real trick is—how do we test these sorts of ideas?
Agreed this can be useful for generating ideas (and I do tons of it myself; I have hundreds of pages of docs filled with speculation on AI; I’d probably think most of it is garbage if I went back and looked at it now).
We can test the ideas in the normal way? Run RCTs, do observational studies, collect statistics, conduct literature reviews, make predictions and check them, etc. The specific methods are going to depend on the question at hand (e.g. in my case, it was “read thousands of articles and papers on AI + AI safety”).
The incentive of social media companies to invest billions into training competitive RL agents that make their users spend as much time as possible in their platform seem like an obvious reason to be concerned. Especially when such RL agents plausibly already select a substantial fraction of the content that people in developed countries consume.
I don’t trust this sort of armchair reasoning. I think this is sufficient reason to raise the hypothesis to attention, but not enough to conclude that it is likely a real concern. And the data I have seen does not seem kind to the hypothesis (though there may be better data out there that does support the hypothesis).
To be worried about a possibility does not require that the possibility is an actuality.
I am more annoyed by the sheer confidence people have. If they were saying “this is a possibility, let’s investigate” that seems fine.
Re: the rest of your comment, I feel like you are casting it into a decision framework while ignoring the possible decision “get more information about whether there is a problem or not”, which seems like the obvious choice given lack of confidence.
If at some point you become convinced that it is impossible / too expensive to get more information (I’d be really suspicious, but it could be true) then I’d agree you should bias towards worry.
I would guess that the fact that people regularly fail to inhabit the mindset of “I don’t know that this is a problem, let’s try to figure out whether it is actually a problem” is a source of tons of problems in society (e.g. anti-vaxxers, worries that WiFi radiation kills you, anti-GMO concerns, worries about blood clots for COVID vaccines, …). Admittedly in these cases the people are making a mistake of being confident, but even if you fixed the overconfidence they would continue to behave similarly if they used the reasoning in your comment. Certainly I don’t personally know why you should be super confident that GMOs aren’t harmful, and I’m unclear on whether humanity as a whole has the knowledge to be super confident in that.
It’s common for people to be worried about recommender systems being addictive or promoting filter bubbles etc, but as far as I can tell, they don’t have very good arguments for these worries. Whenever I talk to someone who seems to have actually studied the topic in depth, it seems they think that there are problems with recommender systems, but they are different from what people usually imagine.
I’ll go through the articles I’ve read that argue for worrying about recommender systems, and explain why I find them unconvincing. I’ve only looked at the ones that are widely read; there are probably significantly better arguments that are much less widely read.
Aligning Recommender Systems as Cause Area. I responded briefly on the post. Their main arguments and my counterarguments are:
A few sources say that it is bad + it has incredible scale + it should be super easy to solve. (I don’t trust the sources and suspect the authors didn’t check them; I agree there’s huge scale; I don’t see why it should be super easy to solve even if there is a problem, especially given that many of the supposed problems seem to have existed before recommender systems.)
Maybe working on recommender systems would have spillover effects on AI alignment. (This seems dominated by just working directly on AI alignment. Also the core feature of AI alignment is that the AI system deliberately and intentionally does things, and creates plans in new situations that you hadn’t seen before, which is not the case with recommender systems, so I don’t expect many spillover effects.)
80K podcast with Tristan Harris. This was actively annoying for a variety of reasons:
I don’t know what the main claim was. Ostensibly it was meant to be “it is bad that companies have monetized human attention since this leads to lots of bad incentives and bad outcomes”. But then so many specific things mentioned have nothing to do with this claim and instead seem to be a vague general “tech companies are bad”. Most egregiously, in section Global effects [01:02:44], Rob argues “WhatsApp doesn’t have ads / recommender systems, so it acts as a control group, but it too has bad outcomes, doesn’t this mean the problem isn’t ads / recommender systems?” and Tristan says “That’s right, WhatsApp is terrible, it’s causing mass lynchings” as though that supports his point.
When Rob made some critique of the main argument, Tristan deflected with an example of tech doing bad things. But it’s always vaguely related, so you think he’s addressing the critique, even though he hasn’t actually. (I’m reminded of the Zootopia strategy for press conferences.) See sections “The messy real world vs. an imagined idealised world [00:38:20]” (Rob: weren’t negative things happening before social media? Tristan: it’s easy to fake credibility in text), “The persuasion apocalypse [00:47:46]” (Rob: can’t one-on-one conversations be persuasive too? Tristan: you can lie in political ads), “Revolt of the Public [00:56:48]” (Rob: doesn’t the internet allow ordinary people to challenge established institutions in good ways? Tristan: Alex Jones has been recommended 15 billion times.)
US politics [01:13:32] is a rare counterexample, where Rob says “why aren’t other countries getting polarized”, and Tristan replies “since it’s a positive feedback loop only countries with high initial polarization will see increasing polarization”. It’s not a particularly convincing response, but at least it’s a response.
Tristan seems to be very big on “the tech companies changed what they were doing, that proves we were right”. I think it is just as consistent to say “we yelled at the companies a lot and got the public to yell at them too, and that caused a change, regardless of whether the problem was serious or not, or whether the solution was net positive or not”.
The second half of the podcast focuses more on solutions. Given that I am unconvinced about the problem, I wasn’t all that interested, but it seemed generally reasonable.
(This post responds to the object level claims, which I have not done because I don’t know much about the object level.)
There’s also the documentary “The Social Dilemma”, but I expect it’s focused entirely on problems, probably doesn’t try to have good rigorous statistics, and surely will make no attempt at a cost-benefit analysis so I seriously doubt it would change my mind on anything. (And it is associated with Tristan Harris so I’d assume that most of the relevant details would have made it into the 80K podcast.)
Recommender systems are still influential, and you could want to work on them just because of their huge scale. I like Designing Recommender Systems to Depolarize as an example of what this might look like.
Thanks for this Rohin. I’ve been trying to raise awareness about the potential dangers persuasion/propaganda tools, but you are totally right that I haven’t actually done anything close to a rigorous analysis. I agree with what you say here that a lot of the typical claims being thrown around seem based more on armchair reasoning than hard data. I’d love to see someone really lay out the arguments and analyze them… My current take is that (some of) the armchair theories seem pretty plausible to me, such that I’d believe them unless the data contradicts. But I’m extremely uncertain about this.
I should note that there’s a big difference between “recommender systems cause polarization as a side effect of optimizing for engagement” and “we might design tools that explicitly aim at persuasion / propaganda”. I’m confident we could (eventually) do the latter if we tried to; the question is primarily whether we will try to and if we do what it’s effects will be.
Usually, for any sufficiently complicated question (which automatically includes questions about the impact of technologies used by billions of people, since people are so diverse), I think an armchair theory is only slightly better than a monkey throwing darts, so I’m more in the position of “yup, sounds plausible, but that doesn’t constrain my beliefs about what the data will show and medium quality data will trump the theory no matter how it comes out”.
Oh, then maybe we don’t actually disagree that much! I am not at all confident that optimizing for engagement has the side effect of increasing polarization. It seems plausible but it’s also totally plausible that polarization is going up for some other reason(s). My concern (as illustrated in the vignette I wrote) is that we seem to be on a slippery slope to a world where persuasion/propaganda is more effective and widespread than it has been historically, thanks to new AI and big data methods. My model is: Ideologies and other entities have always been using propaganda of various kinds, and there’s always been a race between improving propaganda tech and improving truth-finding tech, but we are currently in a big AI boom and in particular in a Big Data and Natural Language Processing boom, and this seems like it’ll be a big boost to propaganda tech, and unfortunately I can’t think of ways in which it will correspondingly boost truth-finding-ness across society, because while it can be used to make truth-finding tech maybe (e.g. prediction markets, fact-checkers, etc.) it seems like most people in practice just don’t want to adopt truth-finding tech. It’s true that we could design a different society/culture that used all this awesome new tech to be super truth-seeking and have a very epistemically healthy discourse, but it seems like we are not about to do that anytime soon, instead we are going in the opposite direction.
I think that story involves lots of assumptions I don’t immediately believe (but don’t disbelieve either):
People are very deliberately building persuasion / propaganda tech (as opposed to e.g. people like to loudly state opinions and the persuasive ones rise to the top)
Such people will quickly realize that AI will be very useful for this
They will actually try to build it (as opposed to e.g. raising a moral outcry and trying to get it banned)
The resulting AI system will in fact be very good at persuasion / propaganda
AI that fights persuasion / propaganda either won’t be built or will be ineffective (my unreliable armchair reasoning suggests the opposite; it seems to me like right now human fact-checking labor can’t keep up with human controversy-creating labor partly because humans enjoy the latter more than the former; this won’t be true with AI)
And probably there are a bunch of other assumptions I haven’t even thought to question.
I think it seems fine to raise the possibility and do more research (and for all I know CSET or GovAI has done this research) but at least under my beliefs the current action should not be “raise awareness”, it should be “figure out whether the assumptions are justified”.
That’s all I’m trying to do at this point, to be clear. Perhaps “raise awareness” was the wrong choice of phrase.
Re: the object-level points: For how I see this going, see my vignette, and my reply to steve. The bullet points you put here make it seem like you have a different story in mind. [EDIT: But I agree with you that it’s all super unclear and more research is needed to have confidence in any of this.]
Excellent :)
(Link is broken, but I found the comment.) After reading that reply I still feel like it involves the assumptions I mentioned above.
Maybe your point is that your story involves “silos” of Internet-space within which particular ideologies / propaganda reign supreme. I don’t really see that as changing my object-level points very much but perhaps I’m missing something.
I was confusing, sorry—what I meant was, technically my story involves assumptions like the ones you list in the bullet points, but the way you phrase them is… loaded? Designed to make them seem implausible? idk, something like that, in a way that made me wonder if you had a different story in mind. Going through them one by one:
People are very deliberately building persuasion / propaganda tech (as opposed to e.g. people like to loudly state opinions and the persuasive ones rise to the top)
This is already happening in 2021 and previous, in my story it happens more.
Such people will quickly realize that AI will be very useful for this
Again, this is already happening.
They will actually try to build it (as opposed to e.g. raising a moral outcry and trying to get it banned)
Plenty of people are already raising a moral outcry. In my story these people don’t succeed in getting it banned, but I agree the story could be wrong. I hope it is!
The resulting AI system will in fact be very good at persuasion / propaganda
Yep. I don’t have hard evidence, but intuitively this feels like the sort of thing today’s AI techniques would be good at, or at least good-enough-to-improve-on-the-state-of-the-art.
AI that fights persuasion / propaganda either won’t be built or will be ineffective (my unreliable armchair reasoning suggests the opposite; it seems to me like right now human fact-checking labor can’t keep up with human controversy-creating labor partly because humans enjoy the latter more than the former; this won’t be true with AI)
I think it won’t be built & deployed in such a way that collective epistemology is overall improved. Instead, the propaganda-fighting AIs will themselves have blind spots, to allow in the propaganda of the “good guys.” The CCP will have their propaganda-fighting AIs, the Western Left will have theirs, the Western Right will have theirs, etc. (I think what happened with the internet is precedent for this. In theory, having all these facts available at all of our fingertips should have led to a massive improvement in collective epistemology and a massive improvement in truthfulness, accuracy, balance, etc. in the media. But in practice it didn’t.) It’s possible I’m being too cynical here of course!
I don’t think it’s designed to make them seem implausible? Maybe the first one? Idk, I could say that your story is designed to make them seem plausible (e.g. by not explicitly mentioning them as assumptions).
I think it’s fair to say it’s “loaded”, in the sense that I am trying to push towards questioning those assumptions, but I don’t think I’m doing anything epistemically unvirtuous.
This does not seem obvious to me (but I also don’t pay much attention to this sort of stuff so I could be missing evidence that makes it very obvious).
That seems correct. But plausibly the best way for these AIs to fight propaganda is to respond with truthful counterarguments.
I don’t really see “number of facts” as the relevant thing for epistemology. In my anecdotal experience, people disagree on values and standards of evidence, not on facts. AIs that can respond to anti-vaxxers in their own language seem way, way more impactful than what we have now.
(I just tried to find the best argument that GMOs aren’t going to cause long-term harms, and found nothing. We do at least have several arguments that COVID vaccines won’t cause long-term harms. I armchair-conclude that a thing has to get to the scale of COVID vaccine hesitancy before people bother trying to address the arguments from the other side.)
Perhaps I shouldn’t have mentioned any of this. I also don’t think you are doing anything epistemically unvirtuous. I think we are just bouncing off each other for some reason, despite seemingly being in broad agreement about things. I regret wasting your time.
The first bit seems in tension with the second bit, no? At any rate, I also don’t see number of facts as the relevant thing for epistemology. I totally agree with your take here.
“Truthful counterarguments” is probably not the best phrase; I meant something more like “epistemically virtuous counterarguments”. Like, responding to “what if there are long-term harms from COVID vaccines” with “that’s possible but not very likely, and it is much worse to get COVID, so getting the vaccine is overall safer” rather than “there is no evidence of long-term harms”.
If you look at my posting history, you’ll see that all posts I’ve made on LW (two!) are negative toward social media and one calls out recommender systems explicitly. This post has made me reconsider some of my beliefs, thank you.
I realized that, while I have heard Tristan Harris, read The Attention Merchants, and perused other, similar sources, I haven’t looked for studies or data to back it all up. It makes sense on a gut level—that these systems can feed carefully curated information to softly steer a brain toward what the algorithm is optimizing for—but without more solid data, I found I can’t quite tell if this is real or if it’s just “old man yells at cloud.”
Subjectively, I’ve seen friends and family get sucked into social media and change into more toxic versions of themselves. Or maybe they were always assholes, and social media just lent them a specific, hivemind kind of flavor, which triggered my alarms? Hard to say.
Thanks, that’s good to hear.
Fwiw, I am a lot more compelled by the general story “we are now seeing examples of bad behavior from the ‘other’ side that are selected across hundreds of millions of people, instead of thousands of people; our intuitions are not calibrated for this” (see e.g. here). That issue seems like a consequence of more global reach + more recording of bad stuff that happens. Though if I were planning to make it my career I would spend way more time figuring out whether that story is true as well.
This was a good post. I’d bookmark it, but unfortunately that functionality doesn’t exist yet.* (Though if you have any open source bookmark plugins to recommend, that’d be helpful.) I’m mostly responding to say this though:
While it wasn’t otherwise mentioned in the abstract of the paper (above), this was stated once:
I though this was worth calling out, although I am still in the process of reading that 10⁄14 page paper. (There are 4 pages of references.)
And some other commentary while I’m here:
I imagine the recommender system is only as good as what it has to work with, content wise—and that’s before getting into ‘what does the recommender system have to go off of’, and ‘what does it do with what it has’.
This part wasn’t elaborated on. To put it a different way:
Do the people ‘who know what’s going’ on (presumably) have better arguments? Do you?
*I also have a suspicion it’s not being used. I.e., past a certain number of bookmarks like 10, it’s not actually feasible to use the LW interface to access them.
Possibly, but if so, I haven’t seen them.
My current belief is “who knows if there’s a major problem with recommender systems or not”. I’m not willing to defer to them, i.e. say “there probably is a problem based on the fact that the people who’ve studied them think there’s a problem”, because as far as I can tell all of those people got interested in recommender systems because of the bad arguments and so it feels a bit suspicious / selection-effect-y that they still think there are problems. I would engage with arguments they provide and come to my own conclusions (whereas I probably would not engage with arguments from other sources).
No. I just have anecdotal experience + armchair speculation, which I don’t expect to be much better at uncovering the truth than the arguments I’m critiquing.
This might still be good for generating ideas (if not far more accurate than brainstorming or trying to come up with a way to generate models via ‘brute force’).
But the real trick is—how do we test these sorts of ideas?
Agreed this can be useful for generating ideas (and I do tons of it myself; I have hundreds of pages of docs filled with speculation on AI; I’d probably think most of it is garbage if I went back and looked at it now).
We can test the ideas in the normal way? Run RCTs, do observational studies, collect statistics, conduct literature reviews, make predictions and check them, etc. The specific methods are going to depend on the question at hand (e.g. in my case, it was “read thousands of articles and papers on AI + AI safety”).
The incentive of social media companies to invest billions into training competitive RL agents that make their users spend as much time as possible in their platform seem like an obvious reason to be concerned. Especially when such RL agents plausibly already select a substantial fraction of the content that people in developed countries consume.
I don’t trust this sort of armchair reasoning. I think this is sufficient reason to raise the hypothesis to attention, but not enough to conclude that it is likely a real concern. And the data I have seen does not seem kind to the hypothesis (though there may be better data out there that does support the hypothesis).
[Deleted]
I am more annoyed by the sheer confidence people have. If they were saying “this is a possibility, let’s investigate” that seems fine.
Re: the rest of your comment, I feel like you are casting it into a decision framework while ignoring the possible decision “get more information about whether there is a problem or not”, which seems like the obvious choice given lack of confidence.
If at some point you become convinced that it is impossible / too expensive to get more information (I’d be really suspicious, but it could be true) then I’d agree you should bias towards worry.
I would guess that the fact that people regularly fail to inhabit the mindset of “I don’t know that this is a problem, let’s try to figure out whether it is actually a problem” is a source of tons of problems in society (e.g. anti-vaxxers, worries that WiFi radiation kills you, anti-GMO concerns, worries about blood clots for COVID vaccines, …). Admittedly in these cases the people are making a mistake of being confident, but even if you fixed the overconfidence they would continue to behave similarly if they used the reasoning in your comment. Certainly I don’t personally know why you should be super confident that GMOs aren’t harmful, and I’m unclear on whether humanity as a whole has the knowledge to be super confident in that.