Opinion piece on the Swedish Network for Evidence-Based Policy
Cross-posted from the Effective Altruism Forum
Follow up to: The effectiveness-alone strategy and evidence-based policy
A translation of the opinion piece can be found here.
I
Effective altruism is a great concept, but it’s not trivial to sell. There are therefore good reasons to ally ourselves with other rationalist memes to increase the level of rationality and effectiveness in the world. One powerful such rationalist meme is “evidence-based policy”, which is inspired by the “evidence-based medicine” movement.
The exact meaning of evidence-based policy is somewhat disputed, but generally proponents of evidence-based policy demand that the standards on which policy is based should be raised. Many believe strongly in randomized control trials (RCTs) and in the “hierarchy of evidence”, but there is not complete agreement on the strength of RCTs relative to other kinds of studies.
In the US and the UK, there are several organizations which work on evidence-based policy, such as the British What Works Network and the American Coalition for Evidence-Based Policy. Inspired by them, I took the initative to start a Swedish network for evidence-based policy at the start of this year. We are by now around 50 (depending on how you count) researchers, civil servants, journalists, consultants, students and other activists in the network. Only myself and a few others are EA members, so it’s not an EA organization, but as I argued in my previous post, I do believe working on this nevertheless is an effectively altruistic cause.
One difference between us and What Works is that we aim to be a broad campaigning organization. We believe that policy not being evidence-based is not only due to a lack of knowledge, but also due to a lack of will, especially among politicians. Politicians often disregard expert advice (on what policies are the most effective to reach a given set of goals) which goes against their political prejudices. Therefore we need to put pressure on politicians—not the least in the media—rather than just work behind the scenes as an expert organization.
II (Most linked replies below are in Swedish)
Our activities were fairly modest until last Sunday, when we wrote an opinion piece calling for evidence-based policy (English). The opinion piece was published in the most widely-read broadsheet, Dagens Nyheter, on DN Debatt—a sort of op-ed forum. DN Debatt has a special standing in Swedish politics. Everybody reads it and it’s well-respected.
Hence we had expected a lot of attention, but the results still exceeded them. Ours was the second most shared DN Debatt-article in the month of May. We got seven replies in Dagens Nyheter, were strongly criticized in the other main broadsheet, Svenska Dagbladet (conservative), parodied in a popular public service (equivalent of BBC) podcast, and were also commented on in a number of smaller newspapers. The discussion on Twitter was pretty intense. Subsequently, we also published two replies to replies in Dagens Nyheter and Svenska Dagbladet.
It’s hard to tell what the majority opinion on our piece was. Certainly, there was a lot of praise and a lot of Facebook likes, but also some fierce criticism. This was almost exclusively down to misunderstandings. I won’t bog you down with all of the details here, but will rather summarize my general conclusions. They could be useful for anyone trying to write on evidence-based policy or related concepts in other countries.
I should say that “evidence-based policy” isn’t as entrenched a concept in Sweden as it is in the US and the UK, which probably played to our disadvantage.
1) You need to be very clear over the means-ends-distinction. Evidence-based policy is about making the methods for reaching your political goals (happiness, equality, liberty, etc) more effective by the use of evidence. It is not about propagating any particular set of political goals. We tried to be clear about this, but partly failed for two reasons. Firstly, Dagens Nyheter set the headline, which was misleading. Second, we only clarified this distinction at the end. It should have been at the top.
2) There is a straw man conception of evidence-based policy, or expert-informed policy and political rationality more generally, akin to Julia Galef’s “Straw Vulcan” conception of rationality. Perhaps this varies a bit from country to country, but in Sweden it’s strong. Let’s call it “Straw Soviet” for now (please come with suggestions!).
According to this conception, evidence-based policy means technocracy (of a dictatorial form, according to the more extreme interpretations), disregard of non-quantifiable values (cf “Straw Vulcan”), disregard of emotions, “Mad Scientist”-conception of society as a labratory, etc, etc. You need to everything you can to counter such interpretations. I certainly underestimated the power of this straw man meme. I should also say that the Straw Soviet is probably more vicious than the Straw Vulcan, who seems more innocent (perhaps this is partly down to Julia’s plafyul presentation of it, though).
For instance, Svenska Dagbladet’s criticism was all about the “Straw Soviet”. We were said to want to “design voter behaviour” (this was also partly due to the article having been signed by a few nudgers who call themselves “behavioural engineers”—a big trigger of the Straw Soviet). Here are some more quotes:
It is perhaps not the “enlightened despot” who is called for in the opinion piece, but rather Dr Despot. Today’s most frightening reading came from the recently formed “Network for Evidence-Based Policy” (Dagens Nyheter 1 June).
//
Since there probably are very few citizens who base their votes on research reports, free elections yields results which are not evidence-based. According to the argument in the opinion piece, that means that since we “see the world through partisan lenses”, the election results are as a rule problematic or directly harmful.
//
Now if the network were correct, true evidence-based policies would lead to a single proposal, a solution “free from ideology and populism”. That would in turn mean that all parties arrived at the same answer, and it is absolutely impossible why that – though ever so full of evidence – would be desirable.
A vibrant democracy is based on the existence of conflicts of opinion and value, intellectual diversity and the citizen’s right to freely express it. The complete and rational citizen is an anomaly, and based on the unpleasant idea that enlightened powers can raise, design, a new man.
Paradoxically, it is precisely highly ideological regimes which have attempted just that. The results have been devastating.
We got several other replies along these lines, though we also got a much more positive one from Dagens Nyheter itself. A large group of replies treated more technical and hum-drum issues concerning RCTs, practical policy-making, etc.
3) Connected to the Straw Vulcan and the Straw Soviet, there is a “Straw Naive Positivist Scientist” (again, suggestions for better terms are welcome), who thinks that knowledge is easily obtainable even in messy fields like economics, that it’s easy to reach consensus if you just don’t lead political misconceptions mislead you, that you always easily can infer policy-advice from research, etc. We got a lot of criticism which was based on the Straw Naive Positivist. Obviously, we don’t hold any of those views.
4) People read very superficially. This is not only true of the man in the street, but also of many journalists, politicians, etc. At some level I know this, having myself written about research on this on my blog, but it’s harder to make full use of that knowledge when you write.
Also lots of people don’t use the principle of charity at all. Some of the replies—including one from a philosophy professor—were exceedingly uncharitable. Thus don’t expect people to use the principle of charity—especially when emotional memes like the Straw Soviet are around.
When you fight such powerful memes, you need to be extremely clear. You need to say the things you really want to get across early, to repeat them, and to give examples. If at all possible, you should control the title, since that sets so much of the tone of the piece (give the publishers a juicy suggestion and they might buy it). Don’t say too much, but focus on getting the central message across.
This is so different from writing an academic paper. Of course that’s obvious, but it’s one thing to get it on an intellectual level, quite another to really internalize it. If you could get a skilled public communicator on board, that would be very useful.
I also think it would be good to pre-test major articles (e.g. on Mechanical Turk) to get a clearer picture of whether the message gets across. If you don’t want the content to leak beforehand, that might not be doable, though.
5) We were probably a bit too extreme regarding RCTs, which triggered the Straw Soviet and the Straw Naive Positivist (for epistemological and ethical reasons). It would have been more tactical to emphasize other stuff.
6) We would have come off as more concrete if we had based our opinion piece on a research report on the state of Swedish policy-making. It’s great if you can do that, but I don’t think it would have been rational for us (see below).
7) We should have stressed how big the movement on evidence-based policy is in the US and the UK. For instance, we could have mentioned that “Obama’s 2016 budget calls for an emphasis on evidence-based approaches at all levels of government”. Obama being popular and respected in Sweden, that would have done much to disarm the Straw Soviet.
8) It was a mistake to mention legal means as a way of making politics more evidence-based, since it strongly triggers the Soviet meme. Even those who otherwise supported us criticized this suggestion.
III
In our replies, we focused on rectifying the misunderstandings, focusing on the claim that we are calling for “Dr Despot”. These replies normally gets much less attention, and so it was with ours as well. However, the reception also was more unanimously positive, especially from academics and civil servants who know the field.
I don’t regret writing this opinion piece at this early stage. Before I started writing it (I wrote the body of the text, and the others then made minor tweaks) there wasn’t much activity in our network. Now, we have many more members, including more senior ones. Also, those who already were in the network grew much more enthusiastic after the publication. Thus all-in-all it’s been a major success. Still, I think you can learn a lot from things we could have done better.
I’ll write more later on how the network is developing more generally. Also I should add that I’m still digesting what I’ve learnt, so my conclusions aren’t set in stone. Any comments are welcome.
When it comes to someone calling for a political change, it doesn’t make sense to apply the principle of charity based on a best case interpretation of the outcome but to focus on the likely outcome. Political ideas shouldn’t be accepted on the ground that they are well intentioned.
Whether or not pushes for more evidence based policy making lead to more technocrats is an empirical question.
You claim:
You fail to provide evidence for that claim. You fail to demostrate that countries that adopted evidence-based policy did better on international school ratings.
In practice the call for evidence-based policy in education often means to focus on improving standardized test scores. It’s no strawman. It means to not give schools the freedom to make their own decisions about how to teach but to task them to top-down to teach in a certain way.
I think you’re being way to cavalier about dismissing the “Straw Soviet” objection. The Soviet-style technocrats really did exist and they believed themselves to be doing the same thing you believe yourself to be doing.
Furthermore, the fact that you appear more concerned with memeticly engineering away objections then addressing their substance (or even admitting they have substance) is an even bigger red flag then the things you listed.
This is a misuse of the principal of charity. The principal of charity is applicable when someone is arguing for a logical proposition and might gloss over parts of the argument. When someone is making proposals for a course of action, that they’ll presumably be in charge of implementing, you want to make damn sure he knows what he’s doing.
Think about it this way, would you apply the principal of charity to a surgeon who says something that ambiguously suggest he believes in some discredited theory about how the body works? Especially if there had recently been a scandal involving patients dying at the hands of surgeons who believed that theory.
I think it’s great to see a member of this forum doing something to make the world a more rational place. I’d love to hear more about what other members are doing.
For many people working on policy related work, it’s not always possible to discuss details publicly until after the projects are complete. (And sometimes not even then.)
That said, I’m trying to do my part.
As a side point, of possible interest to the author, Coalition4Evidence was subsumed under http://www.arnoldfoundation.org/ - per this announcement. http://coalition4evidence.org/wp-content/uploads/2015/04/Coalition-Board-of-Advisors-Update-04-24-15.pdf
Other may be interested in the fact that the EU has done some real work in this direction, at least for childhood and family interventions, under the heading of “evidence based practices”; http://europa.eu/epic/topics/evidence-based-practices_en.htm
Great post, thank you! I love your work and have several comments and questions. First of all, is there something like your organisation in Germany?
I don’t know about Sweden, but in Germany politicians are deluged with policy suggestions from all manner of parties, and that includes innocuous-looking non-profits with lofty goals that are actually sockpuppets for lobbyists. You pattern match pretty well to the latter. I’m highly confident many seasoned politicians’ instinct will be to listen to you politely, agree with lots of things you say, and then do nothing and never call back because you resemble untrustworthy players so closely. Are you aware of this possibility, and how are you adressing it?
I suspect you’d gain a lot from getting some civil servants and administration workers on board. There got to be some that have been vocal about government waste in the past. These might be attracted to you because their concerns are part of yours, and they make great allies. If you can’t find them, ask at civil servants unions.
Have you looked into talking to partys’ youth organisations, rather than experienced politicians, in order to change mindsets before they’re too resistant to change and get the next genetration of politicians if you can’t get the current one?
What is your relation to the Swedish Pirate Party?
Politicians do listen to lobbyists when they have policy expertise.
In this case I think the major issue would be that I politician cares about specifics. If you just say: “I want more evidence-based policy without any specifics”, you get pattern matched to being naive.
If you on the other hand say: “I really like the evidence-based policy initiative they do in country X and want this here. The initiative was popular in X. There were issues Y and Z and they fixed it by doing A. Of course we might to change B to implement the policy here.”, then you are talking. You have to address concerns instead of calling them strawmen.
To be a valuable discussion partner for the politician you have to give the impression that you understand the relevant policy tools better than the politician.
It’s also worth thinking about if you can find a case where doing RCT’s is good politics. RCT could be useful as a way of getting rid of responsibility. If you can find a policy issue where a politician would be able to avoid blame for an issue by delegating the issue to RCT the politician might be willing to listen.
You make good points as usual.
The politicians I know will certainly prefer to talk about about very specific subjects, but they will also want to choose the experts to talk to about them. Being chosen as a conversation partner for expertise is usually a matter of having been recommended by someone else. The same seems to be true for journalists.
There are a couple of people who know both the evidence-based scientific view of the world and the balancing of interest based political view of the world: economists, CEOs, political scientists. But that’s a fairly small overlap between two way larger spheres of people. Getting these two spheres to understand each other better is a project not for years but for decades.
Reaching out to a parliamentarian in Berlin isn’t hard even if you don’t have connections. Politicians want to talk to constituents while journalists don’t have the same goal of talking to readers.
Of course that only get’s you the foot in the door. Recommendations certainly have advantages.
In Berlin it also doesn’t take more than being a partner member to be able to go to political party meetings. The main factor that holds people out is that they are boring.
Sure, reaching out is easy. All the politicians I know are very friendly people who love conversation.
But this evidence-based policy thing is about not just voicing an individual voter’s concern. And it is not even about building pressure to change this policy or that, which the politician in question might actually have a tiny bit of influence over. It is about changing the whole framework in which policies are chosen. A framework that all the politician’s friends are comfortable enough with. And this comes with an undertone of “this is so obviously beneficial that you’re a fool if you don’t do this”.
For this you don’t need the foot in the door, you need to get through the door and bring lots of friends, a ton of evidence, and a bank. Before you have at least some of that, putting feet into doors and listening to very polite rejections is likely a distraction, and the last thing young movements need are distractions.
In the way the article is formulated it does come with that undertone. But I don’t think you have to have that undertone when you communicate about the subject.
If you want to change public policy understanding the way politicians think about the issue is important. Quite many activists make a lot of strategic errors because they haven’t invested the effort to understand the positions of various stakeholders.
Thanks! I’m sure Germany and Sweden aren’t that different in this regard.
There are of course many expert groups inside and outside government working on what in a broad sense could be called “evidence-based policy” in most countries (whether or not that very term is used). Most of these groups have the problem that you mention—they have good advice, but the politicians don’t listen to them. That’s why we don’t want to be another group of that kind, but also a campaigning organization. We want to push the message of political rationality fairly aggressively in the media and on social media (though you have to be careful not to trigger the “Straw Soviet”). That way, we’ll try to force politicians’ to listen to ours and other expert organizations’ advice.
I don’t know if there is any similar organization in Germany, though I think there should be.
You’re absolutely right about getting civil servants onboard. A few has joined since we wrote the article, and we are working on recruiting more. We lack a bit of practical know-how now, since most of us are academics or students, but I’m confident we will be able to get more people onboard.
Yes, younger politicians are probably more interested in this than older, for several reasons. We are not affiliated with any party, but are independent. We do talk to the parties, though—and the Pirate Party is among them.
I’ll talk about this on Saturday in Berlin, by the way, at the LW meeting. Will you be there?
Yup. See you there.
Check out http://europa.eu/epic/topics/evidence-based-practices_en.htm—it’s EU, not country specific, but it’s heading in that direction.
Heck there’s a fairly obvious explanation for Sweden’s recent fall in international school rankings. Heck evidence-based-policy might even have noticed it, except there is a strong taboo in Sweden against looking at the relevant evidence.
Not being an expert on Swedish culture, I am interested to hear this obvious explanation. Could you be more explicit?
I tried to guess, but there are too many options. First, you have to choose whether to blame it on feminists or immigrants. Second, how specifically.
Immigrants. Specifically, I’m guessing a large part of the decline will go away once you restrict to ethnic Swedes.
I’d guess it’s the immigrants he refers to. It’s not the case, though. Here’s a link in Swedish, and one in English (less informative, I’d think). Children of immigrants do perform worse in school than native children, but that can explain only a fraction of Sweden’s fall in the PISA ranking.
Your English link is broken. The correct link is here.
I don’t see that it addresses the claim at all.
Not sure if I understand it correctly, here is my understanding of this summary.
1+2: An important factor is at which age did the immigrant child arrive to Sweden. If they arrived younger than 10 years (4th grade), they will be okay at school. If older than 10 years, they will have problems.
3: Girls outperform boys; this effect is for both native and immigrant children.
4: Not all immigrants are the same. Children of Western, East European and Asian origins perfom better, children of African, Middle Eastern and Asian origins perform worse. (Seems to me they mentioned Asian children twice here.)
5: In mathematics the disadvantage of the immigrants is smaller, so it seems the performance problems are with Sweden-specific topics.