I do not like this post. I think it gets most of its rhetorical oomph from speaking in a very moralizing tone, with effectively no data, and presenting everything in the worst light possible; I also think many of its claims are flat-out false. Let’s go through each point in order.
1. You can excuse anything by appealing to The Incentives
No, seriously—anything. Once you start crying that The System is Broken in order to excuse your actions (or inactions), you can absolve yourself of responsibility for all kinds of behaviors that, on paper, should raise red flags. Consider just a few behaviors that few scientists would condone:
Fabricating data or results
Regulary threatening to fire trainees in order to scare them into working harder
Deliberately sabotaging competitors’ papers or grants by reviewing them negatively
Wow, that would be truly shocking; indeed that would be truly an indictment of academia. What’s the evidence?
When Diederik Stapel confessed to fabricating the data used in over 50 publications, he didn’t explain his actions by saying “oh, you know, I’m probably a bit of a psychopath”; instead, he placed much of the blame squarely on The Incentives:
… Did you expect people who were caught doing bad things to say “yup, I’m a terrible person?” Also, a single extreme case does not make the rule. I can only assume you’re going to say that all the academics accepted this excuse; that would be an indictment of academia...
Curiously, I’ve never heard any of my peers—including many of the same people who are quick to invoke The Incentives to excuse their own imperfections—seriously endorse The Incentives as an acceptable justification for Stapel’s behavior. In Stapel’s case, the inference we overwhelmingly jump to is that there must be something deeply wrong with Stapel, seeing as the rest of us also face the same perverse incentives on a daily basis, yet we somehow manage to get by without fabricating data.
… I guess not. Well, point 1 is deeply unconvincing. (I also think it is false, just from my own experience in academia.) Let’s hope the others do better.
2. It would break the world if everyone did it
I abhor arguments of this form; I don’t think I’ve ever seen one used well. It would also break the world if everyone ate food but didn’t farm; are you going to become a farmer now?
The actual content of the section is a bit different from the heading:
To be clear, I’m not saying perverse incentives never induce bad behavior in medicine or other fields. Of course they do. My point is that practitioners in other fields at least appear to have enough sense not to loudly trumpet The Incentives as a reasonable justification for their antisocial behavior—or to pat themselves on the back for being the kind of people who are clever enough to see the fiendish Incentives for exactly what they are.
… So academics are bad because they know that the bad incentives are there, and are willing to talk about them? Do you prefer the world in which they don’t know the incentives are there or don’t talk about them but still act in accordance with them? What?
we do seem to have collectively developed a rather powerful form of learned helplessness that doesn’t seem to be matched by other communities. Which is a fortunate thing, because if every other community also developed the same attitude, we would be in a world of trouble.
If true, that seems bad. I think it is mostly false; no evidence was given in the post. I feel like the author just sees more of academia than other fields; I don’t get the sense that doctors, lawyers, journalists etc. are valiantly fighting off the bad incentives.
Okay, well, onwards to point 3:
3. You are not special
[...]
Well yeah, people actually do suffer. There are many scientists who are willing to do the right things—to preregister their analysis plans, to work hard to falsify rather than confirm their hypotheses, to diligently draw attention to potential confounds that complicate their preferred story, and so on. When you assert your right to opt out of these things because apparently your publications, your promotions, and your students are so much more important than everyone else’s, you’re cheating those people.
This seems to be an argument that most of the academics are truthful, and only a few bad actors are ruining it for everyone else. I certainly agree that if there are a few bad actors violating academic norms, those people are doing something bad / should be punished. But I thought the claim was that nearly all academics are doing bad things? If so, you can’t reasonably argue that these people are being “special”.
You can’t be willfully blind to norms and then claim that people following the norms are bad and think they are “special”.
Let’s say you want to smoke pot, and you’re in an area where there’s a law against smoking pot. However, no one enforces it and everyone smokes pot all the time. Perhaps if everyone stopped smoking pot, you could make the case that smoking pot isn’t addictive and get the law overturned. It is incorrect to blame everyone else for smoking pot, or to say that they are defecting, even though it is true that the law could be reversed if everyone stopped smoking pot.
If the claim is actually that there are only a few bad actors, then I assume the author must be talking about the especially bad cases like fabricating data. I expect that when caught such cases really are shamed, and you can’t get out of it by saying “Oh, The Incentives”.
4. You (probably) have no data
It’s telling that appeals to The Incentives are rarely supported by any actual data. It’s simply taken for granted that engaging in the practice in question would be detrimental to one’s career.
This is a clear isolated demand for rigor. (Ironically, this post itself has no data, making it even clearer than usual.)
Coming by the kind of data you’d need to answer this question is actually not that easy: it’s not enough to reflexively point to, say, the fact that some journals have higher impact factors than others, To identify the utility-maximizing course of action, you’d need to integrate over both benefits and costs, and the costs are not always so obvious.
I can only assume that the author decides when to shower, what to eat, what transit to use, etc by simulating out all the possible consequences of each possible decision several years into the future, aggregating the consequences according to their personal utility function, and then choosing the one that is best. I must applaud the author for their immense brainpower; unfortunately us mere mortals can’t do this ourselves and need to rely on heuristics.
Sarcasm aside; I think it is perfectly possible to make good decisions without having all the available data. We do this all the time in lots of different scenarios. Why expect that people can’t do it here as well?
5. It (probably) won’t matter anyway
Outcomes in academia are multiply determined and enormously complex. You can tell yourself that getting more papers out faster will get you a job if it makes you feel better, but that doesn’t make it true. If you’re a graduate student on the job market these days, I have sad news for you: you’re probably not getting a tenure-track job no matter what you do. It doesn’t matter how many p-hacked papers you publish, or how thinly you slice your dissertation into different “studies”; there are not nearly enough jobs to go around for everyone who wants one.
Conditions of extreme competition over scarce resources are exactly where you expect the most conformance to the incentives.
Suppose you’re right, and your sustained pattern of corner-cutting is in fact helping you get ahead. How far ahead do you think it’s helping you get? Is it taking you from a 3% chance of getting a tenure-track position at an R1 university to an 80% chance? Almost certainly not. Maybe it’s increasing that probability from 7% to 11%; that would still be a non-trivial relative increase, but it doesn’t change the fact that, for the average grad student, there is no full-time faculty position waiting at the end of the road.
I’ll note that these are once again claims with no data. I think it’s false for the people who follow The Incentives who do become professors; I would not be surprised if they went from 20% to 80%, which seems like a substantial change. I agree that on average it’s probably not very high, because a lot of people are not going to become professors regardless (i.e. maybe they go from 2% to 3%), but it’s hard to tell from the inside which of these situations you’re in.
6. You’re (probably) not going to “change things from the inside”
Over the years, I’ve talked to quite a few early-career researchers who have told me that while they can’t really stop engaging in questionable research practices right now without hurting their career, they’re definitely going to do better once they’re in a more established position.
[...]
I can think of at least a half-dozen people off-hand who’ve regaled me with me some flavor of “once I’m in a better position” story, and none of them, to my knowledge, have carried through on their stated intentions in a meaningful way.
I totally agree with this point.
7. You’re not thinking long-term
[...] One thing that I think has been largely overlooked in discussions about the current incentive structure of science is what impact the replication crisis will have on the legacies of a huge number of presently famous scientists.
I’ll tell you what impact it will have: many of those legacies will be completely zeroed out. And this isn’t just hypothetical scaremongering. It’s happening right now to many former stars of psychology (and, I imagine, other fields I’m less familiar with).
[...] So if your justification for cutting corners is that you can’t otherwise survive or thrive in the present environment, you should consider the prospect—and I mean, really take some time to think about it—that any success you earn within the next 10 years by playing along with The Incentives could ultimately make your work a professional joke within the 20 years after that.
I am pretty uncertain about this point. Most people do not become famous and have their work scrutinized in detail. I agree that long-term considerations do point in the direction of bucking the incentives; I just don’t know how strongly.
8. It achieves nothing and probably makes things worse
(The “It” here is “complaining about incentives”, not “following the incentives”.)
If your complaints are achieving anything at all, they’re probably actually making things worse by constantly (and incorrectly) reminding everyone around you about just how powerful The Incentives are. Here’s a suggestion: maybe try not talking about The Incentives for a while.
I don’t really see the point here. Surely fixing the problem involves first creating common knowledge about the problem? But in any case, this seems to be orthogonal to the main point (that following the incentives is bad), so I’m going to skip it.
9. It’s your job
This last one seems so obvious it should go without saying, but it does need saying, so I’ll say it: a good reason why you should avoid hanging bad behavior on The Incentives is that you’re a scientist, and trying to get closer to the truth, and not just to tenure, is in your fucking job description. Taxpayers don’t fund you because they care about your career; they fund you to learn shit, cure shit, and build shit.
This same complaint can be leveled at most of society. You don’t get to choose a job description and then have people faithfully execute that job description; what you get depends on the incentives you give.
Would the world be better if you could just give a job description and have people faithfully do that? Probably. Does the world work that way? No. Just yelling “I want you to do X, therefore you should do X” seems counterproductive and willfully blind about how the world works.
----
I do not know what people get out of this post; the arguments given do not remotely support the conclusions, with a couple of exceptions. If I had to guess some effects that would make people upvote / nominate the post:
Powerfully suggested that people should bear responsibility for bucking the incentives
Written as a rant, and so was fun to read
Anti-academia, which agrees with the reader’s beliefs
Pro-principles, which agrees with the reader’s beliefs / morals
(I am not claiming that people consciously thought about these reasons.)
I’ll ignore 2, 3, and 4; I think it’s obvious why those are not reasons we want to promote on LW.
I think the first one could plausibly be a reason that we would want to promote this on LW. Unfortunately, I think it is wrong: I do not think that people should usually feel upon themselves the burden of bucking bad incentives. There are many, many bad incentives in the world; you cannot buck them all simultaneously and make the world a better place. Rather, you need to conform with the bad incentives, even though it makes your blood boil, and choose a select few areas in which you are going to change the world, and focus on those.
(Tbc, this is not a recommendation to p-hack; that is not the norm in academia. The norm in academia was to do other questionable things that in effect constitute p-hacking, but it was never to explicitly p-hack. And that norm might be changing at this point, I don’t know. Personally, while there are questionable academic practices that I follow when writing papers, such as including all the pros of a proposal in the introduction but ignoring the cons, I try to make my experiments as un-p-hacked as I can, though I have the advantage of not seeking a job in academia, as well as a good network, so incentives affect me less than they do most academics.)
I agree with most of this review, and also didn’t really like this post when it came out.
I think the first one could plausibly be a reason that we would want to promote this on LW. Unfortunately, I think it is wrong: I do not think that people should usually feel upon themselves the burden of bucking bad incentives. There are many, many bad incentives in the world; you cannot buck them all simultaneously and make the world a better place. Rather, you need to conform with the bad incentives, even though it makes your blood boil, and choose a select few areas in which you are going to change the world, and focus on those.
Just for the record, and since I think this is actually an important point, my perspective is that indeed people cannot take on themselves the burden of bucking bad all bad incentives, but that there are a few domains of society where not following these incentives is much worse than others and where I currently expect the vast majority of contributors to be net-negative participants because of those incentives (and as such establishing standards of “deal with it or leave it” is a potentially reasonable choice).
I think truth-seeking institutions are one of those domains, and that in those places, slightly bad incentives seem to have larger negative effects, and also that it is very rarely worth gaining other resources in exchange for making your truth-seeking institutions worse.
For almost any other domain of the world (with the notable exception of institutions that are directly responsible for handling highly dangerous technologies), I am much less worried about incentives and generally wouldn’t judge someone very much for conforming to most of them.
Note that this review is not of the content that was nominated; nomination justifications strongly suggest that the comment suggestion, not the linkpost, was nominated.
I think the comments are in large part about the post, though, and it matters a lot whether the post is wrong or misleading.
I also think that, while, this post wouldn’t be eligible for the 2019 Review, an important point of the overall review process is still to have a coordinated time where everyone evaluates posts that have permeated the culture. I think this review is quite valuable along those lines.
That’s fair-I wasn’t disparaging the usefulness of the comment, just pointing out that the post itself is not actually what’s being reviewed, which is important, because it means that a low-quality post that sparks high-quality discussion isn’t disqualifying.
As I read it, two of the nominations are for the post itself, and one is for the comments...
...is what I was going to say until I checked and saw that this comment is a review, not a nomination. So one is for the post, and one for the comments.
----
I agree with Raemon that even if the nomination is for the comments, evaluating the post is important. I actually started writing a section on the comments, but didn’t have that much to say, because they all seem predicated on the post stating something true about the world.
The highest-voted top-level comment, as well as Zvi’s position in this comment thread, seem to basically be considering the case where academia as a whole is net negative. I broadly agree with Zvi that it is not acceptable for an academic to go around faking data; if that were the norm in academia I expect I would think that academia was net negative and one could not justify joining it (unless you were going to buck the incentives). But… that isn’t the norm in academia. I feel like these comments are only making an important point if you actually believe the original post, which I don’t. The other comments seem to have only a little content, or to be on relatively tangential topics.
That’s a fair point-see my comment to Raemon. The way I read it, the mod consensus was that we can’t just curate the post, meaning that comments are essentially the only option. To me, this means an incorrect/low quality post isn’t disqualifying, which doesn’t decrease the utility of the review, just the frame under which it should be interpreted.
I do not like this post. I think it gets most of its rhetorical oomph from speaking in a very moralizing tone, with effectively no data, and presenting everything in the worst light possible; I also think many of its claims are flat-out false. Let’s go through each point in order.
Wow, that would be truly shocking; indeed that would be truly an indictment of academia. What’s the evidence?
… Did you expect people who were caught doing bad things to say “yup, I’m a terrible person?” Also, a single extreme case does not make the rule. I can only assume you’re going to say that all the academics accepted this excuse; that would be an indictment of academia...
… I guess not. Well, point 1 is deeply unconvincing. (I also think it is false, just from my own experience in academia.) Let’s hope the others do better.
I abhor arguments of this form; I don’t think I’ve ever seen one used well. It would also break the world if everyone ate food but didn’t farm; are you going to become a farmer now?
The actual content of the section is a bit different from the heading:
… So academics are bad because they know that the bad incentives are there, and are willing to talk about them? Do you prefer the world in which they don’t know the incentives are there or don’t talk about them but still act in accordance with them? What?
If true, that seems bad. I think it is mostly false; no evidence was given in the post. I feel like the author just sees more of academia than other fields; I don’t get the sense that doctors, lawyers, journalists etc. are valiantly fighting off the bad incentives.
Okay, well, onwards to point 3:
This seems to be an argument that most of the academics are truthful, and only a few bad actors are ruining it for everyone else. I certainly agree that if there are a few bad actors violating academic norms, those people are doing something bad / should be punished. But I thought the claim was that nearly all academics are doing bad things? If so, you can’t reasonably argue that these people are being “special”.
You can’t be willfully blind to norms and then claim that people following the norms are bad and think they are “special”.
Let’s say you want to smoke pot, and you’re in an area where there’s a law against smoking pot. However, no one enforces it and everyone smokes pot all the time. Perhaps if everyone stopped smoking pot, you could make the case that smoking pot isn’t addictive and get the law overturned. It is incorrect to blame everyone else for smoking pot, or to say that they are defecting, even though it is true that the law could be reversed if everyone stopped smoking pot.
If the claim is actually that there are only a few bad actors, then I assume the author must be talking about the especially bad cases like fabricating data. I expect that when caught such cases really are shamed, and you can’t get out of it by saying “Oh, The Incentives”.
This is a clear isolated demand for rigor. (Ironically, this post itself has no data, making it even clearer than usual.)
I can only assume that the author decides when to shower, what to eat, what transit to use, etc by simulating out all the possible consequences of each possible decision several years into the future, aggregating the consequences according to their personal utility function, and then choosing the one that is best. I must applaud the author for their immense brainpower; unfortunately us mere mortals can’t do this ourselves and need to rely on heuristics.
Sarcasm aside; I think it is perfectly possible to make good decisions without having all the available data. We do this all the time in lots of different scenarios. Why expect that people can’t do it here as well?
Conditions of extreme competition over scarce resources are exactly where you expect the most conformance to the incentives.
I’ll note that these are once again claims with no data. I think it’s false for the people who follow The Incentives who do become professors; I would not be surprised if they went from 20% to 80%, which seems like a substantial change. I agree that on average it’s probably not very high, because a lot of people are not going to become professors regardless (i.e. maybe they go from 2% to 3%), but it’s hard to tell from the inside which of these situations you’re in.
I totally agree with this point.
I am pretty uncertain about this point. Most people do not become famous and have their work scrutinized in detail. I agree that long-term considerations do point in the direction of bucking the incentives; I just don’t know how strongly.
(The “It” here is “complaining about incentives”, not “following the incentives”.)
I don’t really see the point here. Surely fixing the problem involves first creating common knowledge about the problem? But in any case, this seems to be orthogonal to the main point (that following the incentives is bad), so I’m going to skip it.
This same complaint can be leveled at most of society. You don’t get to choose a job description and then have people faithfully execute that job description; what you get depends on the incentives you give.
Would the world be better if you could just give a job description and have people faithfully do that? Probably. Does the world work that way? No. Just yelling “I want you to do X, therefore you should do X” seems counterproductive and willfully blind about how the world works.
----
I do not know what people get out of this post; the arguments given do not remotely support the conclusions, with a couple of exceptions. If I had to guess some effects that would make people upvote / nominate the post:
Powerfully suggested that people should bear responsibility for bucking the incentives
Written as a rant, and so was fun to read
Anti-academia, which agrees with the reader’s beliefs
Pro-principles, which agrees with the reader’s beliefs / morals
(I am not claiming that people consciously thought about these reasons.)
I’ll ignore 2, 3, and 4; I think it’s obvious why those are not reasons we want to promote on LW.
I think the first one could plausibly be a reason that we would want to promote this on LW. Unfortunately, I think it is wrong: I do not think that people should usually feel upon themselves the burden of bucking bad incentives. There are many, many bad incentives in the world; you cannot buck them all simultaneously and make the world a better place. Rather, you need to conform with the bad incentives, even though it makes your blood boil, and choose a select few areas in which you are going to change the world, and focus on those.
(Tbc, this is not a recommendation to p-hack; that is not the norm in academia. The norm in academia was to do other questionable things that in effect constitute p-hacking, but it was never to explicitly p-hack. And that norm might be changing at this point, I don’t know. Personally, while there are questionable academic practices that I follow when writing papers, such as including all the pros of a proposal in the introduction but ignoring the cons, I try to make my experiments as un-p-hacked as I can, though I have the advantage of not seeking a job in academia, as well as a good network, so incentives affect me less than they do most academics.)
I agree with most of this review, and also didn’t really like this post when it came out.
Just for the record, and since I think this is actually an important point, my perspective is that indeed people cannot take on themselves the burden of bucking bad all bad incentives, but that there are a few domains of society where not following these incentives is much worse than others and where I currently expect the vast majority of contributors to be net-negative participants because of those incentives (and as such establishing standards of “deal with it or leave it” is a potentially reasonable choice).
I think truth-seeking institutions are one of those domains, and that in those places, slightly bad incentives seem to have larger negative effects, and also that it is very rarely worth gaining other resources in exchange for making your truth-seeking institutions worse.
For almost any other domain of the world (with the notable exception of institutions that are directly responsible for handling highly dangerous technologies), I am much less worried about incentives and generally wouldn’t judge someone very much for conforming to most of them.
Note that this review is not of the content that was nominated; nomination justifications strongly suggest that the comment suggestion, not the linkpost, was nominated.
I think the comments are in large part about the post, though, and it matters a lot whether the post is wrong or misleading.
I also think that, while, this post wouldn’t be eligible for the 2019 Review, an important point of the overall review process is still to have a coordinated time where everyone evaluates posts that have permeated the culture. I think this review is quite valuable along those lines.
That’s fair-I wasn’t disparaging the usefulness of the comment, just pointing out that the post itself is not actually what’s being reviewed, which is important, because it means that a low-quality post that sparks high-quality discussion isn’t disqualifying.
As I read it, two of the nominations are for the post itself, and one is for the comments...
...is what I was going to say until I checked and saw that this comment is a review, not a nomination. So one is for the post, and one for the comments.
----
I agree with Raemon that even if the nomination is for the comments, evaluating the post is important. I actually started writing a section on the comments, but didn’t have that much to say, because they all seem predicated on the post stating something true about the world.
The highest-voted top-level comment, as well as Zvi’s position in this comment thread, seem to basically be considering the case where academia as a whole is net negative. I broadly agree with Zvi that it is not acceptable for an academic to go around faking data; if that were the norm in academia I expect I would think that academia was net negative and one could not justify joining it (unless you were going to buck the incentives). But… that isn’t the norm in academia. I feel like these comments are only making an important point if you actually believe the original post, which I don’t. The other comments seem to have only a little content, or to be on relatively tangential topics.
That’s a fair point-see my comment to Raemon. The way I read it, the mod consensus was that we can’t just curate the post, meaning that comments are essentially the only option. To me, this means an incorrect/low quality post isn’t disqualifying, which doesn’t decrease the utility of the review, just the frame under which it should be interpreted.