As discussed in chapter 1, we have used a score of 27 on the APSD as our cut-off point for a classification of psychopathic tendencies in many of our studies.
I’ve noticed this elsewhere (looking into ADHD), Psychiatrists seem interested in developing a criteria which seems naturally continuous, and then using a cutoff without arguing for why that’s a good idea. I can easily imagine that some conditions are discrete, but many of them must be pretty continuous. It seems like they would lose a lot of statistical power with a cutoff approach.
Is this purely a historical accident? Is it because discrete judgments seem more authoritative? Is there an actual good reason that I can’t see? What’s going on here? This sort of thing makes me suspicious of the quality of psychiatry research.
I think this is part of a general phenomenon that might simply be more visible in certain fields than others, and which seems to arise from the fact that natural language cannot adequately deal with continuous phenomena. Richard Dawkins discusses this briefly in the popular piece The tyranny of the discontinuous mind.
I’d imagine a strong chunk of it is that it’s a lot easier to work with a cutoff, and it also makes it much easier to transmit to courts etc. What is a court supposed to do when the psychiatrist reports back about the defendant that ‘he scored a 27 on the APSD’ or ‘he scored a 15 on the Hare’? Or even a more sophisticated measure like ‘he scored a 27 which is in the 94th percentile’?
A lot easier to work with? Is running a linear regression that hard? Or do you have in mind something beyond that?
The courts thing makes more sense, though I’d be surprised if courts couldn’t understand percentiles fairly well (“more psychopathic than 94 percent of the population”).
Another theory is that, the first success stories of identifying things in psychiatry and treating them came from things which are discrete so there’s a tradition of treating things that way, and the contrary evidence doesn’t hit them in the face.
A lot easier to work with? Is running a linear regression that hard?
Yep. Remember, every step of statistical sophistication is a filter. Some people who understand means will not understand correlations; some people who understand correlations will not understand standard deviations; some people who understand deviations will not understand t-tests; and everyone won’t understand p-values judging by how often I see the ‘5% odds this was due to chance’ fallacy even here on LessWrong. Linear regression is another step on top of that, etc.
The courts thing makes more sense, though I’d be surprised if courts couldn’t understand percentiles fairly well (“more psychopathic than 94 percent of the population”).
What does that even mean for the court, though? They aren’t judging the overall population; heck, criminal defendants aren’t even a sample from the overall population but are heavily biased towards particular places, races, IQs, and annual incomes etc etc.
Another theory is that, the first success stories of identifying things in psychiatry and treating them came from things which are discrete so there’s a tradition of treating things that way, and the contrary evidence doesn’t hit them in the face.
Psychiatrists are the first to talk about how so many things are continuous, and these days disorders as continuous phenomenon is dogma, so I doubt it’s a psychiatry thing. It’s not like medicine in general doesn’t use tons of discrete cutoffs. (If your vitamin D blood level is below 20 or whatever, you Have A Deficiency; if it’s 21, you Do Not Have A Deficiency.) It’s just much much easier.
Yes, but many people use both cut-offs and regressions. It is quite common to see a regression of the effect of a continuous input on the probability of crossing a cut-off. This seems indefensible to me.
I think it’s mostly a historical accident but also has to do with practical issues. Natural language and continous phenomena don’t go well hand in hand, and the criteria have to be applicable in the clinical setting and the guidelines. DSM is the basis for most psychiatric research, and almost all of the disorders currently described in it are categorical with clear cutoff points instead of continous. Psychiatrists acknowledge this problem, and DSM-5 is going for a more dimensional approach as opposed to categorical.
I can easily imagine that some conditions are discrete, but many of them must be pretty continuous. It seems like they would lose a lot of statistical power with a cutoff approach.
Having a cutoff doesn’t stop you from running your statistical test on the scores.
You need a cutoff to decide whether to accept an individual into your study. If you want to study whether psychopathy gets reduced by a treatment it makes sense restrict your study to indiduals who’s score is over a certain value.
Having all studies use the same cutoff value helps you to compare different treatments.
That’s true, but at least in the ADHD research I saw this is not the way cutoffs were used, they were used to categorize people as “have ADHD” or “not have ADHD” and the “have ADHD” were compared to “not have ADHD”.
ADHD seems to get diagnosed based on the DSM-IV. It’s not binary. There are three types “ADHD, Combined Type”, “ADHD, Predominantly Inattentive Type” and “ADHD, Predominantly Hyperactive-Impulsive Type”. A DSM-IV diagnosis however doesn’t give you are score.
There are huge issues with the DSM IV. In it’s philosophy the DSM describes symptoms. As a result it’s authors don’t see the necessarity to back up their diagnosis with scientific evidence that get’s cited in the DSM.
Psychopathy on the other hand gets diagnosed based on a specific PCL-R test that Robert D. Hare developed. The test is result of psychometric work. It’s designed to predict recidivism and violence.
The DSM-IV doesn’t recognize psychopathy but instead uses the category of antisocial personality disorder.
In the case of depression you also have on the one hand the DSM-IV criteria and on the other hand the Hamilton Rating Scale for Depression.
Psychiatry studies that use tests that are optimized to predict something like the PCL-R should probably be trusted more than the DSM-IV.
The present debate about the new DSM-V can also give you a good illustration of it’s nature.
Virtually any kind of statistical study is going to require you to classify into different groups using some thresholding definition. To fail to do so is to make any reportable results muddier and muddier.
I don’t think so, just use the ASPD score as a predictor in a linear regression. Seems like an obvious response, so maybe I didn’t understand what you said.
I’ve noticed this elsewhere (looking into ADHD), Psychiatrists seem interested in developing a criteria which seems naturally continuous, and then using a cutoff without arguing for why that’s a good idea. I can easily imagine that some conditions are discrete, but many of them must be pretty continuous. It seems like they would lose a lot of statistical power with a cutoff approach.
Is this purely a historical accident? Is it because discrete judgments seem more authoritative? Is there an actual good reason that I can’t see? What’s going on here? This sort of thing makes me suspicious of the quality of psychiatry research.
I think this is part of a general phenomenon that might simply be more visible in certain fields than others, and which seems to arise from the fact that natural language cannot adequately deal with continuous phenomena. Richard Dawkins discusses this briefly in the popular piece The tyranny of the discontinuous mind.
I’d imagine a strong chunk of it is that it’s a lot easier to work with a cutoff, and it also makes it much easier to transmit to courts etc. What is a court supposed to do when the psychiatrist reports back about the defendant that ‘he scored a 27 on the APSD’ or ‘he scored a 15 on the Hare’? Or even a more sophisticated measure like ‘he scored a 27 which is in the 94th percentile’?
A lot easier to work with? Is running a linear regression that hard? Or do you have in mind something beyond that?
The courts thing makes more sense, though I’d be surprised if courts couldn’t understand percentiles fairly well (“more psychopathic than 94 percent of the population”).
Another theory is that, the first success stories of identifying things in psychiatry and treating them came from things which are discrete so there’s a tradition of treating things that way, and the contrary evidence doesn’t hit them in the face.
Yep. Remember, every step of statistical sophistication is a filter. Some people who understand means will not understand correlations; some people who understand correlations will not understand standard deviations; some people who understand deviations will not understand t-tests; and everyone won’t understand p-values judging by how often I see the ‘5% odds this was due to chance’ fallacy even here on LessWrong. Linear regression is another step on top of that, etc.
What does that even mean for the court, though? They aren’t judging the overall population; heck, criminal defendants aren’t even a sample from the overall population but are heavily biased towards particular places, races, IQs, and annual incomes etc etc.
Psychiatrists are the first to talk about how so many things are continuous, and these days disorders as continuous phenomenon is dogma, so I doubt it’s a psychiatry thing. It’s not like medicine in general doesn’t use tons of discrete cutoffs. (If your vitamin D blood level is below 20 or whatever, you Have A Deficiency; if it’s 21, you Do Not Have A Deficiency.) It’s just much much easier.
Yes, but many people use both cut-offs and regressions. It is quite common to see a regression of the effect of a continuous input on the probability of crossing a cut-off. This seems indefensible to me.
I think it’s mostly a historical accident but also has to do with practical issues. Natural language and continous phenomena don’t go well hand in hand, and the criteria have to be applicable in the clinical setting and the guidelines. DSM is the basis for most psychiatric research, and almost all of the disorders currently described in it are categorical with clear cutoff points instead of continous. Psychiatrists acknowledge this problem, and DSM-5 is going for a more dimensional approach as opposed to categorical.
The official source is here.
Very interesting! Sounds like a positive development. Interesting that it’s the DSM at the cutting edge, I assumed it was a book of standards.
Having a cutoff doesn’t stop you from running your statistical test on the scores.
You need a cutoff to decide whether to accept an individual into your study. If you want to study whether psychopathy gets reduced by a treatment it makes sense restrict your study to indiduals who’s score is over a certain value.
Having all studies use the same cutoff value helps you to compare different treatments.
That’s true, but at least in the ADHD research I saw this is not the way cutoffs were used, they were used to categorize people as “have ADHD” or “not have ADHD” and the “have ADHD” were compared to “not have ADHD”.
ADHD seems to get diagnosed based on the DSM-IV. It’s not binary. There are three types “ADHD, Combined Type”, “ADHD, Predominantly Inattentive Type” and “ADHD, Predominantly Hyperactive-Impulsive Type”. A DSM-IV diagnosis however doesn’t give you are score.
There are huge issues with the DSM IV. In it’s philosophy the DSM describes symptoms. As a result it’s authors don’t see the necessarity to back up their diagnosis with scientific evidence that get’s cited in the DSM.
Psychopathy on the other hand gets diagnosed based on a specific PCL-R test that Robert D. Hare developed. The test is result of psychometric work. It’s designed to predict recidivism and violence. The DSM-IV doesn’t recognize psychopathy but instead uses the category of antisocial personality disorder.
In the case of depression you also have on the one hand the DSM-IV criteria and on the other hand the Hamilton Rating Scale for Depression.
Psychiatry studies that use tests that are optimized to predict something like the PCL-R should probably be trusted more than the DSM-IV. The present debate about the new DSM-V can also give you a good illustration of it’s nature.
Sure, it’s not binary but it is discrete.
The process used to design the PCL-R test does seem better.
Virtually any kind of statistical study is going to require you to classify into different groups using some thresholding definition. To fail to do so is to make any reportable results muddier and muddier.
I don’t think so, just use the ASPD score as a predictor in a linear regression. Seems like an obvious response, so maybe I didn’t understand what you said.