Actually, now you’ve nudged my mind in the right direction! Let’s consider an example even more remote in time, and even more outlandish by modern standards than slavery or absolute monarchy: medieval trials by ordeal.
The modern consensus belief is that this was just awful superstition in action, and our modern courts of law are obviously a vast improvement. That’s certainly what I had thought until I read a recent paper titled “Ordeals” by one Peter T. Leeson, who argues that these ordeals were in fact, in the given circumstances, a highly accurate way of separating the guilty from the innocent given the prevailing beliefs and customs of the time. I highly recommend reading the paper, or at least the introduction, as an entertaining de-biasing experience. [Update: there is also an informal exposition of the idea by the author, for those who are interested but don’t feel like going through the math of the original paper.]
I can’t say with absolute confidence if Leeson’s arguments are correct or not, but they sound highly plausible to me, and certainly can’t be dismissed outright. However, if he is correct, then two interesting propositions are within the realm of the possible: (1) in the given circumstances in which medieval Europeans lived, trials by ordeal were perhaps more effective in making correct verdicts in practice than if they had used something similar to our modern courts of law instead, and (2) the verdict accuracy rate by trials by ordeal could well have been greater than that achieved by our modern courts of law, which can’t be realistically considered to be anywhere near perfect. As Leeson says:
Ordeals are inferior to modern trial methods because modern defendants don’t believe in iudicium Dei, not because trial by jury is inherently superior. If modern citizens did have the superstitious belief required for ordeals to work, it might make sense to bring back the cauldrons of boiling water.
Now, let’s look at the issue and separate the relevant normative and factual beliefs involved. The prevailing normative belief today is that the only acceptable way to determine criminal guilt is to use evidence-based trials in front of courts, whose job is to judge the evidence as free of bias as possible. It’s a purely normative view, which states that anything else would simply be unjust and illegitimate, period. However, underlying this normative belief, and serving as its important consequentialist basis, there is also the factual belief that despite all the unavoidable biases, evidence-based trials necessarily produce more accurate verdicts than other methods, especially ancient methods such as the trial by ordeal that involved superstitions.
Yet, if Leeson is correct—and we should seriously consider that possibility—this factual belief, despite having been universally accepted in our civilization for centuries, is false. What follows is that there may actually be a non-obvious way to produce more accurate verdicts even in our day and age, based on different institutions, but nobody is taking the possibility seriously because of the universal (and biased) factual belief about the practical optimality of the modern court system. It also follows that a thousand years ago, Europeans could easily have caused more wrongful punishment by abolishing trials by ordeal and replacing them with evidence-based trials, even though such a change would be judged by the modern consensus view as a vast improvement, both morally and in practical accuracy.
Another interesting remark is that, from what I’ve seen on legal blogs, Leeson’s paper was met with polite and interested skepticism, not derision and hostility. However, it seems to me that this is because the topic is so extremely remote that it has no bearing whatsoever on any modern ideological controversies; I have no doubt that a similar positive reexamination of some negatively judged past belief or institution that still has significant ideological weight would provoke far more hostility. That seems to be another piece of evidence suggesting that severe biases might be found lurking under the modern consensus on a great many issues, operating via the mechanism I’m proposing.
I skimmed Leeson’s paper, and it looks like it has no quantitative evidence for the true accuracy of trial by ordeal. It has quantitative evidence for one of the other predictions he makes with his theory (the prediction that most people who go through ordeals are exonerated by them, which prediction is supported by the corresponding numbers, though not resoundingly), but Leeson doesn’t know what the actual hit rate of trial by orderal is.
This doesn’t mean Leeson’s a bad guy or anything—I bet no one can get a good estimate of trial by ordeal’s accuracy, since we’re here too late to get the necessary data. But it does mean he’s exaggerating (probably unconsciously) the implications of his paper—ultimately, his model will always fit the data as long as sufficiently many people believed trial by ordeal was accurate, independent of true accuracy. So the fact that his model pretty much fits the data is not strong evidence of true accuracy. Given that Leeson’s model fits the data he does have, and the fact that fact-finding methods were relatively poor in medieval times, I think your ‘interesting proposition’ #1 is quite likely, but we don’t gain much new information about #2.
The modern consensus belief is that this was just awful superstition in action, and our modern courts of law are obviously a vast improvement. That’s certainly what I had thought until I read a recent paper titled “Ordeals” by one Peter T. Leeson, who argues that these ordeals were in fact, in the given circumstances, a highly accurate way of separating the guilty from the innocent given the prevailing beliefs and customs of the time.
That’s interesting. I think you’re right that no one reacts too negatively to this news because they don’t see any real danger that it would be implemented.
But suppose there were a real movement to bring back trial by ordeal. According to the paper’s abstract, trial by ordeal was so effective because the defendants held certain superstitious belief. Therefore, if we wanted it to work again, we would need to change peoples’ worldview so that they again held such beliefs.
But there’s reason to expect that these beliefs would cause a great deal of harm — enough to outweigh the benefit from more accurate trials. For example, maybe airlines wouldn’t perform such careful maintenance on an airplane if a bunch of nuns would be riding it, since God wouldn’t allow a plane full of nuns to go down.
Well, look at me — I launched right into rationalizing a counter-argument. As with so many of the biases that Robin Hanson talks about, one has to ask, does my dismissal of the suggestion show that we’re right to reject it, or am I just providing another example of the bias in action?
Actually, now you’ve nudged my mind in the right direction! Let’s consider an example even more remote in time, and even more outlandish by modern standards than slavery or absolute monarchy: medieval trials by ordeal.
The modern consensus belief is that this was just awful superstition in action, and our modern courts of law are obviously a vast improvement. That’s certainly what I had thought until I read a recent paper titled “Ordeals” by one Peter T. Leeson, who argues that these ordeals were in fact, in the given circumstances, a highly accurate way of separating the guilty from the innocent given the prevailing beliefs and customs of the time. I highly recommend reading the paper, or at least the introduction, as an entertaining de-biasing experience. [Update: there is also an informal exposition of the idea by the author, for those who are interested but don’t feel like going through the math of the original paper.]
I can’t say with absolute confidence if Leeson’s arguments are correct or not, but they sound highly plausible to me, and certainly can’t be dismissed outright. However, if he is correct, then two interesting propositions are within the realm of the possible: (1) in the given circumstances in which medieval Europeans lived, trials by ordeal were perhaps more effective in making correct verdicts in practice than if they had used something similar to our modern courts of law instead, and (2) the verdict accuracy rate by trials by ordeal could well have been greater than that achieved by our modern courts of law, which can’t be realistically considered to be anywhere near perfect. As Leeson says:
Now, let’s look at the issue and separate the relevant normative and factual beliefs involved. The prevailing normative belief today is that the only acceptable way to determine criminal guilt is to use evidence-based trials in front of courts, whose job is to judge the evidence as free of bias as possible. It’s a purely normative view, which states that anything else would simply be unjust and illegitimate, period. However, underlying this normative belief, and serving as its important consequentialist basis, there is also the factual belief that despite all the unavoidable biases, evidence-based trials necessarily produce more accurate verdicts than other methods, especially ancient methods such as the trial by ordeal that involved superstitions.
Yet, if Leeson is correct—and we should seriously consider that possibility—this factual belief, despite having been universally accepted in our civilization for centuries, is false. What follows is that there may actually be a non-obvious way to produce more accurate verdicts even in our day and age, based on different institutions, but nobody is taking the possibility seriously because of the universal (and biased) factual belief about the practical optimality of the modern court system. It also follows that a thousand years ago, Europeans could easily have caused more wrongful punishment by abolishing trials by ordeal and replacing them with evidence-based trials, even though such a change would be judged by the modern consensus view as a vast improvement, both morally and in practical accuracy.
Another interesting remark is that, from what I’ve seen on legal blogs, Leeson’s paper was met with polite and interested skepticism, not derision and hostility. However, it seems to me that this is because the topic is so extremely remote that it has no bearing whatsoever on any modern ideological controversies; I have no doubt that a similar positive reexamination of some negatively judged past belief or institution that still has significant ideological weight would provoke far more hostility. That seems to be another piece of evidence suggesting that severe biases might be found lurking under the modern consensus on a great many issues, operating via the mechanism I’m proposing.
I skimmed Leeson’s paper, and it looks like it has no quantitative evidence for the true accuracy of trial by ordeal. It has quantitative evidence for one of the other predictions he makes with his theory (the prediction that most people who go through ordeals are exonerated by them, which prediction is supported by the corresponding numbers, though not resoundingly), but Leeson doesn’t know what the actual hit rate of trial by orderal is.
This doesn’t mean Leeson’s a bad guy or anything—I bet no one can get a good estimate of trial by ordeal’s accuracy, since we’re here too late to get the necessary data. But it does mean he’s exaggerating (probably unconsciously) the implications of his paper—ultimately, his model will always fit the data as long as sufficiently many people believed trial by ordeal was accurate, independent of true accuracy. So the fact that his model pretty much fits the data is not strong evidence of true accuracy. Given that Leeson’s model fits the data he does have, and the fact that fact-finding methods were relatively poor in medieval times, I think your ‘interesting proposition’ #1 is quite likely, but we don’t gain much new information about #2.
(Edit—it might also be possible to incorporate ordeal-like tests into modern police work! ‘Machine is never wrong, son.’)
That’s interesting. I think you’re right that no one reacts too negatively to this news because they don’t see any real danger that it would be implemented.
But suppose there were a real movement to bring back trial by ordeal. According to the paper’s abstract, trial by ordeal was so effective because the defendants held certain superstitious belief. Therefore, if we wanted it to work again, we would need to change peoples’ worldview so that they again held such beliefs.
But there’s reason to expect that these beliefs would cause a great deal of harm — enough to outweigh the benefit from more accurate trials. For example, maybe airlines wouldn’t perform such careful maintenance on an airplane if a bunch of nuns would be riding it, since God wouldn’t allow a plane full of nuns to go down.
Well, look at me — I launched right into rationalizing a counter-argument. As with so many of the biases that Robin Hanson talks about, one has to ask, does my dismissal of the suggestion show that we’re right to reject it, or am I just providing another example of the bias in action?
It’s the old noble lie in a different package.