First, great post. Second, general injunctions against giving very low probabilities to things seems to be taken by many casual readers as endorsements of the (bad) behavior “privilege the hypothesis”—e.g. moving the probability from very small to moderately small that God exists. That’s not right, but I don’t have excellent arguments for why it’s not right. I’d love it if you wrote an article on choosing good priors.
Cosma Shalizi has done some technical work that seems (to my incompetent eye) to be relevant:
That is, he takes Bayesian updating, which requires modeling the world, and answers the question ‘when would it be okay to use Bayesian updating, even though we know the model is definitely wrong—e.g. too simple?’. (Of course, making your model “not obviously wrong” by adding complexity isn’t a solution.)
I am still confused about how small the probability I should use in the God question is. I understand the argument about privileging the hypothesis and about intelligent beings being very complex and fantastically unlikely.
But I also feel that if I tried to use an argument at least that subtle, when applied to something I am at least as confused about as how ontologically complex a first cause should be, to disprove things at least as widely believed as religion, a million times, I would be wrong at least once.
But I also feel that if I tried to use an argument at least that subtle, when applied to something I am at least as confused about as how ontologically complex a first cause should be, to disprove things at least as widely believed as religion, a million times, I would be wrong at least once.
See Advancing Certainty. The fact that this statement sounds comfortably modest does not exempt it from the scrutiny of the Fundamental Question of Rationality (why do you believe what you believe?). I respectfully submit that if the answer is “because I have been wrong before, where I was equally confident, in previous eras of my life when I wasn’t using arguments this powerful (they just felt powerful to me at the time)”, that doesn’t suffice—for the same reason that the Lord Kelvin argument doesn’t suffice to show that arguments from physics can’t be trusted (unless you don’t think physics has learned anything since Kelvin).
I’ve got to admit I disagree with a lot of Advancing Certainty. The proper reference class for a modern physicist who is well acquainted with the mistakes of Lord Kelvin and won’t do them again is “past scientists who were well acquainted with the mistakes of their predecessors and plan not to do them again”, which I imagine has less than a hundred percent success rate and which might have included Kelvin.
It would be a useful exercise to see whether the most rational physicists of 1950 have more successful predictions as of 2000 than the most rational physicists of 1850 did as of 1900. It wouldn’t surprise me if this were true, and so, then the physicists of 2000 could justly put themselves in a new reference class and guess they will be even more successful as of 2050 than the 1950ers were in 2000. But if the success rate after fifty years remains constant, I wouldn’t want to say “Yeah, well , we’ve probably solved all those problems now, so we’ll do better”.
I’ve got to admit I disagree with a lot of Advancing Certainty
Do you actually disagree with any particular claim in Advancing Certainty, or does it just seem “off” to you in its emphasis? Because when I read your post, I felt myself “disagreeing” (and panicking at the rapid upvoting), but reflection revealed that I was really having something more like an ADBOC reaction. It felt to me that the intent of your post was to say “Boo confident probabilities!”, while I tend to be on the side of “Yay confident probabilities!”—not because I’m in favor of overconfidence, but rather because I think many worries about overconfidence here tend to be ill-founded (I suppose I’m something of a third-leveler on this issue.)
And indeed, when you see people complaining about overconfidence on LW, it’s not usually because someone thinks that some political candidate has a 0.999999999 chance of winning an election; almost nobody here would think that a reasonable estimate. Instead, what you get is people saying that 0.0000000001 is too low a probability that God exists—on the basis of nothing else than general worry about human overconfidence.
I think my anti-anti-overconfidence vigilance started when I realized I had been socially intimidated into backing off from my estimate of 0.001 in the Amanda Knox case, when in fact that was and remains an entirely reasonable number given my detailed knowledge of the case. The mistake I made was to present this number as if it were something that participants in my survey should have arrived at from a few minutes of reading. Those states—the ones that survey participants were in, with reference classes like “highly controversial conviction with very plausible defense arguments”—are what probabilities like 0.1 or 0.3 are for. My state, on the other hand, was more like “highly confident inside-view conclusion bolstered by LW survey results decisively on the same side of 50%”.
But this isn’t what the overconfidence-hawks argued. What they said, in essence, was that 0.001 was just somehow “inherently” too confident. Only “irrational” people wear the attire of “P(X) = 0.001”; We Here, by contrast, are Aware Of Biases Like Overconfidence, and only give Measured, Calm, Reasonable Probabilities.
That is the mistake I want to fight, now that I have the courage to do so. Though I can’t find much to literally disagree about in your post, it unfortunately feels to me like ammunition for the enemy.
I definitely did have the “ammunition for the enemy” feeling about your post, and the “belief attire” point is a good one, but I think the broad emotional disagreement does express itself in a few specific claims:
Even if you were to control for getting tired and hungry and so on, even if you were to load your intelligence into a computer and have it do the hard work, I still don’t think you could judge a thousand such trials and be wrong only once. I admit this may not be as real a disagreement as I’m thinking, because it may be a confusion on what sort of reference class we should use to pick trials for you.
I think we might disagree on the Lord Kelvin claim. I think I would predict more of today’s physical theories are wrong than you would.
I think my probability that God exists would be several orders of magnitude higher than yours, even though I think you probably know about the same number of good arguments on the issue as I do.
Maybe our disagreement can be resolved empirically—if we were to do enough problems where we gave confidence levels on questions like “The area of Canada is greater than the area of the Mediterranean Sea” and use log odds scoring we might find one of us doing significantly better than the other—although we would have to do quite a few to close off my possible argument that we just didn’t hit that one “black swan” question on which you’d say you’re one in a million confident and then get it wrong. Would you agree that this would get to the heart of our disagreement, or do you think it revolves solely around more confusing philosophical questions?
(I took a test like that yesterday to test something and I came out overconfident, missing 2⁄10 questions at the 96% probability level. I don’t know how that translates to more real-world questions and higher confidence levels, but it sure makes me reluctant to say I’m chronically underconfident)
I still don’t think you could judge a thousand such trials and be wrong only once.
When I first saw this, I agreed with it. But now I don’t, partly because of the story (which I don’t have a link to, but it was linked to from LW somewhere) about someone who would bet they knew whether or not a number was a prime. This continued until they made a mistake (doing it mentally), and then they lost.
If they had a calculator, could they go up to the 1000th odd number and be wrong at most once? I’m pretty sure they could, actually. And so the question isn’t “can you judge 1000 trials and only get one wrong?” but “can you judge 1000 obvious trials and only get one wrong?”, or, more appropriately, “can you judge 1000 trials as either ‘obvious’ and ‘contested’ and only be wrong at most once?”. Because originally I was imagining being a normal trial judge- but a normal trial judge has to deal with difficult cases. Ones like the Amanda Knox case (are/should be) rare. I’m pretty confident that once you put in a reasonable amount of effort (however much komponisto did for this case), you can tell whether or not the case is one you can be confident about or one you can’t, assuming you’re carefully thinking about what would make them not open-and-shut cases.
First, great post. Second, general injunctions against giving very low probabilities to things seems to be taken by many casual readers as endorsements of the (bad) behavior “privilege the hypothesis”—e.g. moving the probability from very small to moderately small that God exists. That’s not right, but I don’t have excellent arguments for why it’s not right. I’d love it if you wrote an article on choosing good priors.
Cosma Shalizi has done some technical work that seems (to my incompetent eye) to be relevant:
http://projecteuclid.org/DPubS?verb=Display&version=1.0&service=UI&handle=euclid.ejs/1256822130&page=record
That is, he takes Bayesian updating, which requires modeling the world, and answers the question ‘when would it be okay to use Bayesian updating, even though we know the model is definitely wrong—e.g. too simple?’. (Of course, making your model “not obviously wrong” by adding complexity isn’t a solution.)
I am still confused about how small the probability I should use in the God question is. I understand the argument about privileging the hypothesis and about intelligent beings being very complex and fantastically unlikely.
But I also feel that if I tried to use an argument at least that subtle, when applied to something I am at least as confused about as how ontologically complex a first cause should be, to disprove things at least as widely believed as religion, a million times, I would be wrong at least once.
See Advancing Certainty. The fact that this statement sounds comfortably modest does not exempt it from the scrutiny of the Fundamental Question of Rationality (why do you believe what you believe?). I respectfully submit that if the answer is “because I have been wrong before, where I was equally confident, in previous eras of my life when I wasn’t using arguments this powerful (they just felt powerful to me at the time)”, that doesn’t suffice—for the same reason that the Lord Kelvin argument doesn’t suffice to show that arguments from physics can’t be trusted (unless you don’t think physics has learned anything since Kelvin).
I’ve got to admit I disagree with a lot of Advancing Certainty. The proper reference class for a modern physicist who is well acquainted with the mistakes of Lord Kelvin and won’t do them again is “past scientists who were well acquainted with the mistakes of their predecessors and plan not to do them again”, which I imagine has less than a hundred percent success rate and which might have included Kelvin.
It would be a useful exercise to see whether the most rational physicists of 1950 have more successful predictions as of 2000 than the most rational physicists of 1850 did as of 1900. It wouldn’t surprise me if this were true, and so, then the physicists of 2000 could justly put themselves in a new reference class and guess they will be even more successful as of 2050 than the 1950ers were in 2000. But if the success rate after fifty years remains constant, I wouldn’t want to say “Yeah, well , we’ve probably solved all those problems now, so we’ll do better”.
Do you actually disagree with any particular claim in Advancing Certainty, or does it just seem “off” to you in its emphasis? Because when I read your post, I felt myself “disagreeing” (and panicking at the rapid upvoting), but reflection revealed that I was really having something more like an ADBOC reaction. It felt to me that the intent of your post was to say “Boo confident probabilities!”, while I tend to be on the side of “Yay confident probabilities!”—not because I’m in favor of overconfidence, but rather because I think many worries about overconfidence here tend to be ill-founded (I suppose I’m something of a third-leveler on this issue.)
And indeed, when you see people complaining about overconfidence on LW, it’s not usually because someone thinks that some political candidate has a 0.999999999 chance of winning an election; almost nobody here would think that a reasonable estimate. Instead, what you get is people saying that 0.0000000001 is too low a probability that God exists—on the basis of nothing else than general worry about human overconfidence.
I think my anti-anti-overconfidence vigilance started when I realized I had been socially intimidated into backing off from my estimate of 0.001 in the Amanda Knox case, when in fact that was and remains an entirely reasonable number given my detailed knowledge of the case. The mistake I made was to present this number as if it were something that participants in my survey should have arrived at from a few minutes of reading. Those states—the ones that survey participants were in, with reference classes like “highly controversial conviction with very plausible defense arguments”—are what probabilities like 0.1 or 0.3 are for. My state, on the other hand, was more like “highly confident inside-view conclusion bolstered by LW survey results decisively on the same side of 50%”.
But this isn’t what the overconfidence-hawks argued. What they said, in essence, was that 0.001 was just somehow “inherently” too confident. Only “irrational” people wear the attire of “P(X) = 0.001”; We Here, by contrast, are Aware Of Biases Like Overconfidence, and only give Measured, Calm, Reasonable Probabilities.
That is the mistake I want to fight, now that I have the courage to do so. Though I can’t find much to literally disagree about in your post, it unfortunately feels to me like ammunition for the enemy.
I definitely did have the “ammunition for the enemy” feeling about your post, and the “belief attire” point is a good one, but I think the broad emotional disagreement does express itself in a few specific claims:
Even if you were to control for getting tired and hungry and so on, even if you were to load your intelligence into a computer and have it do the hard work, I still don’t think you could judge a thousand such trials and be wrong only once. I admit this may not be as real a disagreement as I’m thinking, because it may be a confusion on what sort of reference class we should use to pick trials for you.
I think we might disagree on the Lord Kelvin claim. I think I would predict more of today’s physical theories are wrong than you would.
I think my probability that God exists would be several orders of magnitude higher than yours, even though I think you probably know about the same number of good arguments on the issue as I do.
Maybe our disagreement can be resolved empirically—if we were to do enough problems where we gave confidence levels on questions like “The area of Canada is greater than the area of the Mediterranean Sea” and use log odds scoring we might find one of us doing significantly better than the other—although we would have to do quite a few to close off my possible argument that we just didn’t hit that one “black swan” question on which you’d say you’re one in a million confident and then get it wrong. Would you agree that this would get to the heart of our disagreement, or do you think it revolves solely around more confusing philosophical questions?
(I took a test like that yesterday to test something and I came out overconfident, missing 2⁄10 questions at the 96% probability level. I don’t know how that translates to more real-world questions and higher confidence levels, but it sure makes me reluctant to say I’m chronically underconfident)
When I first saw this, I agreed with it. But now I don’t, partly because of the story (which I don’t have a link to, but it was linked to from LW somewhere) about someone who would bet they knew whether or not a number was a prime. This continued until they made a mistake (doing it mentally), and then they lost.
If they had a calculator, could they go up to the 1000th odd number and be wrong at most once? I’m pretty sure they could, actually. And so the question isn’t “can you judge 1000 trials and only get one wrong?” but “can you judge 1000 obvious trials and only get one wrong?”, or, more appropriately, “can you judge 1000 trials as either ‘obvious’ and ‘contested’ and only be wrong at most once?”. Because originally I was imagining being a normal trial judge- but a normal trial judge has to deal with difficult cases. Ones like the Amanda Knox case (are/should be) rare. I’m pretty confident that once you put in a reasonable amount of effort (however much komponisto did for this case), you can tell whether or not the case is one you can be confident about or one you can’t, assuming you’re carefully thinking about what would make them not open-and-shut cases.