I definitely did have the “ammunition for the enemy” feeling about your post, and the “belief attire” point is a good one, but I think the broad emotional disagreement does express itself in a few specific claims:
Even if you were to control for getting tired and hungry and so on, even if you were to load your intelligence into a computer and have it do the hard work, I still don’t think you could judge a thousand such trials and be wrong only once. I admit this may not be as real a disagreement as I’m thinking, because it may be a confusion on what sort of reference class we should use to pick trials for you.
I think we might disagree on the Lord Kelvin claim. I think I would predict more of today’s physical theories are wrong than you would.
I think my probability that God exists would be several orders of magnitude higher than yours, even though I think you probably know about the same number of good arguments on the issue as I do.
Maybe our disagreement can be resolved empirically—if we were to do enough problems where we gave confidence levels on questions like “The area of Canada is greater than the area of the Mediterranean Sea” and use log odds scoring we might find one of us doing significantly better than the other—although we would have to do quite a few to close off my possible argument that we just didn’t hit that one “black swan” question on which you’d say you’re one in a million confident and then get it wrong. Would you agree that this would get to the heart of our disagreement, or do you think it revolves solely around more confusing philosophical questions?
(I took a test like that yesterday to test something and I came out overconfident, missing 2⁄10 questions at the 96% probability level. I don’t know how that translates to more real-world questions and higher confidence levels, but it sure makes me reluctant to say I’m chronically underconfident)
I still don’t think you could judge a thousand such trials and be wrong only once.
When I first saw this, I agreed with it. But now I don’t, partly because of the story (which I don’t have a link to, but it was linked to from LW somewhere) about someone who would bet they knew whether or not a number was a prime. This continued until they made a mistake (doing it mentally), and then they lost.
If they had a calculator, could they go up to the 1000th odd number and be wrong at most once? I’m pretty sure they could, actually. And so the question isn’t “can you judge 1000 trials and only get one wrong?” but “can you judge 1000 obvious trials and only get one wrong?”, or, more appropriately, “can you judge 1000 trials as either ‘obvious’ and ‘contested’ and only be wrong at most once?”. Because originally I was imagining being a normal trial judge- but a normal trial judge has to deal with difficult cases. Ones like the Amanda Knox case (are/should be) rare. I’m pretty confident that once you put in a reasonable amount of effort (however much komponisto did for this case), you can tell whether or not the case is one you can be confident about or one you can’t, assuming you’re carefully thinking about what would make them not open-and-shut cases.
I definitely did have the “ammunition for the enemy” feeling about your post, and the “belief attire” point is a good one, but I think the broad emotional disagreement does express itself in a few specific claims:
Even if you were to control for getting tired and hungry and so on, even if you were to load your intelligence into a computer and have it do the hard work, I still don’t think you could judge a thousand such trials and be wrong only once. I admit this may not be as real a disagreement as I’m thinking, because it may be a confusion on what sort of reference class we should use to pick trials for you.
I think we might disagree on the Lord Kelvin claim. I think I would predict more of today’s physical theories are wrong than you would.
I think my probability that God exists would be several orders of magnitude higher than yours, even though I think you probably know about the same number of good arguments on the issue as I do.
Maybe our disagreement can be resolved empirically—if we were to do enough problems where we gave confidence levels on questions like “The area of Canada is greater than the area of the Mediterranean Sea” and use log odds scoring we might find one of us doing significantly better than the other—although we would have to do quite a few to close off my possible argument that we just didn’t hit that one “black swan” question on which you’d say you’re one in a million confident and then get it wrong. Would you agree that this would get to the heart of our disagreement, or do you think it revolves solely around more confusing philosophical questions?
(I took a test like that yesterday to test something and I came out overconfident, missing 2⁄10 questions at the 96% probability level. I don’t know how that translates to more real-world questions and higher confidence levels, but it sure makes me reluctant to say I’m chronically underconfident)
When I first saw this, I agreed with it. But now I don’t, partly because of the story (which I don’t have a link to, but it was linked to from LW somewhere) about someone who would bet they knew whether or not a number was a prime. This continued until they made a mistake (doing it mentally), and then they lost.
If they had a calculator, could they go up to the 1000th odd number and be wrong at most once? I’m pretty sure they could, actually. And so the question isn’t “can you judge 1000 trials and only get one wrong?” but “can you judge 1000 obvious trials and only get one wrong?”, or, more appropriately, “can you judge 1000 trials as either ‘obvious’ and ‘contested’ and only be wrong at most once?”. Because originally I was imagining being a normal trial judge- but a normal trial judge has to deal with difficult cases. Ones like the Amanda Knox case (are/should be) rare. I’m pretty confident that once you put in a reasonable amount of effort (however much komponisto did for this case), you can tell whether or not the case is one you can be confident about or one you can’t, assuming you’re carefully thinking about what would make them not open-and-shut cases.