Rationalists are often overconfident (see: SSC calibration questions) but believe they are well calibrated (bias blind spot, also just knowing about a bias is not enough to unbias you)
If you’re referring to the calibration questions on the 2014 LW survey, rationalists were pretty well calibrated on them (though a bit overconfident). I described some analyses of the data here and here, and here’s a picture:
(where the amount of overconfidence is shown by how far the blue dots are below the black line)
I don’t know of any data on whether rationalists believe they are well calibrated on these sorts of questions—I suspect that a fair number of people would guess that they are overconfident.
I’ll also note here that I’m planning to do some analyses of the calibration questions on the 2016 LW Diaspora Survey during the next month. I think that there are issues with some of the questions that were on the survey, so before I do any analyses I’ll note that my preferred analyses will only include 4 of the questions:
Which is heavier, a virus or a prion? What year was the fast food chain “Dairy Queen” founded? (Within five years) Without counting, how many keys on a standard IBM keyboard released after 1986, within ten? What’s the diameter of a standard soccerball, in cm within 2?
For thoroughness I will also do some secondary analyses which include 7 questions, those 4 plus the following 3 (even though I think that these 3 questions have some issues which make them less good as tests of calibration):
I’m thinking of a number between one and ten, what is it? Alexander Hamilton appears on how many distinct denominations of US Currency? How many calories in a reese’s peanut butter cup within 20?
If you’re referring to the calibration questions on the 2014 LW survey, rationalists were pretty well calibrated on them (though a bit overconfident). I described some analyses of the data here and here, and here’s a picture:
(where the amount of overconfidence is shown by how far the blue dots are below the black line)
I don’t know of any data on whether rationalists believe they are well calibrated on these sorts of questions—I suspect that a fair number of people would guess that they are overconfident.
I’ll also note here that I’m planning to do some analyses of the calibration questions on the 2016 LW Diaspora Survey during the next month. I think that there are issues with some of the questions that were on the survey, so before I do any analyses I’ll note that my preferred analyses will only include 4 of the questions:
Which is heavier, a virus or a prion?
What year was the fast food chain “Dairy Queen” founded? (Within five years)
Without counting, how many keys on a standard IBM keyboard released after 1986, within ten?
What’s the diameter of a standard soccerball, in cm within 2?
For thoroughness I will also do some secondary analyses which include 7 questions, those 4 plus the following 3 (even though I think that these 3 questions have some issues which make them less good as tests of calibration):
I’m thinking of a number between one and ten, what is it?
Alexander Hamilton appears on how many distinct denominations of US Currency?
How many calories in a reese’s peanut butter cup within 20?