OK, so all that makes sense and seems basically correct, but I don’t see how you get from there to being able to map confidence for persons across a question the same way you can for questions across a person.
Adopting that terminology, I’m saying for a typical Less Wrong user, they likely have a similar understanding-the-question module. This module will be right most of the time and wrong some of the time, so they correctly apply the outside view error afterwards on each of their estimates. Since the understanding-the-question module is similar for each person, though, the actual errors aren’t evenly distributed across questions, so they will underestimate on “easy” questions and overestimate on “hard” ones, if easy and hard are determined afterwards by percentage that get the answer correct.
Since the understanding-the-question module is similar for each person, though, the actual errors aren’t evenly distributed across questions, so they will underestimate on “easy” questions and overestimate on “hard” ones, if easy and hard are determined afterwards by percentage that get the answer correct.
That seems reasonable to me, yes, as an easy way for a question to be ‘hard’ is if most answerers interpret it differently from the questioner.
OK, so all that makes sense and seems basically correct, but I don’t see how you get from there to being able to map confidence for persons across a question the same way you can for questions across a person.
Adopting that terminology, I’m saying for a typical Less Wrong user, they likely have a similar understanding-the-question module. This module will be right most of the time and wrong some of the time, so they correctly apply the outside view error afterwards on each of their estimates. Since the understanding-the-question module is similar for each person, though, the actual errors aren’t evenly distributed across questions, so they will underestimate on “easy” questions and overestimate on “hard” ones, if easy and hard are determined afterwards by percentage that get the answer correct.
That seems reasonable to me, yes, as an easy way for a question to be ‘hard’ is if most answerers interpret it differently from the questioner.