Re: 0%, that’s fair. Originally I included 0% because certain questions are either unanswerable (due to being blank, contextless, or whatnot) but even then there’s still a non-zero possibility of guessing the right answer out of a near-infinite number of choices.
Re: Calibration across multiple sessions. Good idea. I’ll start with a local-based solution because that would be easiest and then eventually do an account-based thing.
Re: Blank questions. Yeah, I should probably include some kind of check to see if the question is blank and skip it if so.
I considered that but I think at least for now it may just overcomplicate things for not a ton of benefit. Subjectively it seems that out of 100 questions, there are maybe 10 that I would assign the highest possible confidence. Of those I’d say only 1 out them would be questions that I’d pick 99% confidence if it were available instead of, say, 99.9%.
So assuming (incorrectly) that I’m perfectly calibrated it would take about 7000 questions in order to stand a >50% chance of seeing a meaningful difference between the two confidence levels.
Re: 0%, that’s fair. Originally I included 0% because certain questions are either unanswerable (due to being blank, contextless, or whatnot) but even then there’s still a non-zero possibility of guessing the right answer out of a near-infinite number of choices.
Re: Calibration across multiple sessions. Good idea. I’ll start with a local-based solution because that would be easiest and then eventually do an account-based thing.
Re: Blank questions. Yeah, I should probably include some kind of check to see if the question is blank and skip it if so.
Thanks! BTW, I’d prefer to have 1% and 0.1% and 99% and 99.9% as options, rather than skipping over the 1% and 99% options as you have it now.
I considered that but I think at least for now it may just overcomplicate things for not a ton of benefit. Subjectively it seems that out of 100 questions, there are maybe 10 that I would assign the highest possible confidence. Of those I’d say only 1 out them would be questions that I’d pick 99% confidence if it were available instead of, say, 99.9%.
So assuming (incorrectly) that I’m perfectly calibrated it would take about 7000 questions in order to stand a >50% chance of seeing a meaningful difference between the two confidence levels.