Suppose the student thinks that there is a 10% chance that he is right, and the reward structure is +5/-1 for confidence interval 1.
In fact, make the reward structure:(right/wrong) 1⁄0, 6/-1, 10/-3, 13/-6, 15/-10, 16/-15
That puts the breakpoints at roughly even intervals, keeps the math easy, and with a little bit of clarifying exactly where the breakpoints are, doesn’t reward someone who accurately determines their accuracy and then lies about it.
I sat down late last night trying to prove that this couldn’t work and instead proved that it could. If I did this correctly, in order for it to work, with the confidences increasing from 0 to 1,
left side confidence ⇐ (difference in Y)/(difference in X + difference in Y)
right side confidence >= (difference in Y)/(difference in X + difference in Y).
Differences in X are 5, 4, 3, 2, 1 and differences in Y are 1, 2, 3, 4, 5 leading to values of 1⁄6 through 5⁄6; as 0 < 1⁄6 < 1⁄5 < 2⁄6 < 2⁄5 < 3⁄6 < 3⁄5 < 4⁄6 < 4⁄5 < 5⁄6 < 1 this is immune to lying within a single interval (and also turns out to be so for multiple intervals).
So, what are the downsides of making this a grading standard? The biggest one I see is that it would be unfair except in classes that have as prerequisites an outstanding score in a class that covers credence calibration.
Suppose the student thinks that there is a 10% chance that he is right, and the reward structure is +5/-1 for confidence interval 1.
In fact, make the reward structure:(right/wrong) 1⁄0, 6/-1, 10/-3, 13/-6, 15/-10, 16/-15
That puts the breakpoints at roughly even intervals, keeps the math easy, and with a little bit of clarifying exactly where the breakpoints are, doesn’t reward someone who accurately determines their accuracy and then lies about it.
I sat down late last night trying to prove that this couldn’t work and instead proved that it could. If I did this correctly, in order for it to work, with the confidences increasing from 0 to 1,
left side confidence ⇐ (difference in Y)/(difference in X + difference in Y)
right side confidence >= (difference in Y)/(difference in X + difference in Y).
Differences in X are 5, 4, 3, 2, 1 and differences in Y are 1, 2, 3, 4, 5 leading to values of 1⁄6 through 5⁄6; as 0 < 1⁄6 < 1⁄5 < 2⁄6 < 2⁄5 < 3⁄6 < 3⁄5 < 4⁄6 < 4⁄5 < 5⁄6 < 1 this is immune to lying within a single interval (and also turns out to be so for multiple intervals).
So, what are the downsides of making this a grading standard? The biggest one I see is that it would be unfair except in classes that have as prerequisites an outstanding score in a class that covers credence calibration.