I think to do this instead of preferring certain ratios between answers, we should prefer certain answers.
Under the original scoring scheme 50:50:0:0 doesn’t score differently from 50:0:50:0 or 50:0:0:50. The average credence for each answer between those 3 is 50:17:17:17 so I’d argue that (without some external marking of which incorrect answers are more reasonable) 50:50:0:0 should score the same as 50:17:17:17.
However we could choose a marking scheme where you get back (using my framing of log scoring above):
100% of the points put on A
10% of the points put on B
10% of the points put on C
0% of the points put on D
That way 50:50:0:0 and 50:25:25:0 both end up with 55% of their points but 50:17:17:17 gets 53.4% and 50:0:0:50 gets 50%. Play around with the percentages to get rewards that seem reasonable—I think it would still be a proper scoring rule*. You could do something similar with a quadratic scoring rule.
*I think one danger is that if I am unsure but I think I can guess what the teacher thinks is reasonable/unreasonable then this might tempt me to alter my score based on something other than my actual credence levels.
This makes sense to me.
I think to do this instead of preferring certain ratios between answers, we should prefer certain answers.
Under the original scoring scheme 50:50:0:0 doesn’t score differently from 50:0:50:0 or 50:0:0:50. The average credence for each answer between those 3 is 50:17:17:17 so I’d argue that (without some external marking of which incorrect answers are more reasonable) 50:50:0:0 should score the same as 50:17:17:17.
However we could choose a marking scheme where you get back (using my framing of log scoring above):
100% of the points put on A
10% of the points put on B
10% of the points put on C
0% of the points put on D
That way 50:50:0:0 and 50:25:25:0 both end up with 55% of their points but 50:17:17:17 gets 53.4% and 50:0:0:50 gets 50%. Play around with the percentages to get rewards that seem reasonable—I think it would still be a proper scoring rule*. You could do something similar with a quadratic scoring rule.
*I think one danger is that if I am unsure but I think I can guess what the teacher thinks is reasonable/unreasonable then this might tempt me to alter my score based on something other than my actual credence levels.