The quick hack I’d use if I didn’t want people to be able to easily guess wrong with high certainty would be to use True/False or multiple choice questions. That said, I don’t currently think of this as a big problem?
There are two scores; Calibration and Correct Answers. If someone has remarkably good calibration and almost no correct answers, then they’re probably deliberately guessing outlandish answers and being sure that they’re wrong. That’s not worth bragging rights, it’s the equivalent of running to the side of the obstacles on an obstacle course. Someone who’s correctly 20% confident on most of the questions can get a lower Brier but six Correct Answer points, or an excellent Brier and zero Correct Answer points, and the former is (to me) more impressive. If you are actually totally clueless, then “[wrong answer] with probability epsilon” is actually the right response.
“I notice that I don’t actually know this” is (in my opinion) a useful skill to pick up, if you can avoid also picking up “I should pretend that I know nothing.” Still, the option to make it multiple choice exists, and there might be a better scoring rule. (I deliberately avoided making some kind of combined score, because I didn’t want less obvious strategic exchange rates between correct answers and calibration.)
The quick hack I’d use if I didn’t want people to be able to easily guess wrong with high certainty would be to use True/False or multiple choice questions. That said, I don’t currently think of this as a big problem?
There are two scores; Calibration and Correct Answers. If someone has remarkably good calibration and almost no correct answers, then they’re probably deliberately guessing outlandish answers and being sure that they’re wrong. That’s not worth bragging rights, it’s the equivalent of running to the side of the obstacles on an obstacle course. Someone who’s correctly 20% confident on most of the questions can get a lower Brier but six Correct Answer points, or an excellent Brier and zero Correct Answer points, and the former is (to me) more impressive. If you are actually totally clueless, then “[wrong answer] with probability epsilon” is actually the right response.
“I notice that I don’t actually know this” is (in my opinion) a useful skill to pick up, if you can avoid also picking up “I should pretend that I know nothing.” Still, the option to make it multiple choice exists, and there might be a better scoring rule. (I deliberately avoided making some kind of combined score, because I didn’t want less obvious strategic exchange rates between correct answers and calibration.)