Many questions on Good Judgment Open seem to fit your hard characteristics if not your preferred characteristics. Might you consider running a challenge for people’s Brier score on GJO (if Ts & Cs allow), or cribbing some questions? An advantage here is that judging is settled for you, and people have evidence to model GJO’s resolution
I need to take a good look over what GJO has to offer here- I’m not sure if running a challenge for score on it would meet the goals here well (in particular I think it needs to be bounded in amount of prediction it requires in order to motivate doing it, and yet not gameable by just doing easy questions, and I’d like to be able to see what the probability assignments on specific questions were), but I’ve not looked at it closely with this in mind. I should at least hopefully be able to crib a few questions, or more.
Many questions on Good Judgment Open seem to fit your hard characteristics if not your preferred characteristics. Might you consider running a challenge for people’s Brier score on GJO (if Ts & Cs allow), or cribbing some questions? An advantage here is that judging is settled for you, and people have evidence to model GJO’s resolution
I need to take a good look over what GJO has to offer here- I’m not sure if running a challenge for score on it would meet the goals here well (in particular I think it needs to be bounded in amount of prediction it requires in order to motivate doing it, and yet not gameable by just doing easy questions, and I’d like to be able to see what the probability assignments on specific questions were), but I’ve not looked at it closely with this in mind. I should at least hopefully be able to crib a few questions, or more.