Very interesting and well done. You didn’t detail the methodology much. One issue I am interested in is subconscious cheating in the following way. I make a bunch of predictions saying “their is a 80% chance of X”, and I find that only 60% of these X are actually occurring. So I recalibrate (which is fair). However, lets say I start putting a load of things that really should be in my 95% bucket into the 80% bucket, so calibrate too far the other way. I went from being wrong in one direction to wrong in the other, but the average will still look good. A lot of mistakes in one direction can be (consciously or not) made up for by intentionally leaning too far the other way later.
Very interesting and well done. You didn’t detail the methodology much. One issue I am interested in is subconscious cheating in the following way. I make a bunch of predictions saying “their is a 80% chance of X”, and I find that only 60% of these X are actually occurring. So I recalibrate (which is fair). However, lets say I start putting a load of things that really should be in my 95% bucket into the 80% bucket, so calibrate too far the other way. I went from being wrong in one direction to wrong in the other, but the average will still look good. A lot of mistakes in one direction can be (consciously or not) made up for by intentionally leaning too far the other way later.