You only score the predictions that actually have the court decide the outcome or that have the prosecutor drop the charges without a deal and not those that get decided by plea deals.
Suppose that the prosecutor has some random noise in their charges, such that they sometimes overcharge a bunch and sometimes undercharge a bunch. In that case it seems reasonable to suppose that things are more likely to go to court when the prosecutor is overcharging and the accused therefore thinks they can get more of the accusations dropped. But this would mean that the prosecutors are evaluated on a subset of the charges that are systematically too high, and therefore to compensate they end up lowering their assessed probabilities below what is actually counterfactually accurate if people just went to court about it always.
I don’t know how big a problem this would be, but it seems like something that would be good to evaluate in the proposal.
But this would mean that the prosecutors are evaluated on a subset of the charges that are systematically too high, and therefore to compensate they end up lowering their assessed probabilities below what is actually counterfactually accurate if people just went to court about it always.
While I do think such an effect could exist, it should be relatively constant and I would expect defense attornies to understand the effect and correct for it in their decisions when they advice their clients.
You only score the predictions that actually have the court decide the outcome or that have the prosecutor drop the charges without a deal and not those that get decided by plea deals.
Suppose that the prosecutor has some random noise in their charges, such that they sometimes overcharge a bunch and sometimes undercharge a bunch. In that case it seems reasonable to suppose that things are more likely to go to court when the prosecutor is overcharging and the accused therefore thinks they can get more of the accusations dropped. But this would mean that the prosecutors are evaluated on a subset of the charges that are systematically too high, and therefore to compensate they end up lowering their assessed probabilities below what is actually counterfactually accurate if people just went to court about it always.
I don’t know how big a problem this would be, but it seems like something that would be good to evaluate in the proposal.
While I do think such an effect could exist, it should be relatively constant and I would expect defense attornies to understand the effect and correct for it in their decisions when they advice their clients.