This is a cool idea in theory, but imagine how it would play out in reality when billions of dollars are at stake. Who decides the damage amount and the probabilities involved and how? Even if these were objectively computable and independent of metaethical uncertainty, the incentives for distorting them would be immense. This only seems feasible when damages and risks are well understood and there is consensus around an agreed-upon causal model.
And then we also have the whole moral hazzard problem with those types of incentives. Could I put myself at a little risk of some AI damages that might be claimed to have much broader potential?
This is a cool idea in theory, but imagine how it would play out in reality when billions of dollars are at stake. Who decides the damage amount and the probabilities involved and how? Even if these were objectively computable and independent of metaethical uncertainty, the incentives for distorting them would be immense. This only seems feasible when damages and risks are well understood and there is consensus around an agreed-upon causal model.
And then we also have the whole moral hazzard problem with those types of incentives. Could I put myself at a little risk of some AI damages that might be claimed to have much broader potential?