My first impulse is to generalize all that and get out of the morality domain (because of mind-killer potential).
You’re interested in ranking a set of choices. You have multiple models (or functions) which are capable of evaluating your choices, but you don’t know which model is the correct one—or, indeed, if any of them is the correct one. However you can express your confidence in each of these models, e.g. by assigning a probability-of-being-correct to it.
There are a few wrinkles (e.g. are all models mutually exclusive?) but otherwise it looks like a standard statistics problem, no?
My first impulse is to generalize all that and get out of the morality domain (because of mind-killer potential).
I originally did it outside the morality domain (just some innocent VNM math) for that reason. I think it does have to be dropped back into the intended domain eventually, though.
Do you think that there’s any particular way that I did get hit by the mindkiller? It’s possible.
...
Yes, we have a bunch of models, unsure which (but we can assume completeness and mutual exclusion because we can always extend it to “all possible models”), want to get useful results even with uncertainty. What does useful results mean?
If this were a model of a process, we would not be able to go farther than a simple probability distribution without some specification of what we want from it. For example, if we are designing a machine and have uncertainty about what material it will be made of, and want to know how thick some member should be, the thickness we want depends how much additional thickness costs and the consequences of failure. The usual solution is to define a utility function over the situation and then frame the thing as a decision.
I guess we could do the same here with some kind of utility function over mistakes, but I’m not sure what that would look like yet.
Do you have some other idea in mind? Do you think this angle is worth looking into?
Nah, I don’t think you had mind-killer problems, but as more people wander into the discussion, the potential is there...
For models, it’s not an automatic assumption to assume completeness and mutual exclusion. In particular, mutual exclusion means only one model is right and all the others are wrong—we just don’t know which one. That’s a very black-and-white approach. It’s not as simple but more realistic (for appropriate values of “realistic”) to assume that all models are wrong but some are more wrong than others. Then the correctness of a model is a real number, not a boolean value (and that’s separate from what we know and believe about models), and more, you can then play with dependency structures (e.g. a correlation matrix) which essentially say “if model X is relatively-more-correct then model Y is also relatively-more-correct and model X is relatively-more-wrong”.
Without dependencies this doesn’t look complicated. If the models (and their believability, aka the probability of being the Correct One) are independent then the probabilities are summable. You basically weight the rankings (or scores) by models’ probabilities and you’re good to go. With dependencies it gets more interesting.
I am not sure why you’re questioning what are useful results in this setting. As formulated, the useful result is a list of available choices ranked by estimated utility. Ultimately, you just want to know the top choice, you don’t even care much about how others rank.
My first impulse is to generalize all that and get out of the morality domain (because of mind-killer potential).
You’re interested in ranking a set of choices. You have multiple models (or functions) which are capable of evaluating your choices, but you don’t know which model is the correct one—or, indeed, if any of them is the correct one. However you can express your confidence in each of these models, e.g. by assigning a probability-of-being-correct to it.
There are a few wrinkles (e.g. are all models mutually exclusive?) but otherwise it looks like a standard statistics problem, no?
Why is this downvoted? It’s a perfectly reasonable constructive suggestion.
I originally did it outside the morality domain (just some innocent VNM math) for that reason. I think it does have to be dropped back into the intended domain eventually, though.
Do you think that there’s any particular way that I did get hit by the mindkiller? It’s possible.
Yes, we have a bunch of models, unsure which (but we can assume completeness and mutual exclusion because we can always extend it to “all possible models”), want to get useful results even with uncertainty. What does useful results mean?
If this were a model of a process, we would not be able to go farther than a simple probability distribution without some specification of what we want from it. For example, if we are designing a machine and have uncertainty about what material it will be made of, and want to know how thick some member should be, the thickness we want depends how much additional thickness costs and the consequences of failure. The usual solution is to define a utility function over the situation and then frame the thing as a decision.
I guess we could do the same here with some kind of utility function over mistakes, but I’m not sure what that would look like yet.
Do you have some other idea in mind? Do you think this angle is worth looking into?
Nah, I don’t think you had mind-killer problems, but as more people wander into the discussion, the potential is there...
For models, it’s not an automatic assumption to assume completeness and mutual exclusion. In particular, mutual exclusion means only one model is right and all the others are wrong—we just don’t know which one. That’s a very black-and-white approach. It’s not as simple but more realistic (for appropriate values of “realistic”) to assume that all models are wrong but some are more wrong than others. Then the correctness of a model is a real number, not a boolean value (and that’s separate from what we know and believe about models), and more, you can then play with dependency structures (e.g. a correlation matrix) which essentially say “if model X is relatively-more-correct then model Y is also relatively-more-correct and model X is relatively-more-wrong”.
Without dependencies this doesn’t look complicated. If the models (and their believability, aka the probability of being the Correct One) are independent then the probabilities are summable. You basically weight the rankings (or scores) by models’ probabilities and you’re good to go. With dependencies it gets more interesting.
I am not sure why you’re questioning what are useful results in this setting. As formulated, the useful result is a list of available choices ranked by estimated utility. Ultimately, you just want to know the top choice, you don’t even care much about how others rank.