...shades into decision theory...Models that take longer to compute data probabilities should similarly have a probability penalty, not simply because they’re implausible, but because we don’t want to use them unless the data force us to.
Whoa! that sounds dangerous! Why not keep the beliefs and costs separate and only apply this penalty at the decision theory stage?
Well, I said shaded into the lines of decision theory...
Yes, it absolutely is dangerous, and thinking about it more I agree it should not be done this way. Probability penalties do not scale correctly with the data collected: they’re essentially just a fixed offset. Modified utility of using a particular method really is different. If a method is unusable, we shouldn’t use it, and methods that trade off accuracy for manageability should be decided at that level, once we can judge the accuracy—not earlier.
EDIT: I suppose I was hoping for a valid way of justifying the fact that we throw out models that are too hard to use or analyze—they never make it into our set of hypotheses in the first place. It’s amazing how often conjugate priors “just happen” to be chosen...
Whoa! that sounds dangerous! Why not keep the beliefs and costs separate and only apply this penalty at the decision theory stage?
Well, I said shaded into the lines of decision theory...
Yes, it absolutely is dangerous, and thinking about it more I agree it should not be done this way. Probability penalties do not scale correctly with the data collected: they’re essentially just a fixed offset. Modified utility of using a particular method really is different. If a method is unusable, we shouldn’t use it, and methods that trade off accuracy for manageability should be decided at that level, once we can judge the accuracy—not earlier.
EDIT: I suppose I was hoping for a valid way of justifying the fact that we throw out models that are too hard to use or analyze—they never make it into our set of hypotheses in the first place. It’s amazing how often conjugate priors “just happen” to be chosen...