I disagree with this framing. Sure, if you have 5 different cakes, you can eat some and have some. But for any particular cake, you can’t do both. Similarly, if you face 5 (or infinitely many) identical decision problems, you can choose to be updateful in some of them (thus obtaining useful Value of Information, that increases your utility in some worlds), and updateless in others (thus obtaining useful strategic coherence, that increases your utility in other worlds). The fundamental dichotomy remains as sharp, and it’s misleading to imply we can surmount it. It’s great to discuss, given this dichotomy, which trade-offs we humans are more comfortable making. But I’ve felt this was obscured in many relevant conversations.
I don’t get your disagreement. If your view is that you can’t eat one cake and keep it too, and my view is that you can eat some cakes and keep other cakes, isn’t the obvious conclusion that these two views are compatible?
I would also argue that you can slice up a cake and keep some slices but eat others (this corresponds to mixed strategies), but this feels like splitting hairs rather than getting at some big important thing. My view is mainly about iterated situations (more than one cake).
Maybe your disagreement would be better stated in a way that didn’t lean on the cake analogy?
My point is that the theoretical work you are shooting for is so general that it’s closer to “what sorts of AI designs (priors and decision theories) should always be implemented”, rather than “what sorts of AI designs should humans in particular, in this particular environment, implement”. And I think we won’t gain insights on the former, because there are no general solutions, due to fundamental trade-offs (“no-free-lunchs”). I think we could gain many insights on the former, but that the methods better fit for that are less formal/theoretical and way messier/”eye-balling”/iterating.
Well, one way to continue this debate would be to discuss the concrete promising-ness of the pseudo-formalisms discussed in the post. I think there are some promising-seeming directions.
Another way to continue the debate would be to discuss theoretically whether theoretical work can be useful.
It sort of seems like your point is that theoretical work always needs to be predicated on simplifying assumptions. I agree with this, but I don’t think it makes theoretical work useless. My belief is that we should continue working to make the assumptions more and more realistic, but the ‘essential picture’ is often preserved under this operation. (EG, Newtonian gravity and general relativity make most of the same predictions in practice. Kolmogorov axioms vindicated a lot of earlier work on probability theory.)
I don’t get your disagreement. If your view is that you can’t eat one cake and keep it too, and my view is that you can eat some cakes and keep other cakes, isn’t the obvious conclusion that these two views are compatible?
I would also argue that you can slice up a cake and keep some slices but eat others (this corresponds to mixed strategies), but this feels like splitting hairs rather than getting at some big important thing. My view is mainly about iterated situations (more than one cake).
Maybe your disagreement would be better stated in a way that didn’t lean on the cake analogy?
Well, one way to continue this debate would be to discuss the concrete promising-ness of the pseudo-formalisms discussed in the post. I think there are some promising-seeming directions.
Another way to continue the debate would be to discuss theoretically whether theoretical work can be useful.
It sort of seems like your point is that theoretical work always needs to be predicated on simplifying assumptions. I agree with this, but I don’t think it makes theoretical work useless. My belief is that we should continue working to make the assumptions more and more realistic, but the ‘essential picture’ is often preserved under this operation. (EG, Newtonian gravity and general relativity make most of the same predictions in practice. Kolmogorov axioms vindicated a lot of earlier work on probability theory.)