On the other hand, Naturalized Decision Theory is not about resource bounded Expected Utility Theory.
I think there’s a sense in which I buy this but it might be worth explaining more.
My current suspicion is that “agents that have utility functions over the outcome of the physics they are embedded in” is not the right concept for understanding naturalized agency (in particular, the “motive forces” of the things that emerge from processes like abiogenesis/evolution/culture/AI research). This concept is often argued for using dutch-book arguments (e.g. VNM). I think these arguments are probably invalid when applied to naturalized agents (if taken literally they assume something like a “view from nowhere” and unbounded computation, etc). As such, re-examining what arguments can be made about coherent naturalized agency while avoiding inscription errors* seems like a good path towards recovering the correct concepts for thinking about naturalized agency.
*I’m getting the term “inscription error” from Brian Cantwell Smith (On the Origin of Objects, p. 50):
It is a phenomenon that I will in general call an inscription error: a tendency for a theorist or observer, first, to write or project or impose or inscribe a set of ontological assumptions onto a computational system (onto the system itself, onto the task domain, onto the relation between
the two, and so forth), and then, second, to read those assumptions or their consequences back off the system, as if that constituted an independent empirical discovery or theoretical result.
I think there’s a sense in which I buy this but it might be worth explaining more.
My current suspicion is that “agents that have utility functions over the outcome of the physics they are embedded in” is not the right concept for understanding naturalized agency (in particular, the “motive forces” of the things that emerge from processes like abiogenesis/evolution/culture/AI research). This concept is often argued for using dutch-book arguments (e.g. VNM). I think these arguments are probably invalid when applied to naturalized agents (if taken literally they assume something like a “view from nowhere” and unbounded computation, etc). As such, re-examining what arguments can be made about coherent naturalized agency while avoiding inscription errors* seems like a good path towards recovering the correct concepts for thinking about naturalized agency.
*I’m getting the term “inscription error” from Brian Cantwell Smith (On the Origin of Objects, p. 50):