We take “planning” to include things that are relevantly similar to this procedure, such as following a bag of heuristics that approximates it.
In theory, optimal policies could be tabularly implemented. In this case, it is impossible for them to further improve their “planning.” Yet optimal policies tend to seek power and pursue convergent instrumental subgoals, such as staying alive.
So I’m not (yet) convinced that this frame is useful reductionism for better understanding subgoals. It feels somewhat unnatural to me, although I am also getting a tad more S1-excited about the frame as I type this comment.
In particular, I think this is a great point:
Instrumental goals are about passing the buck: if you are a planner, and you can’t achieve your final goal with a single obvious action (or sequence of actions), you can instead pass the buck to something else, typically your future self. There will often be obvious available actions that put the receiver of the buck “closer” to achieving the final goal than you.
This at least rings true in my experience—when I don’t know what to do for my research, I’ll idle by “powering up” and reading more textbooks, delegating to my future self (and also allowing time for subconscious brainstorming).
I realized we forgot to put in the footnotes! There was one footnote which was pretty important, I’ll put it here because it’s related to what you said. It was a footnote after the “make the planners with your goal better at planning” sub-maxim.
This was an “aha” moment for me: Even such everyday actions as “briefly glance up from your phone so you can see where you are going when walking through a building” are instances of following this maxim! You are looking up from your phone so that you can acquire more relevant data (the location of the door, the location of the door handle, etc.) for your immediate-future-self to make use of. Your immediate-future-self will have a slightly better world-model as a result, and thus be better than you at making plans. In particular, your immediate future self will be able, e.g., to choose the correct moment & location to grab the door handle, by contrast with your present self who is looking at Twitter and does not know where to grab.
In theory, optimal policies could be tabularly implemented. In this case, it is impossible for them to further improve their “planning.”
That sounds wrong. Planning as defined in this post is sufficiently broad that acting like a planner makes you a planner. So if you unwrap a structural planner into a tabular policy, the latter would improve its planning (for example by taking actions that instrumentally help it accomplish the goal we can best ascribe it using the intentional stance).
Another way of framing the point IMO is that the OPs define planning in terms of computation instead of algorithm, and so planning better means facilitating or making the following part of the computation more efficient.
In theory, optimal policies could be tabularly implemented. In this case, it is impossible for them to further improve their “planning.” Yet optimal policies tend to seek power and pursue convergent instrumental subgoals, such as staying alive.
So I’m not (yet) convinced that this frame is useful reductionism for better understanding subgoals. It feels somewhat unnatural to me, although I am also getting a tad more S1-excited about the frame as I type this comment.
In particular, I think this is a great point:
This at least rings true in my experience—when I don’t know what to do for my research, I’ll idle by “powering up” and reading more textbooks, delegating to my future self (and also allowing time for subconscious brainstorming).
I realized we forgot to put in the footnotes! There was one footnote which was pretty important, I’ll put it here because it’s related to what you said. It was a footnote after the “make the planners with your goal better at planning” sub-maxim.
Footnotes are now added in (thanks Ramana!)
That sounds wrong. Planning as defined in this post is sufficiently broad that acting like a planner makes you a planner. So if you unwrap a structural planner into a tabular policy, the latter would improve its planning (for example by taking actions that instrumentally help it accomplish the goal we can best ascribe it using the intentional stance).
Another way of framing the point IMO is that the OPs define planning in terms of computation instead of algorithm, and so planning better means facilitating or making the following part of the computation more efficient.