Thanks a lot for posting this! A minor point about the 2nd intuition pump (100-timesteps, 4 actions: Take $1, Do Nothing, Buy Apple, Buy Banana; the point being that most action sequences take the Take $1 action a lot rather than the Do Nothing action): the “goal” of getting 3 apples seems irrelevant to the point, and may be misleading if you think that that goal is where the push to acquire resources comes from. A more central source seems to me to be the “rule” of not ending with a negative balance: this is what prunes paths through the tree that contain more “do nothing” actions.
Yup! More generally, key pieces for modeling a “resource”: amounts of the resource are additive, and more resources open up more actions (operationalized by the need for a positive balance in this case). If there’s something roughly like that in the problem space, then the resource-seeking argument kicks in.
Thanks a lot for posting this! A minor point about the 2nd intuition pump (100-timesteps, 4 actions: Take $1, Do Nothing, Buy Apple, Buy Banana; the point being that most action sequences take the Take $1 action a lot rather than the Do Nothing action): the “goal” of getting 3 apples seems irrelevant to the point, and may be misleading if you think that that goal is where the push to acquire resources comes from. A more central source seems to me to be the “rule” of not ending with a negative balance: this is what prunes paths through the tree that contain more “do nothing” actions.
Yup! More generally, key pieces for modeling a “resource”: amounts of the resource are additive, and more resources open up more actions (operationalized by the need for a positive balance in this case). If there’s something roughly like that in the problem space, then the resource-seeking argument kicks in.