I’m not saying “These statements can make sense”, I’m saying they do make sense and are correct under their most plain reading.
Re: a possible goal of animals being to optimize the expected sum of future rewards, in the cited paper “rewards” appears to refer to stuff like eating tasty food or mating, where it’s assumed the animal can trade those off against each other consistently:
Decision-making environments are characterized by a few key concepts: a state space..., a set of actions..., and affectively important outcomes (finding cheese, obtaining water, and winning). Actions can move the decision-maker from one state to another (i.e. induce state transitions) and they can
produce outcomes. The outcomes are assumed to have numerical (positive or negative) utilities, which can change according to the motivational state of the decision-maker (e.g. food is less valuable to a satiated animal) or direct experimental manipulation (e.g. poisoning)...
In instrumental conditioning, animals learn to choose actions to obtain rewards and avoid punishments, or, more generally to achieve goals. Various goals are possible, such as optimizing the average rate of acquisition of net rewards (i.e. rewards minus punishments), or some proxy for this such as the expected sum of future rewards[.]
It seems totally plausible to me that an animal could be motivated to optimize the expected sum of future rewards in this sense, given that ‘reward’ is basically defined as “things they value”. It seems like the way this would be false would be if animals rewards are super unstable, or the animal doesn’t coherently trade off things they value. This could happen, but I don’t see why I should see it as overwhelmingly likely.
[EDIT: in other words, the reason the paper conflates ‘rewards’ with ‘optimization target’ is that that’s how they’re defining rewards]
I’m not saying “These statements can make sense”, I’m saying they do make sense and are correct under their most plain reading.
Yup, strong disagree with that.
“rewards” appears to refer to stuff like eating tasty food or mating, where it’s assumed the animal can trade those off against each other consistently:
If that were true, that would definitely be a good counterpoint and mean I misread it. If so, I’d retract my original complaint with that passage. But I’m not convinced that it’s true. The previous paragraph just describes finding cheese as an “affectively important outcome.” Then, later, “outcomes are assumed to have numerical… utilities.” So they’re talking about utility now, OK. But then they talk about rewards. Is this utility? It’s not outcomes (like finding cheese), because you can’t take the expected sum of future finding-cheeses—type error!
When I ctrl+F rewards and scroll through, and it sure seems like they’re talking about dopamine or RPE or that-which-gets-discounted-and-summed-to-produce-the-return, which lines up with my interpretation.
dopamine or RPE or that-which-gets-discounted-and-summed-to-produce-the-return
Those are three pretty different things—the first is a chemical, the second I guess stands for ‘reward prediction error’, and the third is a mathematical quantity! Like, you also can’t talk about the expected sum of dopamine, because dopamine is a chemical, not a number!
Here’s how I interpret the paper: stuff in the world is associated with ‘rewards’, which are real numbers that represent how good the stuff is. Then the ‘return’ of some period of time is the discounted sum of rewards. Rewards represent ‘utilities’ of individual bits of time, but the return function is the actual utility function over trajectories. ‘Predictions of reward’ means predictions of stuff like bits of cheese that is associated with reward. I do think the authors do a bit of equivocation between the numbers and the things that the numbers represent (which IMO is typical for non-mathematicians, see also how physicists constantly conflate quantities like velocity with the functions that take other physical quantities and return the velocity of something), but given that AFAICT my interpretation accounts for the uses of ‘reward’ in that paper (and in the intro). That said, there are a bunch of them, and as a fallible human I’m probably not good at finding the uses that undermine my theory, so if you have a quote or two in mind that makes more sense under the interpretation that ‘reward’ refers to some function of a brain state rather than some function of cheese consumption or whatever, I’d appreciate you pointing them out to me.
I’m not saying “These statements can make sense”, I’m saying they do make sense and are correct under their most plain reading.
Re: a possible goal of animals being to optimize the expected sum of future rewards, in the cited paper “rewards” appears to refer to stuff like eating tasty food or mating, where it’s assumed the animal can trade those off against each other consistently:
It seems totally plausible to me that an animal could be motivated to optimize the expected sum of future rewards in this sense, given that ‘reward’ is basically defined as “things they value”. It seems like the way this would be false would be if animals rewards are super unstable, or the animal doesn’t coherently trade off things they value. This could happen, but I don’t see why I should see it as overwhelmingly likely.
[EDIT: in other words, the reason the paper conflates ‘rewards’ with ‘optimization target’ is that that’s how they’re defining rewards]
Yup, strong disagree with that.
If that were true, that would definitely be a good counterpoint and mean I misread it. If so, I’d retract my original complaint with that passage. But I’m not convinced that it’s true. The previous paragraph just describes finding cheese as an “affectively important outcome.” Then, later, “outcomes are assumed to have numerical… utilities.” So they’re talking about utility now, OK. But then they talk about rewards. Is this utility? It’s not outcomes (like finding cheese), because you can’t take the expected sum of future finding-cheeses—type error!
When I ctrl+F rewards and scroll through, and it sure seems like they’re talking about dopamine or RPE or that-which-gets-discounted-and-summed-to-produce-the-return, which lines up with my interpretation.
Those are three pretty different things—the first is a chemical, the second I guess stands for ‘reward prediction error’, and the third is a mathematical quantity! Like, you also can’t talk about the expected sum of dopamine, because dopamine is a chemical, not a number!
Here’s how I interpret the paper: stuff in the world is associated with ‘rewards’, which are real numbers that represent how good the stuff is. Then the ‘return’ of some period of time is the discounted sum of rewards. Rewards represent ‘utilities’ of individual bits of time, but the return function is the actual utility function over trajectories. ‘Predictions of reward’ means predictions of stuff like bits of cheese that is associated with reward. I do think the authors do a bit of equivocation between the numbers and the things that the numbers represent (which IMO is typical for non-mathematicians, see also how physicists constantly conflate quantities like velocity with the functions that take other physical quantities and return the velocity of something), but given that AFAICT my interpretation accounts for the uses of ‘reward’ in that paper (and in the intro). That said, there are a bunch of them, and as a fallible human I’m probably not good at finding the uses that undermine my theory, so if you have a quote or two in mind that makes more sense under the interpretation that ‘reward’ refers to some function of a brain state rather than some function of cheese consumption or whatever, I’d appreciate you pointing them out to me.