I’m not saying “These statements can make sense”, I’m saying they do make sense and are correct under their most plain reading.
Yup, strong disagree with that.
“rewards” appears to refer to stuff like eating tasty food or mating, where it’s assumed the animal can trade those off against each other consistently:
If that were true, that would definitely be a good counterpoint and mean I misread it. If so, I’d retract my original complaint with that passage. But I’m not convinced that it’s true. The previous paragraph just describes finding cheese as an “affectively important outcome.” Then, later, “outcomes are assumed to have numerical… utilities.” So they’re talking about utility now, OK. But then they talk about rewards. Is this utility? It’s not outcomes (like finding cheese), because you can’t take the expected sum of future finding-cheeses—type error!
When I ctrl+F rewards and scroll through, and it sure seems like they’re talking about dopamine or RPE or that-which-gets-discounted-and-summed-to-produce-the-return, which lines up with my interpretation.
dopamine or RPE or that-which-gets-discounted-and-summed-to-produce-the-return
Those are three pretty different things—the first is a chemical, the second I guess stands for ‘reward prediction error’, and the third is a mathematical quantity! Like, you also can’t talk about the expected sum of dopamine, because dopamine is a chemical, not a number!
Here’s how I interpret the paper: stuff in the world is associated with ‘rewards’, which are real numbers that represent how good the stuff is. Then the ‘return’ of some period of time is the discounted sum of rewards. Rewards represent ‘utilities’ of individual bits of time, but the return function is the actual utility function over trajectories. ‘Predictions of reward’ means predictions of stuff like bits of cheese that is associated with reward. I do think the authors do a bit of equivocation between the numbers and the things that the numbers represent (which IMO is typical for non-mathematicians, see also how physicists constantly conflate quantities like velocity with the functions that take other physical quantities and return the velocity of something), but given that AFAICT my interpretation accounts for the uses of ‘reward’ in that paper (and in the intro). That said, there are a bunch of them, and as a fallible human I’m probably not good at finding the uses that undermine my theory, so if you have a quote or two in mind that makes more sense under the interpretation that ‘reward’ refers to some function of a brain state rather than some function of cheese consumption or whatever, I’d appreciate you pointing them out to me.
Yup, strong disagree with that.
If that were true, that would definitely be a good counterpoint and mean I misread it. If so, I’d retract my original complaint with that passage. But I’m not convinced that it’s true. The previous paragraph just describes finding cheese as an “affectively important outcome.” Then, later, “outcomes are assumed to have numerical… utilities.” So they’re talking about utility now, OK. But then they talk about rewards. Is this utility? It’s not outcomes (like finding cheese), because you can’t take the expected sum of future finding-cheeses—type error!
When I ctrl+F rewards and scroll through, and it sure seems like they’re talking about dopamine or RPE or that-which-gets-discounted-and-summed-to-produce-the-return, which lines up with my interpretation.
Those are three pretty different things—the first is a chemical, the second I guess stands for ‘reward prediction error’, and the third is a mathematical quantity! Like, you also can’t talk about the expected sum of dopamine, because dopamine is a chemical, not a number!
Here’s how I interpret the paper: stuff in the world is associated with ‘rewards’, which are real numbers that represent how good the stuff is. Then the ‘return’ of some period of time is the discounted sum of rewards. Rewards represent ‘utilities’ of individual bits of time, but the return function is the actual utility function over trajectories. ‘Predictions of reward’ means predictions of stuff like bits of cheese that is associated with reward. I do think the authors do a bit of equivocation between the numbers and the things that the numbers represent (which IMO is typical for non-mathematicians, see also how physicists constantly conflate quantities like velocity with the functions that take other physical quantities and return the velocity of something), but given that AFAICT my interpretation accounts for the uses of ‘reward’ in that paper (and in the intro). That said, there are a bunch of them, and as a fallible human I’m probably not good at finding the uses that undermine my theory, so if you have a quote or two in mind that makes more sense under the interpretation that ‘reward’ refers to some function of a brain state rather than some function of cheese consumption or whatever, I’d appreciate you pointing them out to me.