dopamine or RPE or that-which-gets-discounted-and-summed-to-produce-the-return
Those are three pretty different things—the first is a chemical, the second I guess stands for ‘reward prediction error’, and the third is a mathematical quantity! Like, you also can’t talk about the expected sum of dopamine, because dopamine is a chemical, not a number!
Here’s how I interpret the paper: stuff in the world is associated with ‘rewards’, which are real numbers that represent how good the stuff is. Then the ‘return’ of some period of time is the discounted sum of rewards. Rewards represent ‘utilities’ of individual bits of time, but the return function is the actual utility function over trajectories. ‘Predictions of reward’ means predictions of stuff like bits of cheese that is associated with reward. I do think the authors do a bit of equivocation between the numbers and the things that the numbers represent (which IMO is typical for non-mathematicians, see also how physicists constantly conflate quantities like velocity with the functions that take other physical quantities and return the velocity of something), but given that AFAICT my interpretation accounts for the uses of ‘reward’ in that paper (and in the intro). That said, there are a bunch of them, and as a fallible human I’m probably not good at finding the uses that undermine my theory, so if you have a quote or two in mind that makes more sense under the interpretation that ‘reward’ refers to some function of a brain state rather than some function of cheese consumption or whatever, I’d appreciate you pointing them out to me.
Those are three pretty different things—the first is a chemical, the second I guess stands for ‘reward prediction error’, and the third is a mathematical quantity! Like, you also can’t talk about the expected sum of dopamine, because dopamine is a chemical, not a number!
Here’s how I interpret the paper: stuff in the world is associated with ‘rewards’, which are real numbers that represent how good the stuff is. Then the ‘return’ of some period of time is the discounted sum of rewards. Rewards represent ‘utilities’ of individual bits of time, but the return function is the actual utility function over trajectories. ‘Predictions of reward’ means predictions of stuff like bits of cheese that is associated with reward. I do think the authors do a bit of equivocation between the numbers and the things that the numbers represent (which IMO is typical for non-mathematicians, see also how physicists constantly conflate quantities like velocity with the functions that take other physical quantities and return the velocity of something), but given that AFAICT my interpretation accounts for the uses of ‘reward’ in that paper (and in the intro). That said, there are a bunch of them, and as a fallible human I’m probably not good at finding the uses that undermine my theory, so if you have a quote or two in mind that makes more sense under the interpretation that ‘reward’ refers to some function of a brain state rather than some function of cheese consumption or whatever, I’d appreciate you pointing them out to me.