the fact that priors and utility functions can be transformed into each other
Really? How?
Oh, maybe you mean that they both have the type of Universe -> Real? Although really it’s prior :: Universe -> [0, 1] and utilityfunction :: Universe -> Real assuming we have a discrete distribution on Universes. And anyway that’s no justification for substituting a prior for a utilityfunction any more than for substituting tail:: [a] -> [a] for init:: [a] -> [a]. Unless that’s not what you mean.
If you change your utility function and your prior while keeping their product constant, you’ll make the same decisions. See E.T. Jaynes, Probability Theory: The Logic of Science, chapter “Decision theory—historical background”, section “Comments”.
Right, but that still isn’t really a way to turn a prior into a utility function. A prior plus a set of decisions can determine a utility function, but you need to get the decisions from somewhere before you can do that.
Right, but you never see just a prior or just a utility function in an agent anyway. I meant that within any agent you can transform them into each other. The concepts of “prior” and “utility function” are maps, of course, not metaphysically necessary distinctions, and they don’t perfectly cut reality at its joints. Part of what’s under debate is whether we should use the Bayesian decision theoretic framework to talk about agents, especially when we have examples where AIXI-like agents fail and humans don’t. But anyway, even within the naive Bayesian decision theoretic framework, there’s transformability between beliefs and preferences. Sorry for being unclear.
To check if we agree about some basics: do we agree that decisions and decision policies—praxeology—are more fundamental than beliefs and preferences? (I’m not certain I believe this, but I will for sake of argument at least.)
I don’t know. The part I took issue with was saying that goals can be more or less rational, just based on the existence of an “objectively justifiable” universal prior. There are generally many ways to arrange heaps of pebbles into rectangles (assuming we can cut them into partial pebbles). Say that you discover that the ideal width of a pebble rectangle is 13. Well… you still don’t know what the ideal total number of pebbles is. An ideal width of 13 just gives you a preferred way to arrange any number of pebbles. It doesn’t tell you what the preferred length is, and indeed it will vary for different numbers of total pebbles.
Similarly, the important thing for an agent, the thing you can most easily measure, is the decisions they make in various situations. Given this and the “ideal objective solomonoff prior” you could derive a utility function that would explain the agent’s behaviour when combined with the solomonoff prior. But all that is is a way to divide an agent into goals and beliefs.
In other words, an “objectively justifiable” universal prior only enforces an “objectively justifiable” relation between your goals and your actions (aka. num_pebbles = 13 * length). It doesn’t tell you what your goals should be any more than it tells you what your actions should be.
I don’t know if any of that made sense, but basically it looks to me like you’re trying to solve a system of equations in three variables (prior, goals, actions) where you only have two equations (prior = X, actions = prior * goals). It doesn’t have a unique solution.
Really? How?
Oh, maybe you mean that they both have the type of
Universe -> Real? Although really it’sprior :: Universe -> [0, 1]andutilityfunction :: Universe -> Realassuming we have a discrete distribution on Universes. And anyway that’s no justification for substituting apriorfor autilityfunctionany more than for substituting tail:: [a] -> [a]for init:: [a] -> [a]. Unless that’s not what you mean.If you change your utility function and your prior while keeping their product constant, you’ll make the same decisions. See E.T. Jaynes, Probability Theory: The Logic of Science, chapter “Decision theory—historical background”, section “Comments”.
Right, but that still isn’t really a way to turn a prior into a utility function. A prior plus a set of decisions can determine a utility function, but you need to get the decisions from somewhere before you can do that.
Right, but you never see just a prior or just a utility function in an agent anyway. I meant that within any agent you can transform them into each other. The concepts of “prior” and “utility function” are maps, of course, not metaphysically necessary distinctions, and they don’t perfectly cut reality at its joints. Part of what’s under debate is whether we should use the Bayesian decision theoretic framework to talk about agents, especially when we have examples where AIXI-like agents fail and humans don’t. But anyway, even within the naive Bayesian decision theoretic framework, there’s transformability between beliefs and preferences. Sorry for being unclear.
To check if we agree about some basics: do we agree that decisions and decision policies—praxeology—are more fundamental than beliefs and preferences? (I’m not certain I believe this, but I will for sake of argument at least.)
I don’t know. The part I took issue with was saying that goals can be more or less rational, just based on the existence of an “objectively justifiable” universal prior. There are generally many ways to arrange heaps of pebbles into rectangles (assuming we can cut them into partial pebbles). Say that you discover that the ideal width of a pebble rectangle is 13. Well… you still don’t know what the ideal total number of pebbles is. An ideal width of 13 just gives you a preferred way to arrange any number of pebbles. It doesn’t tell you what the preferred length is, and indeed it will vary for different numbers of total pebbles.
Similarly, the important thing for an agent, the thing you can most easily measure, is the decisions they make in various situations. Given this and the “ideal objective solomonoff prior” you could derive a utility function that would explain the agent’s behaviour when combined with the solomonoff prior. But all that is is a way to divide an agent into goals and beliefs.
In other words, an “objectively justifiable” universal prior only enforces an “objectively justifiable” relation between your goals and your actions (aka.
num_pebbles = 13 * length). It doesn’t tell you what your goals should be any more than it tells you what your actions should be.I don’t know if any of that made sense, but basically it looks to me like you’re trying to solve a system of equations in three variables (prior, goals, actions) where you only have two equations (prior = X, actions = prior * goals). It doesn’t have a unique solution.
Everything you have said makes sense to me. Thanks. I will respond substantially at a later time.