You don’t understand. This “rationality” you speak of is monstrous irrationality. And anyway, like I said, Meta knoweth that ye have Meta-shattered values—but your wants are satisfied by serving Meta, not by serving Mammon directly. Maybe you’d get more out of reading the second half of Matthew 6 and the various analyses thereof.
You may be misinterpreting “the rationality of an agent is its goal”. Note that the original is “the light of the body is the eye”.
To put my above point a little differently: Take therefore no thought for godshatter: godshatter shall take thought for the things of itself. Sufficient unto the day is the lack-of-meta thereof.
For clarity’s sake: Yes, I vehemently dispute this idea that a goal can’t be more or less rational. That idea is wrong, which is quickly demonstrated by the fact that priors and utility functions can be transformed into each other and we have an objectively justifiable universal prior. (The general argument goes through even without such technical details of course, such that stupid “but the choice of Turing machine matters” arguments don’t distract.)
This is incorrect. Eyes absorb light and produce electrical signals interpreted as vision by the brain. Further, it seems to me that the set of thing that ‘the light of the body’ describes is an empty set; there’s no literal interpretation (our bodies do not shed visible light) and there’s no construction similar enough that suggests an interpretation (the X of the body / the light of the X). “The light of the sun” / “The light of the moon” is the closest I can find and both of those suggest the literal interpretation.
Originally, I was going to do a very charitable reading: invent a sane meaning for “The X of the Y is the sub-Y” as “sub-Y is how Y handles/uses/interpets/understands X” and say that goals, as subparts of an agent, are how an agent understands its rationality—perhaps, how an agent measures their rationality. Which is indeed how we measure our rationality, by how often we achieve our goals, but this doesn’t say anything new.
But when you say things like
You don’t understand … Maybe you’d get more out of … You may be misinterpreting
as if you were being clear in the first place, it shows me that you don’t deserve a charitable reading.
This is incorrect. Eyes absorb light and produce electrical signals interpreted as vision by the brain. Further, it seems to me that the set of thing that ‘the light of the body’ describes is an empty set; there’s no literal interpretation (our bodies do not shed visible light) and there’s no construction similar enough that suggests an interpretation (the X of the body / the light of the X). “The light of the sun” / “The light of the moon” is the closest I can find and both of those suggest the literal interpretation.
Our body does scatter visible light, though, much like the moon does.
Yes, I vehemently dispute this idea that a goal can’t be more or less [Probable to achieve higher expected utility for other agents than (any other possible goals)]
Yes, I vehemently dispute this idea that a goal can’t be more or less [Probable to achieve higher expected utility according to goal.Parent().utilityFunction].
Yes, I vehemently dispute this idea that a goal can’t be more or less [Kolmogorov-complex].
Yes, I vehemently dispute this idea that a goal can’t be more or less [optimal towards achieving your values].
Yes, I vehemently dispute this idea that a goal can’t be more or less [easy to describe as the ratio of two natural numbers].
Yes, I vehemently dispute this idea that a goal can’t be more or less [correlated in conceptspace to the values in the agent’s utility function].
Yes, I vehemently dispute this idea that a [proposed utility function] can’t be more or less rational.
Yes, I vehemently dispute this idea that a [set of predetermined criteria for building a utility function] can’t be more or less rational.
Care to enlighten me exactly on just what it is you’re disputing, and on just what points should be discussed?
Oh, also “look at the optimization targets of the processes that created the process that is me” is a short program, much shorter than needed to specify paperclip maximization, though it’s somewhat tricky because all that is modulo the symbol grounding problem. And that’s only half a meta level up, you can make it more elegant (shorter) than that.
The shorter your encoded message, the longer the encryption / compression algorithm, until eventually the algorithm is the full raw unencoded message and the encoded message is a single null-valued signal that, when received, decodes into the full message as it is contained within the algorithm.
“look at the optimization targets of the processes that created the process that is me”
My point was that it’s easier to program (“simpler”) than “maximize paperclips”, not that it’s as simple as it sounds. (Nothing is as simple as it sounds, duh.)
I fail to see how coding a meta-algorithm to select optimal extrapolation and/or simulation algorithm in order for those chosen algorithms to determine the probable optimization target (which is even harder if you want a full PA proof) is even remotely in the same order of complexity as a machine learner that uses natural selection for algorithms that increase paperclip-count, which is one of the simplest paperclip maximizers I can think of.
It might not be possible to make such a machine learner into an AGI, which is what I had in mind—narrow AIs only have “goals” and “values” and so forth in an analogical sense. Cf. derived intentionality. If it is that easy to create such an AGI, then I think I’m wrong, e.g. maybe I’m thinking about the symbol grounding problem incorrectly. I still think that in the limit of intelligence/rationality, though, specifying goals like “maximize paperclips” becomes impossible, and this wouldn’t be falsified if a zealous paperclip company were able to engineer a superintelligent paperclip maximizer that actually maximized paperclips in some plausibly commonsense fashion. In fact I can’t actually think of a way to falsify my theory in practice—I guess you’d have to somehow physically show that the axioms of algorithmic information theory and maybe updateless-like decision theories are egregiously incoherent… or something.
(Also your meta-algorithm isn’t quite what I had in mind—what I had in mind is a lot more theoretically elegant and doesn’t involve weird vague things like “extrapolation”—but I don’t think that’s the primary source of our disagreement.)
Why do you think of a statistical tendency toward higher rates of replication at the organism level when I say “the processes that created the process that is [you]”? That seems really arbitrary. Feel the inside of your teeth with your tongue. What processes generated that sensation? What decision policies did they have?
I mean that, and an infinite number of questions more and less like that, categorically, in series and in parallel. (I don’t know how to interpret “”, but I do know to interpret it that it was part of your point that it is difficult to interpret, or analogous to something that is difficult to interpret, perhaps self-similarly, or in a class of things that is analogous to something or a class of things that is difficult to interpret, perhaps self-similarly; also perhaps it has an infinite number of intended or normatively suggested interpretations more or less like those.)
(This comment also helps elucidate my previous comment, in case you had trouble understanding that comment. If you can’t understand either of these comments then maybe you should read more of the Bible, or something, otherwise you stand a decent chance of ending up in hell. This applies to all readers of this comment, not just army1987. You of course have a decent change of ending up in hell anyway, but I’m talking about marginals here, naturally.)
When Will talks about hell, or anything that sounds like a religious concept, you should suppose that in his mind it also has a computational-transhumanist meaning. I hear that in Catholicism, Hell is separation from God, and for Will, God might be something like the universal moral attractor for all post-singularity intelligences in the multiverse, so he may be saying (in the great-grandparent comment) that if you are insufficiently attentive to the question of right and wrong, your personal algorithm may never be re-instantiated in a world remade by friendly AI. To round out this guide for the perplexed, one should not think that Will is just employing a traditional language in order to express a very new concept, you need to entertain the idea that there really is significant referential overlap between what he’s talking about and what people like Aquinas were talking about—that all that medieval talk about essences, and essences of essences, and all this contemporary talk about programs, and equivalence classes of programs, might actually be referring to the same thing. One could also say something about how Will feels when he writes like this—I’d say it sometimes comes from an advanced state of whimsical despair at ever being understood—but the idea that his religiosity is a double reverse metaphor for computational eschatology is the important one. IMHO.
Thank you for the clarification, and my apologies to Will. I do have some questions, but writing a full post from the smartphone I am currently using would be tedious. I’ll wait until I get to a proper computer.
No need to apologize! If I were to get upset about being misunderstood after being purposefully rather cryptic, then I’d clearly be in the wrong. Maybe it would make some sense to apologize if you got angry at me for purposefully being cryptic, because perhaps it would be hasty to make such judgments without first trying harder to understand what sort of constraints I may be under;—but I have no idea what sort of constraints you’re under, so I have no idea whether or not it would be egregiously bad or, alternatively, supereragatorily good for you to get angry at me for not writing so as to be understood or not trying harder to be understood. But my intuition says there’s no need to apologize.
I do apologize for not being able to escape the constraints that have led me to fail to reliably communicate and thus generate a lot of noise/friction.
in his mind it also has a computational-transhumanist meaning
And a cybernetic/economic/ecological/signal-processing meaning, ethical meaning, sometimes a quantum information theoretic meaning, et cetera. I would not be justified in drawing a conclusion about the validity of a concept based on merely a perceived correspondence between two models. That’d be barely any better than talking acausal simulation seriously simply because computational metaphysics and modal-realist-like-ideas are somewhat intuitively attractive and superintelligences seem theoretically possible. One’s inferences should be based on significantly more solid foundations. I just don’t have a way to talk about equivalence classes of things while still being at all understood—then not even people like muflax could reliably understand me, and much of why I write here is to communicate with people like muflax, or angels.
Your talk of God involves a concept of “God” that applies to things that are in some major sense infinite, like a an abstract telos for all superintelligences, and things that are in some sense finite, like all particular superintelligences. Any such perceived crossing of that gap would have to be very carefully justified—e.g., I can’t think of any kind of argument that could prove that a human was the incarnation of the Word. Unlike, say, my model of that Catholics, who explicitly make such inferences on the basis of the theological virtues of faith and hope and not unaided reason as such, I don’t think you’d do be so careless in your own thinking, but I want to signal that I am not nearly so careless in my own, and that you shouldn’t think I am so careless. I think there are decent metaphysical arguments that such interactions are possible in principle, but of course such arguments would have to be made explicit and any particular mechanism (e.g. “simulation” of a human in a (finite? finite-but-perceived-to-be-infinte? infinite?) “heaven” by a finite god approximating an infinitely good telos) should not be a priori assumed to be possible. Only a moron would be so sloppy in his metaphysics and epistemology.
Not to say that you’re implying that I’m trying to convert atheists, but I’m not. I am not to be shepherd, I am not to be a gravedigger; no longer will I speak unto the people, for the last time have I spoken to the dead.
Define God. “This universe was created by an ontologically basic mental entity” (whether true or false) isn’t a goal system, unless you specify something else.
I’d go so far as arguing that preferring Believe(“This universe was created by an ontologically basic mental entity”) over Believe(null) is irrational, considering the lack of tests/evidence and, more importantly, lack of effect it has as an anticipation-controller in my mental model.
so I just realise that many of my reasons are grounded in wanting to be normal. My appreciation for normality is grounded in many antecedent assumptions I didn’t think to question until I reognised that prior assumptions are consequential to the others.
Kaj, I like your writing better than Anna Salamon’s. I feel this post is much better than the one on cached selves. The strategies mentioned there focus on bypassing a tendency. Alternatively, one could treat it as reason to be more selective with one’s social circle and benefit from people’s prosociality.
That being said, both of you do better in your informal experiments than a lot of other respected LWers. For instance, Vladimir_golovin I wouldn’t take that akrasia combat technique seriously unless someone did an experiment controlling for awareness that they are combating akrasia creating the increased perception of improvement.
which is quickly demonstrated by the fact that priors and utility functions can be transformed into each other and we have an objectively justifiable universal prior.
Really? I know that for every prior-utility function pair there are many distinct prior-utility function pairs that are equivalent in that they give rise to the same preferences under uncertainty; but I don’t know of any way to get a canonical utility function out of the universal prior — or any prior, for that matter.
the fact that priors and utility functions can be transformed into each other
Really? How?
Oh, maybe you mean that they both have the type of Universe -> Real? Although really it’s prior :: Universe -> [0, 1] and utilityfunction :: Universe -> Real assuming we have a discrete distribution on Universes. And anyway that’s no justification for substituting a prior for a utilityfunction any more than for substituting tail:: [a] -> [a] for init:: [a] -> [a]. Unless that’s not what you mean.
If you change your utility function and your prior while keeping their product constant, you’ll make the same decisions. See E.T. Jaynes, Probability Theory: The Logic of Science, chapter “Decision theory—historical background”, section “Comments”.
Right, but that still isn’t really a way to turn a prior into a utility function. A prior plus a set of decisions can determine a utility function, but you need to get the decisions from somewhere before you can do that.
Right, but you never see just a prior or just a utility function in an agent anyway. I meant that within any agent you can transform them into each other. The concepts of “prior” and “utility function” are maps, of course, not metaphysically necessary distinctions, and they don’t perfectly cut reality at its joints. Part of what’s under debate is whether we should use the Bayesian decision theoretic framework to talk about agents, especially when we have examples where AIXI-like agents fail and humans don’t. But anyway, even within the naive Bayesian decision theoretic framework, there’s transformability between beliefs and preferences. Sorry for being unclear.
To check if we agree about some basics: do we agree that decisions and decision policies—praxeology—are more fundamental than beliefs and preferences? (I’m not certain I believe this, but I will for sake of argument at least.)
I don’t know. The part I took issue with was saying that goals can be more or less rational, just based on the existence of an “objectively justifiable” universal prior. There are generally many ways to arrange heaps of pebbles into rectangles (assuming we can cut them into partial pebbles). Say that you discover that the ideal width of a pebble rectangle is 13. Well… you still don’t know what the ideal total number of pebbles is. An ideal width of 13 just gives you a preferred way to arrange any number of pebbles. It doesn’t tell you what the preferred length is, and indeed it will vary for different numbers of total pebbles.
Similarly, the important thing for an agent, the thing you can most easily measure, is the decisions they make in various situations. Given this and the “ideal objective solomonoff prior” you could derive a utility function that would explain the agent’s behaviour when combined with the solomonoff prior. But all that is is a way to divide an agent into goals and beliefs.
In other words, an “objectively justifiable” universal prior only enforces an “objectively justifiable” relation between your goals and your actions (aka. num_pebbles = 13 * length). It doesn’t tell you what your goals should be any more than it tells you what your actions should be.
I don’t know if any of that made sense, but basically it looks to me like you’re trying to solve a system of equations in three variables (prior, goals, actions) where you only have two equations (prior = X, actions = prior * goals). It doesn’t have a unique solution.
(Sorry, I commented too quickly and have been iteratively improving my comment. That said I’m only halfway trying to communicate still, so you might not get much out of it.)
You don’t understand. This “rationality” you speak of is monstrous irrationality. And anyway, like I said, Meta knoweth that ye have Meta-shattered values—but your wants are satisfied by serving Meta, not by serving Mammon directly. Maybe you’d get more out of reading the second half of Matthew 6 and the various analyses thereof.
You may be misinterpreting “the rationality of an agent is its goal”. Note that the original is “the light of the body is the eye”.
To put my above point a little differently: Take therefore no thought for godshatter: godshatter shall take thought for the things of itself. Sufficient unto the day is the lack-of-meta thereof.
For clarity’s sake: Yes, I vehemently dispute this idea that a goal can’t be more or less rational. That idea is wrong, which is quickly demonstrated by the fact that priors and utility functions can be transformed into each other and we have an objectively justifiable universal prior. (The general argument goes through even without such technical details of course, such that stupid “but the choice of Turing machine matters” arguments don’t distract.)
This is incorrect. Eyes absorb light and produce electrical signals interpreted as vision by the brain. Further, it seems to me that the set of thing that ‘the light of the body’ describes is an empty set; there’s no literal interpretation (our bodies do not shed visible light) and there’s no construction similar enough that suggests an interpretation (the X of the body / the light of the X). “The light of the sun” / “The light of the moon” is the closest I can find and both of those suggest the literal interpretation.
Originally, I was going to do a very charitable reading: invent a sane meaning for “The X of the Y is the sub-Y” as “sub-Y is how Y handles/uses/interpets/understands X” and say that goals, as subparts of an agent, are how an agent understands its rationality—perhaps, how an agent measures their rationality. Which is indeed how we measure our rationality, by how often we achieve our goals, but this doesn’t say anything new.
But when you say things like
as if you were being clear in the first place, it shows me that you don’t deserve a charitable reading.
Just interpret light as ‘that which allows one to see’. That which allows the body to see is the eye.
That which allows the agent to achieve is its goals? Seems incorrect. (Parsing rationality as “that which allows one to achieve”).
You on the other hand might get a lot out of Matthew 5. (Matthew 5 is currently my favorite part of the Bible.)
Let’s play rationalist Taboo!
Care to enlighten me exactly on just what it is you’re disputing, and on just what points should be discussed?
Edit: Fixed markdown issue, sorry!
Meh. The goal of leading to sentient beings living, to people being happy, to individuals having the freedom to control their own lives, to minds exploring new territory instead of falling into infinite loops, to the universe having a richness and complexity to it that goes beyond pebble heaps, etc. has probably much more Kolmogorov complexity than the goal of maximizing the number of paperclips in the universe. If preferring the former is irrational, I am irrational and proud of it.
Oh, also “look at the optimization targets of the processes that created the process that is me” is a short program, much shorter than needed to specify paperclip maximization, though it’s somewhat tricky because all that is modulo the symbol grounding problem. And that’s only half a meta level up, you can make it more elegant (shorter) than that.
Maybe “maximizing the number of paperclips in the universe” wasn’t the best example. “Throwing as much stuff as possible into supermassive black holes” would have been a better one.
I can only say: black holes are creepy as hell.
The shorter your encoded message, the longer the encryption / compression algorithm, until eventually the algorithm is the full raw unencoded message and the encoded message is a single null-valued signal that, when received, decodes into the full message as it is contained within the algorithm.
...isn’t nearly as short or simple as it sounds. This becomes obvious once you try to replace those words with their associated meaning.
My point was that it’s easier to program (“simpler”) than “maximize paperclips”, not that it’s as simple as it sounds. (Nothing is as simple as it sounds, duh.)
I fail to see how coding a meta-algorithm to select optimal extrapolation and/or simulation algorithm in order for those chosen algorithms to determine the probable optimization target (which is even harder if you want a full PA proof) is even remotely in the same order of complexity as a machine learner that uses natural selection for algorithms that increase paperclip-count, which is one of the simplest paperclip maximizers I can think of.
It might not be possible to make such a machine learner into an AGI, which is what I had in mind—narrow AIs only have “goals” and “values” and so forth in an analogical sense. Cf. derived intentionality. If it is that easy to create such an AGI, then I think I’m wrong, e.g. maybe I’m thinking about the symbol grounding problem incorrectly. I still think that in the limit of intelligence/rationality, though, specifying goals like “maximize paperclips” becomes impossible, and this wouldn’t be falsified if a zealous paperclip company were able to engineer a superintelligent paperclip maximizer that actually maximized paperclips in some plausibly commonsense fashion. In fact I can’t actually think of a way to falsify my theory in practice—I guess you’d have to somehow physically show that the axioms of algorithmic information theory and maybe updateless-like decision theories are egregiously incoherent… or something.
(Also your meta-algorithm isn’t quite what I had in mind—what I had in mind is a lot more theoretically elegant and doesn’t involve weird vague things like “extrapolation”—but I don’t think that’s the primary source of our disagreement.)
That means that I should try to have lots of children?
Why do you think of a statistical tendency toward higher rates of replication at the organism level when I say “the processes that created the process that is [you]”? That seems really arbitrary. Feel the inside of your teeth with your tongue. What processes generated that sensation? What decision policies did they have?
(ETA: I’d upvote my comment if I could.)
You mean, why did I bother wearing braces for years so as to have straight teeth?
I mean that, and an infinite number of questions more and less like that, categorically, in series and in parallel. (I don’t know how to interpret “”, but I do know to interpret it that it was part of your point that it is difficult to interpret, or analogous to something that is difficult to interpret, perhaps self-similarly, or in a class of things that is analogous to something or a class of things that is difficult to interpret, perhaps self-similarly; also perhaps it has an infinite number of intended or normatively suggested interpretations more or less like those.)
(This comment also helps elucidate my previous comment, in case you had trouble understanding that comment. If you can’t understand either of these comments then maybe you should read more of the Bible, or something, otherwise you stand a decent chance of ending up in hell. This applies to all readers of this comment, not just army1987. You of course have a decent change of ending up in hell anyway, but I’m talking about marginals here, naturally.)
“gd&r” is an old Usenet expression, roughly “sorry for the horrible joke”; literally “grins, ducks, and runs”.
I expect “VF” stands for “very fast”.
Comments like this are better for creating atheists, as opposed to converting them.
When Will talks about hell, or anything that sounds like a religious concept, you should suppose that in his mind it also has a computational-transhumanist meaning. I hear that in Catholicism, Hell is separation from God, and for Will, God might be something like the universal moral attractor for all post-singularity intelligences in the multiverse, so he may be saying (in the great-grandparent comment) that if you are insufficiently attentive to the question of right and wrong, your personal algorithm may never be re-instantiated in a world remade by friendly AI. To round out this guide for the perplexed, one should not think that Will is just employing a traditional language in order to express a very new concept, you need to entertain the idea that there really is significant referential overlap between what he’s talking about and what people like Aquinas were talking about—that all that medieval talk about essences, and essences of essences, and all this contemporary talk about programs, and equivalence classes of programs, might actually be referring to the same thing. One could also say something about how Will feels when he writes like this—I’d say it sometimes comes from an advanced state of whimsical despair at ever being understood—but the idea that his religiosity is a double reverse metaphor for computational eschatology is the important one. IMHO.
Thank you for the clarification, and my apologies to Will. I do have some questions, but writing a full post from the smartphone I am currently using would be tedious. I’ll wait until I get to a proper computer.
No need to apologize! If I were to get upset about being misunderstood after being purposefully rather cryptic, then I’d clearly be in the wrong. Maybe it would make some sense to apologize if you got angry at me for purposefully being cryptic, because perhaps it would be hasty to make such judgments without first trying harder to understand what sort of constraints I may be under;—but I have no idea what sort of constraints you’re under, so I have no idea whether or not it would be egregiously bad or, alternatively, supereragatorily good for you to get angry at me for not writing so as to be understood or not trying harder to be understood. But my intuition says there’s no need to apologize.
I do apologize for not being able to escape the constraints that have led me to fail to reliably communicate and thus generate a lot of noise/friction.
And a cybernetic/economic/ecological/signal-processing meaning, ethical meaning, sometimes a quantum information theoretic meaning, et cetera. I would not be justified in drawing a conclusion about the validity of a concept based on merely a perceived correspondence between two models. That’d be barely any better than talking acausal simulation seriously simply because computational metaphysics and modal-realist-like-ideas are somewhat intuitively attractive and superintelligences seem theoretically possible. One’s inferences should be based on significantly more solid foundations. I just don’t have a way to talk about equivalence classes of things while still being at all understood—then not even people like muflax could reliably understand me, and much of why I write here is to communicate with people like muflax, or angels.
Your talk of God involves a concept of “God” that applies to things that are in some major sense infinite, like a an abstract telos for all superintelligences, and things that are in some sense finite, like all particular superintelligences. Any such perceived crossing of that gap would have to be very carefully justified—e.g., I can’t think of any kind of argument that could prove that a human was the incarnation of the Word. Unlike, say, my model of that Catholics, who explicitly make such inferences on the basis of the theological virtues of faith and hope and not unaided reason as such, I don’t think you’d do be so careless in your own thinking, but I want to signal that I am not nearly so careless in my own, and that you shouldn’t think I am so careless. I think there are decent metaphysical arguments that such interactions are possible in principle, but of course such arguments would have to be made explicit and any particular mechanism (e.g. “simulation” of a human in a (finite? finite-but-perceived-to-be-infinte? infinite?) “heaven” by a finite god approximating an infinitely good telos) should not be a priori assumed to be possible. Only a moron would be so sloppy in his metaphysics and epistemology.
Not to say that you’re implying that I’m trying to convert atheists, but I’m not. I am not to be shepherd, I am not to be a gravedigger; no longer will I speak unto the people, for the last time have I spoken to the dead.
Optimization processes (mainly stupid ones such as evolution) can create subprocesses with different goals.
(And stupid ones like humans.)
(Unfortunately.)
Preferring either to God is irrational. Pride comes before the fall.
Define God. “This universe was created by an ontologically basic mental entity” (whether true or false) isn’t a goal system, unless you specify something else.
I’d go so far as arguing that preferring Believe(“This universe was created by an ontologically basic mental entity”) over Believe(null) is irrational, considering the lack of tests/evidence and, more importantly, lack of effect it has as an anticipation-controller in my mental model.
In other words, this belief does not pay rent.
In this context: perfect agent.
how considerate of you!
so I just realise that many of my reasons are grounded in wanting to be normal. My appreciation for normality is grounded in many antecedent assumptions I didn’t think to question until I reognised that prior assumptions are consequential to the others.
Kaj, I like your writing better than Anna Salamon’s. I feel this post is much better than the one on cached selves. The strategies mentioned there focus on bypassing a tendency. Alternatively, one could treat it as reason to be more selective with one’s social circle and benefit from people’s prosociality.
That being said, both of you do better in your informal experiments than a lot of other respected LWers. For instance, Vladimir_golovin I wouldn’t take that akrasia combat technique seriously unless someone did an experiment controlling for awareness that they are combating akrasia creating the increased perception of improvement.
Really? I know that for every prior-utility function pair there are many distinct prior-utility function pairs that are equivalent in that they give rise to the same preferences under uncertainty; but I don’t know of any way to get a canonical utility function out of the universal prior — or any prior, for that matter.
Really? How?
Oh, maybe you mean that they both have the type of
Universe -> Real? Although really it’sprior :: Universe -> [0, 1]andutilityfunction :: Universe -> Realassuming we have a discrete distribution on Universes. And anyway that’s no justification for substituting apriorfor autilityfunctionany more than for substituting tail:: [a] -> [a]for init:: [a] -> [a]. Unless that’s not what you mean.If you change your utility function and your prior while keeping their product constant, you’ll make the same decisions. See E.T. Jaynes, Probability Theory: The Logic of Science, chapter “Decision theory—historical background”, section “Comments”.
Right, but that still isn’t really a way to turn a prior into a utility function. A prior plus a set of decisions can determine a utility function, but you need to get the decisions from somewhere before you can do that.
Right, but you never see just a prior or just a utility function in an agent anyway. I meant that within any agent you can transform them into each other. The concepts of “prior” and “utility function” are maps, of course, not metaphysically necessary distinctions, and they don’t perfectly cut reality at its joints. Part of what’s under debate is whether we should use the Bayesian decision theoretic framework to talk about agents, especially when we have examples where AIXI-like agents fail and humans don’t. But anyway, even within the naive Bayesian decision theoretic framework, there’s transformability between beliefs and preferences. Sorry for being unclear.
To check if we agree about some basics: do we agree that decisions and decision policies—praxeology—are more fundamental than beliefs and preferences? (I’m not certain I believe this, but I will for sake of argument at least.)
I don’t know. The part I took issue with was saying that goals can be more or less rational, just based on the existence of an “objectively justifiable” universal prior. There are generally many ways to arrange heaps of pebbles into rectangles (assuming we can cut them into partial pebbles). Say that you discover that the ideal width of a pebble rectangle is 13. Well… you still don’t know what the ideal total number of pebbles is. An ideal width of 13 just gives you a preferred way to arrange any number of pebbles. It doesn’t tell you what the preferred length is, and indeed it will vary for different numbers of total pebbles.
Similarly, the important thing for an agent, the thing you can most easily measure, is the decisions they make in various situations. Given this and the “ideal objective solomonoff prior” you could derive a utility function that would explain the agent’s behaviour when combined with the solomonoff prior. But all that is is a way to divide an agent into goals and beliefs.
In other words, an “objectively justifiable” universal prior only enforces an “objectively justifiable” relation between your goals and your actions (aka.
num_pebbles = 13 * length). It doesn’t tell you what your goals should be any more than it tells you what your actions should be.I don’t know if any of that made sense, but basically it looks to me like you’re trying to solve a system of equations in three variables (prior, goals, actions) where you only have two equations (prior = X, actions = prior * goals). It doesn’t have a unique solution.
Everything you have said makes sense to me. Thanks. I will respond substantially at a later time.
How so?
(Sorry, I commented too quickly and have been iteratively improving my comment. That said I’m only halfway trying to communicate still, so you might not get much out of it.)
Why on earth would someone upvote this?
(I think they upvoted because I said “sorry”, not that part. “Sorry” generally gets upvotes.)