This is not unlike Neyman-Pearson theory. Surely this will run into the same trouble with more than 2 possible actions.
royf
Karma: 290
Our research group and collaborators, foremost Daniel Polani, have been studying this for many years now. Polani calls an essentially identical concept empowerment. These guys are welcome to the party, and as former outsiders it’s understandable (if not totally acceptable) that they wouldn’t know about these piles of prior work.
You have a good and correct point, but it has nothing to do with your question.
a machine can never halt after achieving its goal because it cannot know with full certainty whether it has achieved its goal
This is a misunderstanding of how such a machine might work.
To verify that it completed the task, the machine must match the current state to the desired state. The desired state is any state where the machine has “made 32 paperclips”. Now what’s a paperclip?
For quite some time we’ve had the technology to identify a paperclip in an image, if one exists. One lesson we’ve learned pretty well is this: don’t overfit. The paperclip you’re going to be tested on is probably not one you’ve seen before. You’ll need to know what features are common in paperclips (and less common in other objects) and how much variability they present. Tolerance to this variability will be necessary for generalization, and this means you can never be sure if you’re seeing a paperclip. In this sense there’s a limit to how well the user can specify the goal.
So after taking a few images of the paperclips it’s made, the machine’s major source of (unavoidable) uncertainty will be “is this what the user meant?”, not “am I really getting a good image of what’s on the table?”. Any half-decent implementation will go do other things (such as go ask the user).
The “world state” of ASH is in fact an “information state” of p(“heads”)>SOME_THRESHOLD
Actually, I meant p(“heads”) = 0.999 or something.
(C), if I’m following you, maps roughly to the English phrase “I know for absolutely certain that the coin is almost surely heads”.
No, I meant: “I know for absolutely certain that the coin is heads”. We agree that this much you can never know. As for getting close to this, for example having the information state (D) where p(“heads”) = 0.999999: if the world is in the state “heads”, (D) is (theoretically) possible; if the world is in the state “ASH”, (D) is impossible.
Can you give me some examples of the kinds of cases you have in mind?
Mundane examples may not be as clear, so: suppose we send a coin-flipping machine deep into intergalactic space. After a few billion years it flies permanently beyond our light cone, and then flips the coin.
Now any information state about the coin, other than complete ignorance, is physically impossible. We can still say that the coin is in one of the two states “heads” and “tails”, only unknown to us. Alternatively we can say that the coin is in a state of superposition. These two models are epistemologically equivalent.
I prefer the latter, and think many people in this community should agree, based on the spirit of other things they believe: the former model is ontologically more complicated. It’s saying more about reality than can be known. It sets the state of the coin as a free-floating property of the world, with nothing to entangle with.
I probably need to write a top-level post to explain this adequately, but in a nutshell:
I’ve tossed a coin. Now we can say that the world is in one of two states: “heads” and “tails”. This view is consistent with any information state. The information state (A) of maximal ignorance is a uniform distribution over the two states. The information state (B) where heads is twice as likely as tails is the distribution p(“heads”) = 2⁄3, p(“tails”) = 1⁄3. The information state (C) of knowing for sure that the result is heads is the distribution p(“heads”) = 1, p(“tails”) = 0.
Alternatively, we can say that the world is in one of these two states: “almost surely heads” and “almost surely tails”. Now information state (A) is a uniform distribution over these states; (B) is perhaps the distribution p(“ASH”) = 0.668, p(“AST”) = 0.332; but (C) is impossible, and so is any information state that is more certain than reality in this strange model.
Now, in many cases we can theoretically have information states arbitrarily close to complete certainty. In such cases we must use the first kind of model. So we can agree to just always use the first kind of model, and avoid all this silly complication.
But then there are cases where there are real (physical) reasons why not every information state is possible. In these cases reality is not constrained to be of the first kind, and it could be of the second kind. As a matter of fact, to say that reality is of the first kind—and that probability is only in the mind—is to say more about reality than can possibly be known. This goes against Jaynesianism.
So I completely agree that not knowing something is a property of the map rather than the territory. But an impossibility of any map to know something is a property of the territory.
To clarify further: likelihood is a relative quantity, like speed—it only has meaning relative to a specific frame of reference.
If you’re judging my calibration, the proper frame of reference is what I knew at the time of prediction. I didn’t know what the result of the fencing match would be, but I had some evidence for who is more likely to win. The (objective) probability distribution given that (subjective) information state is what I should’ve used for prediction.
If you’re judging my diligence as an evidence seeker, the proper frame of reference is what I would’ve known after reasonable information gathering. I could’ve taken some actions to put myself in a difference information state, and then my prediction could be better.
But it’s unreasonable to expect me to know the result beyond any doubt. Even if Omega is in an information state of perfectly predicting the future, this is never a proper frame of reference by which to judge bounded agents.
And this is the major point on which I’m non-Yudkowskian: since Omega is never a useful frame of reference, I’m not constraining reality to be consistent with it. In this sense, some probabilities are in the territory.
This is perhaps not the best description of actualism, but I see your point. Actualists would disagree with this part of my comment:
If I believed that “you will win” (no probability qualifier), then in the many universes where you didn’t I’m in Bayes Hell.
on the grounds that those other universes don’t exist.
But that was just a figure of speech. I don’t actually need those other universes to argue against 0 and 1 as probabilities. And if Frequentists disbelieve in that, there’s no place in Bayes Heaven for them.
we’ve already seen [...] or [...] in advance
Does this answer your question?
Predictions are justified not by becoming a reality, but by the likelihood of their becoming a reality [1]. When this likelihood is hard to estimate, we can take their becoming a reality as weak evidence that the likelihood is high. But in the end, after counting all the evidence, it’s really only the likelihood itself that matters.
If I predict [...] that I will win [...] and I in fact lose fourteen touches in a row, only to win by forfeit
If I place a bet on you to win and this happens, I’ll happily collect my prize, but still feel that I put my money on the wrong athlete. My prior and the signal are rich enough for me to deduce that your victory, although factual, was unlikely. If I believed that you’re likely to win, then my belief wasn’t “true for the wrong reasons”, it was simply false. If I believed that “you will win” (no probability qualifier), then in the many universes where you didn’t I’m in Bayes Hell.
Conversely in the other example, your winning itself is again not the best evidence for its own likelihood. Your scoring 14 touches is. My belief that you’re likely to win is true and justified for the right reasons: you’re clearly the better athlete.
[1] Where likelihood is measured either given what I know, or what I could know, or what anybody could know—depending on why we’re asking the question in the first place.
Thanks!
The best book is doubtlessly Elements of Information Theory by Cover and Thomas. It’s very clear (to someone with some background in math or theoretical computer science) and lays very strong introductory foundations before giving a good overview of some of the deeper aspects of the theory.
It’s fortunate that many concepts of information theory share some of their mathematical meaning with the everyday meaning. This way I can explain the new theory (popularized here for the first time) without defining these concepts.
I’m planning another sequence where these and other concepts will be expressed in the philosophical framework of this community. But I should’ve realized that some readers should be interested in a complete mathematical introduction. That book is what you’re looking for.
This is a perfect agent, of theoretical interest if not practically realizable.
an intelligent agent should update on it and then forget it.
Should being the operative word. This refers to a “perfect” agent (emphasis added in text; thanks!).
People don’t do this, as well they shouldn’t, because we update poorly and need the original data to compensate.
If you forget the discarded cards, and later realize that you may have an incorrect map of the deck, aren’t you SOL?
If I remember the cards in play, I don’t care about the discarded ones. If I don’t, the discarded cards could help a bit, but that’s not the heart of my problem.
if you really care about the values on that list, then there are linear aggregations
Of course existence doesn’t mean that we can actually find these coefficients. Even if you have only 2 well-defined value functions, finding an optimal tradeoff between them is generally computationally hard.
Philosiphically, yes.
Practically, it may be useful to distinguish between a coin and a toss. The coin has persisting features which make it either fair or loaded for a long time, with correlation between past and future. The toss is transient, and essentially all information about it is lost when I put the coin away—except through the memory of agents.
So yes, the toss is a feature of the present state of the world. But it has the very special property, that given the bias of the coin, the toss is independent of the past and the future. It’s sometimes more useful to treat a feature like that as an observation external to the world, but of course it “really” isn’t.
I’m trying to balance between introducing terminology to new readers and not boring those who’ve read my previous posts. Thanks for the criticism, I’ll use it (and its upvotes) to correct my balance.
Well, thank you!
Yes, I do this more for the math and the algorithms than for advice for humans.
Still, the advice is perhaps not so trivial: study not what you’re most uncertain about (highest entropy given what you know) but those things with entropy generated by what you care about. And even this advice is incomplete—there’s more to come.
When the new memory state is generated by a Bayesian update from the previous one and the new observation , it’s a sufficient statistic of these information sources for the world state , so that keeps all the information about the world that was remembered or observed:
=I(W_t;(M_{t-1},O_t)))As this is all the information available, other ways to update can only have less information.
The amount of information gained by a Bayesian update is
)-I(W_t;M_{t-1}))}{\Pr(W_t)\Pr(M_{t-1},O_t)}-\log\frac{\Pr(W_t,M_{t-1})}{\Pr(W_t)\Pr(M_{t-1})}\right])}{\Pr(W_t,M_{t-1})}-\log\frac{\Pr(M_{t-1},O_t)}{\Pr(M_{t-1})}\right])-\log\Pr(O_t%7CM_{t-1})])and because the observation only depends on the world
+S(O_t%7CM_{t-1})])
I explained this in my non-standard introduction to reinforcement learning.
We can define the world as having the Markov property, i.e. as a Markov process. But when we split the world into an agent and its environment, we lose the Markov property for each of them separately.
I’m using non-standard notation and terminology because they are needed for the theory I’m developing in these posts. In future posts I’ll try to link more to the handful of researchers who do publish on this theory. I did publish one post relating the terminology I’m using to more standard research.
Fixed. Thanks!
It seems that your research is coming around to some concepts that are at the basis of mine. Namely, that noise in an optimization process is a constraint on the process, and that the resulting constrained optimization process avoids the nasty properties you describe.
Feel free to contact me if you’d like to discuss this further.