Radical Probabilism [Transcript]

(Talk given on Sunday 21st June, over a zoom call with 40 attendees. Abram Demski is responsible for the talk, Ben Pace is responsible for the transcription)

Talk

Abram Demski: I want to talk about this idea that, for me, is an update from the logical induction result that came out of MIRI a while ago. I feel like it’s an update that I wish the entire LessWrong community had gotten from logical induction but it wasn’t communicated that well, or it’s a subtle point or something.

Abram Demski: But hopefully, this talk isn’t going to require any knowledge of logical induction from you guys. I’m actually going to talk about it in terms of philosophers who had a very similar update starting around, I think, the ’80s.

Abram Demski: There’s this philosophy called ‘radical probabilism’ which is more or less the same insight that you can get from thinking about logical induction. Radical probabilism is spearheaded by this guy Richard Jeffrey who I also like separately for the Jeffrey-Bolker axioms which I’ve written about on LessWrong.

Abram Demski: But, after the Jeffrey-Bolker axioms he was like, well, we need to revise Bayesianism even more radically than that. Specifically he zeroed in on the consequences of Dutch book arguments. So, the Dutch book arguments which are for the Kolmogorov axioms, or alternatively the Jeffrey-Bolker axioms, are pretty solid. However, you may not immediately realize that this does not imply that Bayes’ rule should be an update rule.

Abram Demski: You have Bayes’ rule as a fact about your static probabilities, that’s fine. As a fact about conditional probabilities, Bayes’ rule is just as solid as all the other probability rules. But for some reason, Bayesians take it that you start with these probabilities, you make an observation, and then you have now these probabilities. These probabilities should be updated by Bayes’ rule. And the argument for that is not super solid.

Abram Demski: There are two important flaws with the argument which I want to highlight. There is a Dutch book argument for using Bayes’ rule to update your probabilities, but it makes two critical assumptions which Jeffrey wants to relax. Assumption one is that updates are always and precisely accounted for by propositions which you learn, and everything that you learn and moves your probabilities is accounted for in this proposition. These are usually thought of as sensory data. Jeffrey said, wait a minute, my sensory data isn’t so certain. When I see something, we don’t have perfect introspective access to even just our visual field. It’s not like we get a pixel array and know exactly how everything is. So, I want to treat the things that I’m updating on as, themselves, uncertain.

Abram Demski: Difficulty two with the Dutch book argument for Bayes’ rule as an update rule, is that it assumes you know already how you would update, hypothetically, given different propositions you might observe. Then, given that assumption, you can get this argument that you need to use Bayes’ rule. Because I can Dutch-book you based on my knowledge of how you’re going to update. But if I don’t know how you’re updating, if your update has some random element, subjectively random, if I can’t predict it, then we get this radical treatment of how you’re updating. We get this picture where you believe things one day and then you can just believe different things the next day. And there’s no Dutch book I can make to say you’re irrational for doing that. “I’ve thought about it more and I’ve changed my mind.”

Abram Demski: This is very important for logical uncertainty (which Jeffrey didn’t realize because he wasn’t thinking about logical uncertainty). That’s why we came up with this philosophy, thinking about logical uncertainty. But Jeffrey came up with it just by thinking about the foundations and what we can argue a rational agent must be.

Abram Demski: So, that’s the update I want to convey. I want to convey that Bayes’ rule is not the only way that a rational agent can update. You have this great freedom of how you update.

Q&A

Ben Pace: Thank you very much, Abram. You timed yourself excellently.

Ben Pace: As I understand it, you need to have inexploitability in your belief updates and so on, such that people cannot reliably Dutch book you?

Abram Demski: Yeah. I say radical freedom meaning, if you have belief X one day and you have beliefs Y the next day, any pair of X and Y are justifiable, or potentially rational (as long as you don’t take something that has probability zero and now give it positive probability or something like that).

Abram Demski: There are rationality constraints. It’s not that you can do anything at all. The most concrete example of this is that you can’t change your mind back and forth forever on any one proposition, because then I can money-pump you. Because I know, eventually, your beliefs are going to drift up, which means I can buy low and eventually your beliefs will drift up and then I can sell the bet back to you because now you’re like, “That’s a bad bet,” and then I’ve made money off of you.

Abram Demski: If I can predict anything about how your beliefs are going to drift, then you’re in trouble. I can make money off of you by buying low and selling high. In particular that means you can’t oscillate forever, you have to eventually converge. And there’s lots of other implications.

Abram Demski: But I can’t summarize this in any nice rule is the thing. There’s just a bunch of rationality constraints that come from non-Dutch-book-ability. But there’s no nice summary of it. There’s just a bunch of constraints.

Ben Pace: I’m somewhat surprised and shocked. So, I shouldn’t be able to be exploited in any obvious way, but this doesn’t constrain me to the level of Bayes’ rule. It doesn’t constrain me to clearly knowing how my updates will be affected by future evidence.

Abram Demski: Right. If you do know your updates, then you’re constrained. He calls that the rigidity condition. And even that doesn’t imply Bayes’ rule, because of the first problem that I mentioned. So, if you do know how you’re going to update, then you don’t want to change your conditional probabilities as a result of observing something, but you can still have these uncertain observations where you move a probability but only partially. And this is called a Jeffrey update.

Ben Pace: Phil Hazelden has a question. Phil, do you want to ask your question?

Phil Hazelden: Yeah. So, you said if you don’t know how you’d update on an observation, then you get pure constraints on your belief update. I’m wondering, if someone else knows how you’d update on an observation but you don’t, does that for example, give them the power to extract money from you?

Abram Demski: Yeah, so if somebody else knows, then they can extract money if you’re not at least doing a Jeffrey update. In general, if a bookie knows something that you don’t, then a bookie can extract money from you by making bets. So this is not a proper Dutch book argument, because what we mean by a Dutch book argument is that a totally ignorant bookie can extract money.

Phil Hazelden: Thank you.

Ben Pace: I would have expected that if I was constrained to not be exploitable then this would have resulted in Bayes’ rule, but you’re saying all it actually means is there are some very basic arguments about how you shouldn’t be exploited but otherwise you can move very freely between. You can update upwards on Monday, down on Tuesday, down again on Wednesday, up on Thursday and then stay there and as long as I can’t predict it in advance, you get to do whatever the hell you like with your beliefs.

Abram Demski: Yep, and that’s rational in the sense that I think rational should mean.

Ben Pace: I do sometimes use Bayes’ rule in arguments. In fact, I’ve done it not-irregularly. Do you expect, if I fully propagate this argument I will stop using Bayes’ rule in arguments? I feel it’s very helpful for me to be able to say, all right, I was believing X on Monday and not-X on Wednesday, and let me show you the shape of my update that I made using certain probabilistic updates.

Abram Demski: Yeah, so I think that if you propagate this update you’ll notice cases where your shift simply cannot be accounted for as Bayes’ rule. But, this rigidity condition, the condition of “I already know how I would update hypothetically on various pieces of information”, the way Jeffrey talks about this (or at least the way some Jeffrey-interpreters talk about this), it’s like: if you have considered this question ahead of time, of how you would update on this particular piece of information, then your update had better be either a Bayes’ update or at least a Jeffrey update. In the cases where you think about it, it has this narrowing effect where you do indeed have to be looking more like Bayes.

Abram Demski: As an example of something that’s non-Bayesian that you might become more comfortable with if you fully propagate this: you can notice that something is amiss with your model because the evidence is less probable than you would have expected, without having an alternative that you’re updating towards. You update down your model without updating it down because of normalization constraints of updating something else up. “I’m less confident in this model now.” And somebody asks what Bayesian update did you do, and I’m like “No, it’s not a Bayesian update, it’s just that this model seems shakier.”.

Ben Pace: It’s like the thing where I have four possible hypotheses here, X, Y, Z, and “I do not have a good hypothesis here yet”. And sometimes I just move probability into “the hypothesis is not yet in my space of considerations”.

Abram Demski: But it’s like, how do you do that if “I don’t have a good hypothesis” doesn’t make any predictions?

Ben Pace: Interesting. Thanks, Abram.

[Kenny]

Are there any other detailed descriptions of what a “Jeffrey update” might look like or how one would perform one?

I think I get the point of there being “rationality constraints” that don’t, by implication, strictly require Bayesian updates. But are Jeffrey updates the entire set of possible updates that are required?

Can anyone describe a concrete example contrasting a Bayesian update and a Jeffrey update for the same circumstances, e.g. prior beliefs and new information learned?

It kinda seems like Jeffrey updates are ‘possibly rational updates’ but they’re only justified if one can perform them for no possible (or knowable) reason. That doesn’t seem practical – how could that work?

[abramdemski]

Understandable questions. I hope to expand this talk into a post which will explain things more properly.

Think of the two requirements for Bayes updates as forming a 2x2 matrix. If you have both (1) all information you learned can be summarised into one proposition which you learn with 100% confidence, and (2) you know ahead of time how you would respond to that information, then you must perform a Bayesian update. If you have (2) but not (1), ie you update some X to less than 100% confidence but you knew ahead of time how you would update to changed beliefs about X, then you are required to do a Jeffrey update. But if you don’t have (2), updates are not very constrained by Dutch-book type rationality. So in general, Jeffrey argued that there are many valid updates beyond Bayes and Jeffrey updates.

Jeffrey updates are a simple generalization of Bayes updates. When a Bayesian learns X, they update it to 100%, and take P(Y|X) to be the new P(Y) for all Y. (More formally, we want to update P to get a new probability measure Q. We do so by setting Q(Y)=P(Y|X) for all Y.) Jeffrey wanted to handle the case where you somehow become 90% confident of X, instead of fully confident. He thought this was more true to human experience. A Jeffrey update is just the weighted average of the two possible Bayesian updates. (More formally, we want to update P to get Q where Q(X)=c for some chosen c. We set Q(Y) = cP(Y|X) + (1-c)P(Y|~X).)

A natural response for a classical Bayesian is: where does 90% come from? (Where does c come from?) But the Radical Probabilism retort is: where do observations come from? The Bayesian already works in a framework where information comes in from “outside” somehow. The radical probabilist is just working in a more general framework where more general types of evidence can come in from outside.

Pearl argued against this practice in his book introducing Bayesian networks. But he introduced an equivalent—but more practical—concept which he calls virtual evidence. The Bayesian intuition freaks out at somehow updating X to 90% without any explanation. But the virtual evidence version is much more intuitive. (Look it up; I think you’ll like it better.) I don’t think virtual evidence goes against the spirit of Radical Probabilism at all, and in fact if you look at Jeffrey’s writing he appears to embrace it. So I hope to give that version in my forthcoming post, and explain why it’s nicer than Jeffrey updates in practice.

[DanielFilan]

More formally, we want to update P to get Q where Q(X)=c for some chosen c. We set Q(Y) = cP(Y|X) + (1-c)P(~Y|X).

Huh, I’m really surprised this isn’t Q(Y) = cP(Y|X) + (1-c)P(Y|~X). Was that a typo? If not, why choose your equation over mine?

[abramdemski]

Ah, yep! Corrected.

[Richard_Kennaway]

Jeffrey wanted to handle the case where you somehow become 90% confident of X, instead of fully confident

How does this differ from a Bayesian update? You can update on a new probability distribution over X just as you can on a point value. In fact, if you’re updating the probabilities in a Bayesian network, like you described, then even if the evidence you are updating on is a point value for some initial variable in the graph, the propagation steps will in general be updates on the new probability distributions for parent variables.

[Kenny]

Thanks! That answers a lot of my questions even without a concrete example.

I found this part of your reply particularly interesting:

if you don’t have (2), updates are not very constrained by Dutch-book type rationality. So in general, Jeffrey argued that there are many valid updates beyond Bayes and Jeffrey updates.

The abstract example I came up with after reading that was something like ‘I think A at 60%. If I observe X, then I’d update to A at 70%. If I observe Y, then I’d update to A at 40%. If I observe Z, I don’t know what I’d think.’.

I think what’s a little confusing is that I imagined these kinds of adjustments were already incorporated into ‘Bayesian reasoning’. Like, for the canonical ‘cancer test result’ example, we could easily adjust our understanding of ‘receives a positive test result’ to include uncertainty about the evidence itself, e.g. maybe the test was performed incorrectly or the result was misreported by the lab.

Do the ‘same’ priors cover our ‘base’ credence of different types of evidence? How are probabilities reasonably, or practically, assigned or calculated for different types of evidence? (Do we need to further adjust our confidence of those assignment or calculations?)

Maybe I do still need a concrete example to reach a decent understanding.

[abramdemski]

Richard Bradley gives an example of a non-Bayes non-Jeffrey update in Radical Probabilism and Bayesian Conditioning. He calls his third type of update Adams conditioning. But he goes even further, giving an example which is not Bayes, Jeffrey, or Adams (the example with the pipes toward the end; figure 1 and accompanying text). To be honest I still find the example a bit baffling, because I’m not clear on why we’re allowed to predictably violate the rigidity constraint in the case he considers.

I think what’s a little confusing is that I imagined these kinds of adjustments were already incorporated into ‘Bayesian reasoning’. Like, for the canonical ‘cancer test result’ example, we could easily adjust our understanding of ‘receives a positive test result’ to include uncertainty about the evidence itself, e.g. maybe the test was performed incorrectly or the result was misreported by the lab.

We can always invent a classically-bayesian scenario where we’re uncertain about some particular X, by making it so we can’t directly observe X, but rather get some other observations. EG, if we can’t directly observe the test results but we’re told about it through a fallible line of communication. What’s radical about Jeffrey’s view is to allow the observations themselves to be uncertain. So if you look at e.g. a color but aren’t sure what you’re looking at, you don’t have to contrive a color-like proposition which you do observe in order to record your imperfect observation of color.

You can think of radical probabilism as “Bayesianism at a distance”: like if you were watching a Bayesian agent, but couldn’t bother to record every single little sense-datum. You want to record that the test results are probably positive, without recording your actual observations that make you think that. We can always posit underlying observations which make the radical-probabilist agent classically Bayesian. Think of Jeffrey as pointing out that it’s often easier to work “at a distance” instead, and than once you start thinking this way, you can see it’s closer to your conscious experience anyway—so why posit underlying propositions which make all your updates into Bayes updates?

As for me, I have no problem with supposing the existence of such underlying propositions (I’ll be making a post elaborating on that at some point...) but find radical probabilism to nonetheless be a very philosophically significant point.

[Kenny]

Thanks again!

Your point about “Bayesianism at a distance” makes a lot of sense.

[David Scott Krueger (formerly: capybaralet)]

Abram Demski: But it’s like, how do you do that if “I don’t have a good hypothesis” doesn’t make any predictions?

One way you can imagine this working is that you treat “I don’t have a good hypothesis” as a special hypothesis that is not required to normalize to 1.
For instance, it could say that observing any particular real number, r, has probability epsilon > 0.
So now it “makes predictions”, but this doesn’t just collapse to including another hypothesis and using Bayes rule.

You can also imagine updating this special hypothesis (which I called a “Socratic hypothesis” in comments on the original blog post on Radical Probabilism) in various ways.

[philh]

(Transcription nitpick: IIRC I said “fewer constraints”, not “pure constraints”.)

[Raemon]

A background question I’ve had for a while: people often use Dutch Booking as an example of a failure mode you need your rationality-theory to avoid. Dutch Booking seems like a crisp, formalizable circumstance that makes it easy to think about some problems, but I’m not sure it ever comes up for me. Most people seem to avoid it via “don’t make big bets often”, rather than “make sure your beliefs are rational and inexploitable.”

Is Dutch Book supposed to be a metaphor for something that happens more frequently?

[abramdemski]

Yeah, the position in academic philosophy as I understand it is: Dutch book arguments aren’t really about betting. It’s not actually that we’re so concerned about bets. Rather, it’s a way to illustrate a kind of inconsistency. At first when I heard this I was kind of miffed about it, but now, I think it’s the right idea. I suggest reading the SEP article on Dutch Book arguments, especially Section 1.4 (which voices your concerns) and Section 2.1 or section 2 as a whole (which addresses your concerns in the way I’ve outlined).

Note, however, that we might insist that the meaning of probability is as a guide for actions, and hence, “by definition” we should take bets when they have positive expectation according to our probabilities. If we buy this, then either (1) you’re being irrational in rejecting those bets, or (2) you aren’t really reporting your probabilities in the technical sense of what-guides-your-actions, but rather some subjective assessments which may somehow be related to your true probabilities.

But if you want this kind of “fully pragmatic” notion of probability, a better place to start might be the Complete Class Theorem, which really is a consequentialist argument for having a probability distribution, unlike Dutch Books.