What you are showing with the coin is a hierarchical model over multiple coin flips, and doesn’t need new probability concepts. Let Fi be the flips. All you need in life is the distribution P(F1,F2,…). You can decide to restrict yourself to distributions of the form ∫10dpcoinP(F,G|pcoin)p(pcoin). In practice, you start out thinking about pcoin as a variable atop all the Fi in a graph, and then think in terms of P(F,G|pcoin) and p(pcoin) separately, because that’s more intuitive. This is the standard way of doing things. All you do with Ap is the same, there’s no point at which you do something different in practice, even if you ascribed additional properties to Ap in words.
This isn’t emphasized by Jaynes (though I believe it’s mentioned at the very end of the chapter), but the Ap distribution isn’t new as a formal idea in probability theory. It’s based on De Finetti’s representation theorem. The theorem concerns exchangeable sequences of random variables.
A sequence of random variables {Xi} is exchangeable if the joint distribution of any finite subsequence is invariant under permutations. A sequence of coin flips is the canonical example. Note that exchangeability does not imply independence! If I have a perfectly biased coin where I don’t know the bias, then all the random variables are perfectly dependent on each other (they all must obtain the same value).
De Finetti’s representation theorem says that any exchangeable sequence of random variables can be represented as an integral over identical and independent distributions (i.e binomial distributions). Or in other words, the extent to which random variables in the sequence are dependent on each other is solely due to their mutual relationship to the latent variable (the hidden bias of the coin).
P(X1=x1,…,Xn=xn)=∫10(nk)θk(1−θ)n−kdF(θ)
You are correct that all relevant information is contained in the joint distribution P(F1,F2,...). And while I have no deep familiarity with Bayesian hierarchical modeling, I believe your claim that the decomposition ∫10dpcoinP(F,G|pcoin)p(pcoin) is standard in Bayesian modeling.
But I think the point is that the Ap distribution is a useful conceptual tool when considering distributions governed by a time-invariant generating process. A lot of real-world processes don’t fit that description, but many do fit that description.
A concept like “the probability of me assigning a certain probability” makes sense but I don’t think Jaynes actually did anything like that for real. Here on lesswrong I guess @abramdemski knows about stuff like that.
Yes, this is correct. The part about “the probability of assigning a probability” and the part about interpreting the proposition Ap as a shorthand for an infinite collection evidences are my own interpretations of what the Ap distribution “really” means. Specifically, the part about the “probability that you will assign the probability in the infinite future” is loosely inspired by the idea of Cauchy surfaces from e.g general relativity (or any physical theory that has a causal structure built in). In general relativity, the idea is that if you have boundary conditions specified on a Cauchy surface, then you can time-evolve to solve for the distribution of matter and energy for all time. In something like quantum field theory, a principled choice for the Cauchy surface would be the infinite past (this conceptual idea shows up when understanding the vacuum in QFT). But I think in probability theory, it’s more useful conceptually to take your Cauchy surface of probabilities to be what you expect them to be in the “infinite future”. This is how I make sense of the Ap distribution.
And now that you mention it, this blog post was totally inspired by reading the first couple chapters of “Logical Inductors” (though the inspiration wasn’t conscious on my part).
--PS: I think Jaynes was great in his way of approaching the meaning and intuition of statistics, but the book is bad as a statistics textbook. It’s literally the half-complete posthumous publication of a rambling contrarian physicist, and it shows. So I would not trust any specific statistical thing he does. Taking the general vibe and ideas is good, but when you ask about a specific thing “why is nobody doing this?” it’s most likely because it’s outdated or wrong.
Not a statistician, so I will defer to your expertise that the book is bad as a statistics book (never thought of it as a statistics book to be honest). I think the strongest parts of this book are when he derives statistical mechanics from the maximum entropy principle and when he generalizes the principle of indifference to consider more general group invariances/symmetries. As far as I’m aware, my opinion on which of Jaynes’ ideas are his best ideas matches the consensus.
I suspect the reason why I like the Ap distribution is that I come from a physics background, so his reformulation of standard ideas in Bayesian modeling makes some amount of sense to me even if comes across as weird and crankish to statisticians.
Thanks for the feedback.
This isn’t emphasized by Jaynes (though I believe it’s mentioned at the very end of the chapter), but the Ap distribution isn’t new as a formal idea in probability theory. It’s based on De Finetti’s representation theorem. The theorem concerns exchangeable sequences of random variables.
A sequence of random variables {Xi} is exchangeable if the joint distribution of any finite subsequence is invariant under permutations. A sequence of coin flips is the canonical example. Note that exchangeability does not imply independence! If I have a perfectly biased coin where I don’t know the bias, then all the random variables are perfectly dependent on each other (they all must obtain the same value).
De Finetti’s representation theorem says that any exchangeable sequence of random variables can be represented as an integral over identical and independent distributions (i.e binomial distributions). Or in other words, the extent to which random variables in the sequence are dependent on each other is solely due to their mutual relationship to the latent variable (the hidden bias of the coin).
P(X1=x1,…,Xn=xn)=∫10(nk)θk(1−θ)n−kdF(θ)
You are correct that all relevant information is contained in the joint distribution P(F1,F2,...). And while I have no deep familiarity with Bayesian hierarchical modeling, I believe your claim that the decomposition ∫10 dpcoinP(F,G|pcoin)p(pcoin) is standard in Bayesian modeling.
But I think the point is that the Ap distribution is a useful conceptual tool when considering distributions governed by a time-invariant generating process. A lot of real-world processes don’t fit that description, but many do fit that description.
Yes, this is correct. The part about “the probability of assigning a probability” and the part about interpreting the proposition Ap as a shorthand for an infinite collection evidences are my own interpretations of what the Ap distribution “really” means. Specifically, the part about the “probability that you will assign the probability in the infinite future” is loosely inspired by the idea of Cauchy surfaces from e.g general relativity (or any physical theory that has a causal structure built in). In general relativity, the idea is that if you have boundary conditions specified on a Cauchy surface, then you can time-evolve to solve for the distribution of matter and energy for all time. In something like quantum field theory, a principled choice for the Cauchy surface would be the infinite past (this conceptual idea shows up when understanding the vacuum in QFT). But I think in probability theory, it’s more useful conceptually to take your Cauchy surface of probabilities to be what you expect them to be in the “infinite future”. This is how I make sense of the Ap distribution.
And now that you mention it, this blog post was totally inspired by reading the first couple chapters of “Logical Inductors” (though the inspiration wasn’t conscious on my part).
Not a statistician, so I will defer to your expertise that the book is bad as a statistics book (never thought of it as a statistics book to be honest). I think the strongest parts of this book are when he derives statistical mechanics from the maximum entropy principle and when he generalizes the principle of indifference to consider more general group invariances/symmetries. As far as I’m aware, my opinion on which of Jaynes’ ideas are his best ideas matches the consensus.
I suspect the reason why I like the Ap distribution is that I come from a physics background, so his reformulation of standard ideas in Bayesian modeling makes some amount of sense to me even if comes across as weird and crankish to statisticians.