Leon Lang comments on DSLT 1. The RLCT Measures the Effective Dimension of Neural Networks

[Leon Lang]

Thanks for the answer Liam! I especially liked the further context on the connection between Bayesian posteriors and SGD. Below a few more comments on some of your answers:

The partition function is equal to the model evidence $Z_n=p\left(D_n\right)$, yep. It isn’t equal to $p\left(\right(Y_i\left)|\right(X_i\left)\right),$(I assume $i$ is fixed here?) but is instead expressed in terms of the model likelihood and prior (and can simply be thought of as the “normalising constant” of the posterior),

$$p\left(D_n\right)={\int }_W\phi \left(w\right)\prod _{i=1}^{n}p(y_i,x_i|w)dw$$

and then under this supervised learning setup where we know $q\left(x_i\right)$, we have $p(y_i,x_i|w)=p(y_i|x_i,w)q\left(x_i\right)$. Also note that this does “factor over $i$” (if I’m interpreting you correctly) since the data is independent and identically distributed.

I think I still disagree. I think everything in these formulas needs to be conditioned on the $X$-part of the dataset. In particular, I think the notation $p\left(D_n\right)$ is slightly misleading, but maybe I’m missing something here.

I’ll walk you through my reasoning: When I write $\left(X_i\right)$ or $\left(Y_i\right)$, I mean the whole vectors, e.g., $(X_i{)}_{i=1,\dots ,n}$. Then I think the posterior compuation works as follows:

$$p(w\mid D_n)=p(w\mid \left(Y_i\right),\left(X_i\right))=\frac{p(\left(Y_i\right)\mid \left(X_i\right),w)\cdot p(w\mid \left(X_i\right))}{p(\left(Y_i\right)\mid \left(X_i\right))}.$$

That is just Bayes rule, conditioned on $\left(X_i\right)$ in every term. Then, $p(w\mid \left(X_i\right))=\phi \left(w\right)$ because from $X$alone you don’t get any new information about the conditional $q(Y\mid X)$ (A more formal way to see this is to write down the Bayesian network of the model and to see that $w$ and $X_i$ are d-separated). Also, conditioned on $w$, $p$ is independent over data points, and so we obtain

$$p(w\mid D_n)=\frac{1}{p(\left(Y_i\right)\mid \left(X_i\right))}\cdot e^{-n{L}_n\left(w\right)}\cdot \phi \left(w\right).$$

So, comparing with your equations, we must have $Z_n=p(\left(Y_i\right)\mid \left(X_i\right)).$ Do you think this is correct?

Btw., I still don’t think this “factors over $i$”. I think that

$Z_n\ne {\prod }_{i=1}^{n}p(Y_i\mid X_i).$

The reason is that old data points should inform the parameter $w$, which should have an influence on future updates. I think the independence assumption only holds for the true distribution and the model conditioned on $w$.

If you expand that term out you find that

$$\begin{array}{cc}& {\int }_W(w-w_0{)}^T\frac{\partial \phi }{\partial w}{|}_{w=w_0}\exp (-(w-w_0{)}^{T}I(w_0\left)\right(w-w_0\left)\right)dw\\ & =\frac{\partial \phi }{\partial w}{|}_{w=w_0}{\int }_W(w-w_0{)}^T\exp (-(w-w_0{)}^{T}I(w_0\left)\right(w-w_0\left)\right)dw=0\end{array}$$

because the second integral is the first central moment of a Gaussian. The derivative of the prior is irrelevant.

Right. that makes sense, thank you! (I think you missed a factor of $n/2$, but that doesn’t change the conclusion)

Thanks also for the corrected volume formula, it makes sense now :)