Alexander Gietelink Oldenziel comments on Alexander Gietelink Oldenziel’s Shortform

Alexander Gietelink Oldenziel 28 Sep 2023 20:56 UTC
2 points
0
SLT and phase transitions
The morphogenetic SLT story says that during training the Bayesian posterior concentrates around a series of subspaces $W_{0} (1) ⇝ . . . ⇝ W_{0} (n)$ with rlcts $λ_{1} < . . . < λ_{n}$ and losses $L_{1} = L (w_{1}), . . ., L_{n} = L (w_{n}), w_{i} \in W_{0} (i)$ . As the size of the data sample $N$ is scaled the Bayesian posterior makes transitions $W_{0} (i) ⇝ W_{0} (i + 1)$ trading off higher complexity (higher $λ_{i + 1} > λ_{i}$ ) for better accuracy (lower loss $L_{i + 1} < L_{i}$ ).
This is the radical new framework of SLT: phase transitions happen in pure Bayesian learning as the data size is scaled.
N.B. The phase transition story actually needs a version of SLT for the nonrealizable case despite most sources focusing solely on the realizable case! The nonrealizable case makes everything more complicated and the formulas from the realizable case have to be altered.
We think of the local RLCT $λ_{w}$ at a parameter $w$ as a measure of its inherent complexity. Side-stepping the subtleties with this point of view let us take a look at Watanabe’s formula for the Bayesian generalization error:
$G_{N} (W) = L_{N} (w_{0}) + \frac{λ}{N} + o (\frac{1}{N}) \approx N L (w_{0}) + \frac{λ}{N} + o (\frac{1}{N})$
where $W$ is a neighborhood of the local minimum $w_{0}$ and $λ$ is its local RLCT. In our case $W = W_{0} (i)$ .
--EH I wanted to say something here but don’t think it makes sense on closer inspection