lberglund comments on Hessian and Basin volume

[lberglund]

Thanks for posting this.

I’m a bit confused here, when you talk about the Hessian are you talking about the Hessian evaluated at the point of minimum loss? If so, isn’t the bellow statement not strictly right?

If we start at our minimum and walk away in a principal direction, the loss as a function of distance traveled is $L\left(x\right)=\frac{1}{2}{\lambda }_i{x}^2$, where ${\lambda }_i$ is the Hessian eigenvalue for that direction.

Like, isn’t $L\left(x\right)=\frac{1}{2}{\lambda }_i{x}^2$ just an approximation of the loss here?

[Vivek Hebbar]

Yes, it is an approximation, as noted at the start of that section:

If all eigenvalues of the Hessian are positive and non trivial,^[3] we can approximate the loss as a paraboloid centered on our minimum: