If you move a small distance ds in parallel to the manifold, then your distance from the manifold goes as ds2
This doesn’t make sense to me, why is this true?
My alternate intuition is that the directional derivative (which is a dot product between the gradient and ds) along the manifold is zero because we aren’t changing our behavior on the training set.
The directional derivative is zero, so the change is zero to first order. The second order term can exist. (Consider what happens if you move along the tangent line to a circle. The distance from the circle goes ~quadratically, since the circle looks locally parabolic.) Hence ds2.
This doesn’t make sense to me, why is this true?
My alternate intuition is that the directional derivative (which is a dot product between the gradient and ds) along the manifold is zero because we aren’t changing our behavior on the training set.
The directional derivative is zero, so the change is zero to first order. The second order term can exist. (Consider what happens if you move along the tangent line to a circle. The distance from the circle goes ~quadratically, since the circle looks locally parabolic.) Hence ds2.
I see, thanks!