Garrett Baker comments on Performing an SVD on a time-series matrix of gradient updates on an MNIST network produces 92.5 singular values

[Garrett Baker]

1. This is a good hypothesis, and seems like it can be checked by removing the first however many timesteps from the svd calculation

2. I have! I tried the same thing on a simpler network trained on an algorithmic task, and got similar results. In that case I got 10 singular vectors on 8M(?) time steps.