Ulisse Mini comments on Neural Tangent Kernel Distillation

[Ulisse Mini]

I believe you’ve got a typo in the defn of $K$

$$K(x,x^{\prime })=\parallel x\parallel \parallel x\parallel \kappa \left(\frac{x\cdot x^{\prime }}{\parallel x\parallel \parallel x\parallel }\right)$$

Shouldn’t it be $\parallel x\parallel \parallel x^{\prime }\parallel$?

Also it appears like you can simplify

$$\kappa \left(u\right):=u\frac{1}{\pi }(\pi -\arccos (u\left)\right)+\frac{1}{\pi }\left(u\right(\pi -\arccos \left(u\right))+\sqrt{1-u^2})$$ Into $$\kappa \left(u\right):=2u\frac{1}{\pi }(\pi -\arccos (u\left)\right)+\frac{1}{\pi }\sqrt{1-u^2}$$ Not sure if this wasn’t done for some reason or it was a typo.

[Jeremy Gillen]

Thanks, you are right on both. I don’t know how I missed the simplification, I remember wanting to make the analytical form as simple as possible.

I really should have added the reference for this, since I just copied it from a paper, so I did that in a footnote. I just followed up the derivation a bit further and the parts I checked seem solid, but annoying that it’s spread out over three papers.