Leon Lang comments on DSLT 1. The RLCT Measures the Effective Dimension of Neural Networks

Leon Lang 27 Jun 2023 1:55 UTC
1 point
0
Thanks for the answer mfar!
Yeah I remember also struggling to parse this statement when I first saw it. Liam answered but in case it’s still not clear and/or someone doesn’t want to follow up in Liam’s thesis, $x$ is a free variable, and the condition is talking about linear dependence of functions of $x$ .
Consider a toy example (not a real model) to help spell out the mathematical structure involved: Let $f (x, w) = (w_{1} + 2 w_{2}) x$ so that $\frac{\partial}{\partial w_{1}} f (x, w) = x$ and $\frac{\partial}{\partial w_{2}} f (x, w) = 2 x$ . Then let $g$ and $h$ be functions such that $g (x) = x$ and $h (x) = 2 x$ .. Then the set of functions ${g, h}$ is a linearly dependent set of functions because $h - 2 g = 0$ .
Thanks! Apparently the proof of the thing I was wondering about can be found in Lemma 3.4 in Liam’s thesis. Also thanks for your other comments!