dsj comments on Empirical risk minimization is fundamentally confused

[dsj]

That’s because the naive inner product suggested by the risk is non-informative,

$$⟨f,g⟩=R\left(f\right)-R\left(g\right)=\int (\ell \left(f\right(x),y)-\ell \left(g\right(x),y))q(x,y)dxdy.$$

Hmm, how is this an inner product? I note that it lacks, among other properties, positive definiteness:

$⟨f,f⟩=R\left(f\right)-R\left(f\right)=0$

Edit: I guess you mean a distance metric induced by an inner product (similar to the examples later on, where you have distance metrics induced by a norm), not an actual inner product? I’m confused by the use of standard inner product notation if that’s the intended meaning. Also, in this case, this doesn’t seem to be a valid distance metric either, as it lacks symmetry and non-negativity. So I think I’m still confused as to what this is saying.

[Jesse Hoogland]

You’re right, thanks for pointing that out! I fixed the notation. Like you say, the difference of risks doesn’t even qualify as a metric (the other choices mentioned do, however).

[dsj]

Thanks! This is clearer. (To be pedantic, the ${\ell }^p$ distance should have a $p^{\text{th}}$ root around the integral, but it’s clear what you mean.)

[Jesse Hoogland]

Thank you. Pedantic is good (I fixed the root)!