Like David Holmes I am not an expert in tropical geometry so I can’t give the best case for why tropical geometry may be useful. Only a real expert putting in serious effort can make that case.
Let me nevertheless respond to some of your claims.
PL functions are quite natural for many reasons. They are simple. They naturally appear as minimizers of various optimization procedures, see e.g. the discussion in section 5 here.
Polynomials don’t satisfy the padding argument and architectures based on them therefore will typically fail to have the correct simplicitity bias.
As for
1.” Algebraic geometry isn’t good at dealing with deep composition of functions, and especially approximate composition.” I agree a typical course in algebraic geometry will not much consider composition of functions but that doesn’t seem to me a strong argument for the contention that the tools of algebraic geometry are not relevant here. Certainly, more sophisticated methods beyond classical scheme theory may be important [likely involving something like PROPs] but ultimately I’m not aware of any fundamental obstruction here.
2. >> I don’t agree with the contention that algebraic geometry is somehow not suited for questions of approximation. e.g. the Weil conjectures is really an approximate/ average statement about points of curves over finite fields. The same objection you make could have been made about singularity theory before we knew about SLT.
I agree with you that a probabilistic perspective on ReLUs/ piece-wise linear functions is probably important. It doesn’t seem unreasonable to me in the slightest to consider some sort of tempered posterior on the space of piecewise linear functions. I don’t think this invalidates the potential of polytope-flavored thinking.
Like David Holmes I am not an expert in tropical geometry so I can’t give the best case for why tropical geometry may be useful. Only a real expert putting in serious effort can make that case.
Let me nevertheless respond to some of your claims.
PL functions are quite natural for many reasons. They are simple. They naturally appear as minimizers of various optimization procedures, see e.g. the discussion in section 5 here.
Polynomials don’t satisfy the padding argument and architectures based on them therefore will typically fail to have the correct simplicitity bias.
As for
1.” Algebraic geometry isn’t good at dealing with deep composition of functions, and especially approximate composition.” I agree a typical course in algebraic geometry will not much consider composition of functions but that doesn’t seem to me a strong argument for the contention that the tools of algebraic geometry are not relevant here. Certainly, more sophisticated methods beyond classical scheme theory may be important [likely involving something like PROPs] but ultimately I’m not aware of any fundamental obstruction here.
2. >>
I don’t agree with the contention that algebraic geometry is somehow not suited for questions of approximation. e.g. the Weil conjectures is really an approximate/ average statement about points of curves over finite fields. The same objection you make could have been made about singularity theory before we knew about SLT.
I agree with you that a probabilistic perspective on ReLUs/ piece-wise linear functions is probably important. It doesn’t seem unreasonable to me in the slightest to consider some sort of tempered posterior on the space of piecewise linear functions. I don’t think this invalidates the potential of polytope-flavored thinking.