Neural networks with ReLU activation are the things you obtain when you combine two kinds of linearity, namely the standard linearity that we all should be familiar with and tropical linearity.
Give R two operations ⊗,⊕ defined by setting x⊕y=max(x,y),x⊗y=x+y. Then the operations ⊕,⊗ are associative, commutative, and they satisfy the distributivity property x⊗(y⊕z)=(x⊗y)⊕(x⊗z). We shall call the operations ⊕,⊗ tropical operations on R.
We can even perform matrix and vector operations by replacing the operations +,⋅ with their tropical counterparts ⊕,⊗. More explicitly, we can define the tropical matrix addition and multiplication operations by setting
(ai,j)i,j⊕(bi,j)i,j=(ai,j⊕bi,j)i,j=(max(ai,j,bi,j))i,j and
Here, the ReLU operation is just ReLU(x)=x⊕0, and if 0 is the zero vector and v is a real vector, then ReLU(v)=v⊕0, so ReLU does not even rely on tropical matrix multiplication.
Of course, one can certainly construct and train neural networks using tropical matrix multiplication in the layers of the form v↦B⊗(Av+a)⊕b where A,B are weight matrices and a,b are bias vectors, but I do not know of any experiments done with these sorts of neural networks, so I am uncertain of what advantages they offer.
Since ReLU neural networks are a combination of two kinds of linearity, one might expect for ReLU neural networks to behave nearly linearly. And it is not surprising that ReLU networks look more like the standard linear transformations than the tropical linear transformations since the standard linear transformations in a neural network are far more complex than the ReLU. ReLU just provides the bare minimum non-linearity for a neural network without doing anything fancy.
Neural networks with ReLU activation are the things you obtain when you combine two kinds of linearity, namely the standard linearity that we all should be familiar with and tropical linearity.
Give R two operations ⊗,⊕ defined by setting x⊕y=max(x,y),x⊗y=x+y. Then the operations ⊕,⊗ are associative, commutative, and they satisfy the distributivity property x⊗(y⊕z)=(x⊗y)⊕(x⊗z). We shall call the operations ⊕,⊗ tropical operations on R.
We can even perform matrix and vector operations by replacing the operations +,⋅ with their tropical counterparts ⊕,⊗. More explicitly, we can define the tropical matrix addition and multiplication operations by setting
(ai,j)i,j⊕(bi,j)i,j=(ai,j⊕bi,j)i,j=(max(ai,j,bi,j))i,j and
(ai,j)i,j⊗(bi,j)i,j=(⊕kai,k⊗bk,j)i,j=(maxk(ai,k+bk,j))i,j.
Here, the ReLU operation is just ReLU(x)=x⊕0, and if 0 is the zero vector and v is a real vector, then ReLU(v)=v⊕0, so ReLU does not even rely on tropical matrix multiplication.
Of course, one can certainly construct and train neural networks using tropical matrix multiplication in the layers of the form v↦B⊗(Av+a)⊕b where A,B are weight matrices and a,b are bias vectors, but I do not know of any experiments done with these sorts of neural networks, so I am uncertain of what advantages they offer.
Since ReLU neural networks are a combination of two kinds of linearity, one might expect for ReLU neural networks to behave nearly linearly. And it is not surprising that ReLU networks look more like the standard linear transformations than the tropical linear transformations since the standard linear transformations in a neural network are far more complex than the ReLU. ReLU just provides the bare minimum non-linearity for a neural network without doing anything fancy.