ML-related math trick: I find it easier to imagine a 4D tensor, say of dimensions B×F×M×N, as a big matrix with dimensions B×F within which are nested matrices of dimensions M×N. The nice thing about this is, at least for me, it makes it easier to imagine applying operations over the B×F matrices in parallel, which is something I’ve had to thing about a number of times doing ML-related programming, e.g. trying to figure out how write the code to apply a 1D convolution-like operation to an entire batch in parallel.
I’ve been studying tensor decompositions and approximate tensor formats for half a year. Since I’ve learned about tensor networks, I’ve noticed that I can draw them to figure out how to code some linear operations on tensors.
Once I used this to figure out how to implement backward method of some simple neural network layer (not something novel, it was for the sake of learning how deep learning frameworks work). Another time I needed to figure out how to implement forward method for a Conv2d layer with weights tensor in CP format. After drawing its output as a tensor network diagram, it was clear that I could just do a sequence of 3 Conv2d layers: pointwise, depthwise, pointwise.
I am not saying that you should learn tensor networks, it’s probably a lot of buck for not too large bang unless you want to work with tensor decompositions and formats.
From cursory Googling, it looks like tensor networks are mostly used for understanding quantum systems. I’m not opposed to learning about them, but is there a good resource you can point me to that introduces them independent of the physics concepts? Were you learning them for use in physics?
For example, have you happened to read this Google AI paper introducing their TensorNetworks library and giving an overview?
Unfortunately I don’t know any quantum stuff. I learned them for machine learning purposes.
A monograph by Cichocki et al. (part 1, part 2) is an overview of how tensor decompositions, tensor formats, and tensor networks can be used in machine learning and signal processing. I think it lacks some applications, including acceleration and compression of neural networks by compression of weights of layers using tensor decompositions (this also sometimes improves accuracy, probably by reducing overfit).
Tensor decompositions and Applications by Kolda, Bader 2009 - this is an overview of tensor decompositions. It doesn’t have many machine learning applications. Also it doesn’t talk of tensor networks, only about some simplest tensor decompositions and specific tensor formats which are the most popular types of tensor networks. This paper was the first thing I read about all the tensor stuff, and it’s one of the easier things to read. I recommend you read it first and then look at the topics that seem interesting to you in Cichocki et al.
Tensor spaces and numerical tensor calculus—Hackbusch 2012 - this textbook covers mathematics of tensor formats and tensor decompositions for hilbert and banach spaces. No applications, a lot of math, functions analysis is kinda a prerequisite. Very dense and difficult to read textbook. Also doesn’t talk of tensor networks, only about specific tensor formats.
Handwaving and interpretive dance—This is simple, it’s about tensor networks, not other tensor stuff. It’s for physicists but chapter 1 and maybe other chapters can be read without physics background.
Regarding the TensorNetwork library. I’ve skim-read it. I haven’t tried using it. I think it’s in early alpha or something. How usable it is for me depends on how well it can interact with pytorch and how easy it is to do autodifferentiation w.r.t. core tensors and use the tensor network in a pytorch model. Intuitively the API seemed nice. I think their idea is to that you take a tensor, make it into a matrix, do truncated svd, now you have 2 matrices, turn them back to tensors. Now you do the same for them. This way you can perform some but not all popular tensor decomposition algorithms.
P.S. Fel free to message me if you have questions about tensor decomposition/network/formats stuff
ML-related math trick: I find it easier to imagine a 4D tensor, say of dimensions B×F×M×N, as a big matrix with dimensions B×F within which are nested matrices of dimensions M×N. The nice thing about this is, at least for me, it makes it easier to imagine applying operations over the B×F matrices in parallel, which is something I’ve had to thing about a number of times doing ML-related programming, e.g. trying to figure out how write the code to apply a 1D convolution-like operation to an entire batch in parallel.
I’ve been studying tensor decompositions and approximate tensor formats for half a year. Since I’ve learned about tensor networks, I’ve noticed that I can draw them to figure out how to code some linear operations on tensors.
Once I used this to figure out how to implement backward method of some simple neural network layer (not something novel, it was for the sake of learning how deep learning frameworks work). Another time I needed to figure out how to implement forward method for a Conv2d layer with weights tensor in CP format. After drawing its output as a tensor network diagram, it was clear that I could just do a sequence of 3 Conv2d layers: pointwise, depthwise, pointwise.
I am not saying that you should learn tensor networks, it’s probably a lot of buck for not too large bang unless you want to work with tensor decompositions and formats.
From cursory Googling, it looks like tensor networks are mostly used for understanding quantum systems. I’m not opposed to learning about them, but is there a good resource you can point me to that introduces them independent of the physics concepts? Were you learning them for use in physics?
For example, have you happened to read this Google AI paper introducing their TensorNetworks library and giving an overview?
Unfortunately I don’t know any quantum stuff. I learned them for machine learning purposes.
A monograph by Cichocki et al. (part 1, part 2) is an overview of how tensor decompositions, tensor formats, and tensor networks can be used in machine learning and signal processing. I think it lacks some applications, including acceleration and compression of neural networks by compression of weights of layers using tensor decompositions (this also sometimes improves accuracy, probably by reducing overfit).
Tensor decompositions and Applications by Kolda, Bader 2009 - this is an overview of tensor decompositions. It doesn’t have many machine learning applications. Also it doesn’t talk of tensor networks, only about some simplest tensor decompositions and specific tensor formats which are the most popular types of tensor networks. This paper was the first thing I read about all the tensor stuff, and it’s one of the easier things to read. I recommend you read it first and then look at the topics that seem interesting to you in Cichocki et al.
Tensor spaces and numerical tensor calculus—Hackbusch 2012 - this textbook covers mathematics of tensor formats and tensor decompositions for hilbert and banach spaces. No applications, a lot of math, functions analysis is kinda a prerequisite. Very dense and difficult to read textbook. Also doesn’t talk of tensor networks, only about specific tensor formats.
Handwaving and interpretive dance—This is simple, it’s about tensor networks, not other tensor stuff. It’s for physicists but chapter 1 and maybe other chapters can be read without physics background.
Regarding the TensorNetwork library. I’ve skim-read it. I haven’t tried using it. I think it’s in early alpha or something. How usable it is for me depends on how well it can interact with pytorch and how easy it is to do autodifferentiation w.r.t. core tensors and use the tensor network in a pytorch model. Intuitively the API seemed nice. I think their idea is to that you take a tensor, make it into a matrix, do truncated svd, now you have 2 matrices, turn them back to tensors. Now you do the same for them. This way you can perform some but not all popular tensor decomposition algorithms.
P.S. Fel free to message me if you have questions about tensor decomposition/network/formats stuff