PCA and other dimensionality reduction techniques are great, but there’s another very useful technique that most people (even statisticians) are unaware of: dimensional analysis, and in particular, the Buckingham pi theorem. For some reason, this technique is used primarily by engineers in fluid dynamics and heat transfer despite its broad applicability. This is the technique that allows scale models like wind tunnels to work, but it’s more useful than just allowing for scaling. I find it very useful to reduce the number of variables when developing models and conducting experiments.
Dimensional analysis recognizes a few basic axioms about models with dimensions and sees what they imply. You can use these to construct new variables from the old variables. The model is usually complete in a smaller number of these new variables. The technique does not tell you which variables are “correct”, just how many independent ones are needed. Identifying “correct” variables requires data, domain knowledge, or both. (And sometimes, there’s no clear “best” variable; multiple work equivalently well.)
Dimensional analysis does not help with categorical variables, or numbers which are already dimensionless (though by luck, sometimes combinations of dimensionless variables are actually what’s “correct”). This is the main restriction that applies. And you can expect at best a reduction in the number of variables of about 3. Dimensional analysis is most useful for physical problems with maybe 3 to 10 variables.
The basic idea is this: Dimensions are some sort of metadata which can tell you something about the structure of the problem. You can always rewrite a dimensional equation, for example, to be dimensionless on both sides. You should notice that some terms become constants when this is done, and that simplifies the equation.
Here’s a physical example: Let’s say you want to measure the drag on a sphere (units: N). You know this depends on the air speed (units: m/s), viscosity (units: m^2/s), air density (units: kg/m^3), and the diameter of the sphere (units: m). So, you have 5 variables in total. Let’s say you want to do a factorial design with 4 levels in each variable, with no replications. You’d have to do 4^4 = 256 experiments. This is clearly too complicated.
What fluid dynamicists have recognized is that you can rewrite the relationship in terms of different variables, and nothing is missing. The Buckingham pi theorem mentioned previously says that we only need 2 dimensionless variables given our 5 dimensional variables. So, instead of the drag force, you use the drag coefficient, and instead of the speed, viscosity, etc., you use the Reynolds number. Now, you only need to do 4 experiments to get the same level of representation.
As it turns out, you can use techniques like PCA on top of dimensional analysis to determine that certain dimensionless parameters are unimportant (there are other ways too). This further simplifies models.
There’s a lot more on this topic than what I have covered and mentioned here. I would recommend reading the book Dimensional analysis and the theory of models for more details and the proof of the pi theorem.
In general, if your problem displays any kind of symmetry* you can exploit that to simplify things. I think most people are capable of doing this intuitively when the symmetry is obvious. The Buckingham pi theorem is a great example of a systematic way to find and exploit a symmetry that isn’t so obvious.
* By “symmetry” I really mean “invariance under a group of transformations”.
This is a great point. Other than fairly easy geometric and time symmetries, do you have any advice or know of any resources which might be helpful towards finding these symmetries?
Here’s what I do know: Sometimes you can recognize these symmetries by analyzing a model differential equation. Here’s a book on the subject that I haven’t read, but might read in the future. My PhD advisor tells me I already know one reliable way to find these symmetries (e.g., like how to find the change of variables used here), so reading this would be a poor use of time in his view. This approach also requires knowing a fair bit more about a phenomena than just which variables it depends on.
The book you linked is the sort of thing I had in mind. The historical motivation for Lie groups was to develop a systematic way to use symmetry to attack differential equations.
This is a great point. Other than fairly easy geometric and time symmetries, do you have any advice or know of any resources which might be helpful towards finding these symmetries?
Are you familiar with Noether’s Theorem? It comes up in some explanations of Buckingham pi, but the point is mostly “if you already know that something is symmetric, then something is conserved.”
The most similar thing I can think of, in terms of “resources for finding symmetries,” might be related to finding Lyapunov stability functions. It seems there’s not too much in the way of automated function-finding for arbitrary systems; I’ve seen at least one automated approach for systems with polynomial dynamics, though.
Not familiar with Noether’s theorem. Seems useful for constructing models, and perhaps determining if something else beyond mass, momentum, and energy is conserved. Is the converse true as well, i.e., does conservation imply that symmetries exist?
I’m also afraid I know nearly nothing about non-linear stability, so I’m not sure what you’re referring to, but it sounds interesting. I’ll have to read the Wikipedia page. I’d be interested if you know any other good resources for learning this.
Is the converse true as well, i.e., does conservation imply that symmetries exist?
I think this is what Lie groups are all about, but that’s a bit deeper in group theory than I’m comfortable speaking on.
I’d be interested if you know any other good resources for learning this.
I learned it the long way by taking classes, and don’t recall being particularly impressed by any textbooks. (I can lend you the ones I used.) I remember thinking that reading through Akella’s lecture notes was about as good as taking the course, and so if you have the time to devote to it you might be able to get those from him by asking nicely.
Conservation gives a local symmetry but there may not be a global symmetry.
For instance, you can imagine a physical system with no forces at all, so everything is conserved. But there are still some parameters that define the location of the particles. Then the physical system is locally very symmetric, but it may still have some symmetric global structure where the particles are constrained to lie on a surface of nontrivial topology.
Noether’s theorem has nothing to do with Buckingham’s theorem. Buckingham’s theorem is quite general (and vacuous), while Noether’s theorem is only about hamiltonian/lagrangian mechanics.
Added: Actually, Buckingham and Noether do have something in common: they both taught at Bryn Mawr.
Noether’s theorem has nothing to do with Buckingham’s theorem.
Both of them are relevant to the project of exploiting symmetry, and deal with solidifying a mostly understood situation. (You can’t apply Buckingham’s theorem unless you know all the relevant pieces.) The more practical piece that I had in mind is that someone eager to apply Noether’s theorem will need to look for symmetries; they may have found techniques for hunting for symmetries that will be useful in general. It might be worth looking into material that teaches it, not because it itself is directly useful, but because the community that knows it may know other useful things.
It’s a quite bit more general than Lagrangian mechanics. You can extend it to any functional that takes functions between two manifolds to complex numbers.
I’ve always been amazed at the power of dimensional analysis. To me the best example is the problem of calculating the period of an oscillating mass on a spring. The relevant values are the spring constant K (kg/s^2) and the mass M (kg), and the period T is in (s). The only way to combine K and M to obtain a value with dimensions of (s) is sqrt(M/K), and that’s the correct form of the actual answer—no calculus required!
Actually, there’s another parameter, the displacement. It turns out that the spring period does not depend on the displacement, but that’s a miracle that is special to springs. Instead, look at the pendulum. The same dimensional analysis gives the square root of the length divided by gravitational acceleration. That’s off by a dimensionless constant, 2π. Moreover, even that is only approximately correct. The real answer depends on the displacement in a complicated way.
This is a good point. At best you can figure out that period is proportional to (not equal to) sqrt(M/K) multiplied by some function of other parameters, say, one involving displacement and another characterizing the non-linearity (if K is just the initial slope, as I’ve seen done before). It’s a fortunate coincidence if the other parameters are unimportant. You can not determine based solely on dimensional analysis whether certain parameters are unimportant.
That’s because outside of physics (and possibly chemistry) there are enough constants running around that all quantities are effectively dimensionless. I’m having a hard time seeing a situation in say biology where I could propose dimensional analysis with a straight face, to say nothing of softer sciences.
As I said, dimensional analysis does not help with categorical variables. And when the number of dimensions is low and/or the number of variables is large, dimensional analysis can be useless. I think it’s a necessary component of any model builder’s toolbox, but not a tool you will use for every problem. Still, I would argue that it’s underutilized. When dimensional analysis is useful, it definitely should be used. (For example, despite its obvious applications in physics, I don’t think most physics undergrads learn the Buckingham pi theorem. It’s usually only taught to engineers learning fluid dynamics and heat transfer.)
Two very common dimensionless parameters are the ratio and fraction. Both certainly appear in biology. Also, the subject of allometry in biology is basically simple dimensional analysis.
I’ve seen dimensional analysis applied in other soft sciences as well, e.g., political science, psychology, and sociology are a few examples I am aware of. I can’t comment much on the utility of its application in these cases, but it’s such a simple technique that I think it’s worth trying whenever you have data with units.
Speaking more generally, the idea of simplification coming from applying transformations to data has broad applicability. Dimensional analysis is just one example of this.
One thing that most scientists in these soft scientists already have a good grasp on, but a lot of laypeople do not, is the idea of appropriately normalizing parameters. For instance dividing something by the mass of the body, or the population of a nation, to do comparisons between individuals/nations of different sizes.
People will often make bad comparisons where they don’t normalize properly. But hopefully most people reading this article are not at risk for that.
PCA and other dimensionality reduction techniques are great, but there’s another very useful technique that most people (even statisticians) are unaware of: dimensional analysis, and in particular, the Buckingham pi theorem. For some reason, this technique is used primarily by engineers in fluid dynamics and heat transfer despite its broad applicability. This is the technique that allows scale models like wind tunnels to work, but it’s more useful than just allowing for scaling. I find it very useful to reduce the number of variables when developing models and conducting experiments.
Dimensional analysis recognizes a few basic axioms about models with dimensions and sees what they imply. You can use these to construct new variables from the old variables. The model is usually complete in a smaller number of these new variables. The technique does not tell you which variables are “correct”, just how many independent ones are needed. Identifying “correct” variables requires data, domain knowledge, or both. (And sometimes, there’s no clear “best” variable; multiple work equivalently well.)
Dimensional analysis does not help with categorical variables, or numbers which are already dimensionless (though by luck, sometimes combinations of dimensionless variables are actually what’s “correct”). This is the main restriction that applies. And you can expect at best a reduction in the number of variables of about 3. Dimensional analysis is most useful for physical problems with maybe 3 to 10 variables.
The basic idea is this: Dimensions are some sort of metadata which can tell you something about the structure of the problem. You can always rewrite a dimensional equation, for example, to be dimensionless on both sides. You should notice that some terms become constants when this is done, and that simplifies the equation.
Here’s a physical example: Let’s say you want to measure the drag on a sphere (units: N). You know this depends on the air speed (units: m/s), viscosity (units: m^2/s), air density (units: kg/m^3), and the diameter of the sphere (units: m). So, you have 5 variables in total. Let’s say you want to do a factorial design with 4 levels in each variable, with no replications. You’d have to do 4^4 = 256 experiments. This is clearly too complicated.
What fluid dynamicists have recognized is that you can rewrite the relationship in terms of different variables, and nothing is missing. The Buckingham pi theorem mentioned previously says that we only need 2 dimensionless variables given our 5 dimensional variables. So, instead of the drag force, you use the drag coefficient, and instead of the speed, viscosity, etc., you use the Reynolds number. Now, you only need to do 4 experiments to get the same level of representation.
As it turns out, you can use techniques like PCA on top of dimensional analysis to determine that certain dimensionless parameters are unimportant (there are other ways too). This further simplifies models.
There’s a lot more on this topic than what I have covered and mentioned here. I would recommend reading the book Dimensional analysis and the theory of models for more details and the proof of the pi theorem.
(Another advantage of dimensional analysis: If you discover a useful dimensionless variable, you can get it named after yourself.)
In general, if your problem displays any kind of symmetry* you can exploit that to simplify things. I think most people are capable of doing this intuitively when the symmetry is obvious. The Buckingham pi theorem is a great example of a systematic way to find and exploit a symmetry that isn’t so obvious.
* By “symmetry” I really mean “invariance under a group of transformations”.
This is a great point. Other than fairly easy geometric and time symmetries, do you have any advice or know of any resources which might be helpful towards finding these symmetries?
Here’s what I do know: Sometimes you can recognize these symmetries by analyzing a model differential equation. Here’s a book on the subject that I haven’t read, but might read in the future. My PhD advisor tells me I already know one reliable way to find these symmetries (e.g., like how to find the change of variables used here), so reading this would be a poor use of time in his view. This approach also requires knowing a fair bit more about a phenomena than just which variables it depends on.
The book you linked is the sort of thing I had in mind. The historical motivation for Lie groups was to develop a systematic way to use symmetry to attack differential equations.
Are you familiar with Noether’s Theorem? It comes up in some explanations of Buckingham pi, but the point is mostly “if you already know that something is symmetric, then something is conserved.”
The most similar thing I can think of, in terms of “resources for finding symmetries,” might be related to finding Lyapunov stability functions. It seems there’s not too much in the way of automated function-finding for arbitrary systems; I’ve seen at least one automated approach for systems with polynomial dynamics, though.
Not familiar with Noether’s theorem. Seems useful for constructing models, and perhaps determining if something else beyond mass, momentum, and energy is conserved. Is the converse true as well, i.e., does conservation imply that symmetries exist?
I’m also afraid I know nearly nothing about non-linear stability, so I’m not sure what you’re referring to, but it sounds interesting. I’ll have to read the Wikipedia page. I’d be interested if you know any other good resources for learning this.
I think this is what Lie groups are all about, but that’s a bit deeper in group theory than I’m comfortable speaking on.
I learned it the long way by taking classes, and don’t recall being particularly impressed by any textbooks. (I can lend you the ones I used.) I remember thinking that reading through Akella’s lecture notes was about as good as taking the course, and so if you have the time to devote to it you might be able to get those from him by asking nicely.
Conservation gives a local symmetry but there may not be a global symmetry.
For instance, you can imagine a physical system with no forces at all, so everything is conserved. But there are still some parameters that define the location of the particles. Then the physical system is locally very symmetric, but it may still have some symmetric global structure where the particles are constrained to lie on a surface of nontrivial topology.
Noether’s theorem has nothing to do with Buckingham’s theorem. Buckingham’s theorem is quite general (and vacuous), while Noether’s theorem is only about hamiltonian/lagrangian mechanics.
Added: Actually, Buckingham and Noether do have something in common: they both taught at Bryn Mawr.
Both of them are relevant to the project of exploiting symmetry, and deal with solidifying a mostly understood situation. (You can’t apply Buckingham’s theorem unless you know all the relevant pieces.) The more practical piece that I had in mind is that someone eager to apply Noether’s theorem will need to look for symmetries; they may have found techniques for hunting for symmetries that will be useful in general. It might be worth looking into material that teaches it, not because it itself is directly useful, but because the community that knows it may know other useful things.
It’s a quite bit more general than Lagrangian mechanics. You can extend it to any functional that takes functions between two manifolds to complex numbers.
In what sense do you mean Buckingham’s theorem is vacuous?
I’ve always been amazed at the power of dimensional analysis. To me the best example is the problem of calculating the period of an oscillating mass on a spring. The relevant values are the spring constant K (kg/s^2) and the mass M (kg), and the period T is in (s). The only way to combine K and M to obtain a value with dimensions of (s) is sqrt(M/K), and that’s the correct form of the actual answer—no calculus required!
Actually, there’s another parameter, the displacement. It turns out that the spring period does not depend on the displacement, but that’s a miracle that is special to springs. Instead, look at the pendulum. The same dimensional analysis gives the square root of the length divided by gravitational acceleration. That’s off by a dimensionless constant, 2π. Moreover, even that is only approximately correct. The real answer depends on the displacement in a complicated way.
This is a good point. At best you can figure out that period is proportional to (not equal to) sqrt(M/K) multiplied by some function of other parameters, say, one involving displacement and another characterizing the non-linearity (if K is just the initial slope, as I’ve seen done before). It’s a fortunate coincidence if the other parameters are unimportant. You can not determine based solely on dimensional analysis whether certain parameters are unimportant.
That’s because outside of physics (and possibly chemistry) there are enough constants running around that all quantities are effectively dimensionless. I’m having a hard time seeing a situation in say biology where I could propose dimensional analysis with a straight face, to say nothing of softer sciences.
As I said, dimensional analysis does not help with categorical variables. And when the number of dimensions is low and/or the number of variables is large, dimensional analysis can be useless. I think it’s a necessary component of any model builder’s toolbox, but not a tool you will use for every problem. Still, I would argue that it’s underutilized. When dimensional analysis is useful, it definitely should be used. (For example, despite its obvious applications in physics, I don’t think most physics undergrads learn the Buckingham pi theorem. It’s usually only taught to engineers learning fluid dynamics and heat transfer.)
Two very common dimensionless parameters are the ratio and fraction. Both certainly appear in biology. Also, the subject of allometry in biology is basically simple dimensional analysis.
I’ve seen dimensional analysis applied in other soft sciences as well, e.g., political science, psychology, and sociology are a few examples I am aware of. I can’t comment much on the utility of its application in these cases, but it’s such a simple technique that I think it’s worth trying whenever you have data with units.
Speaking more generally, the idea of simplification coming from applying transformations to data has broad applicability. Dimensional analysis is just one example of this.
One thing that most scientists in these soft scientists already have a good grasp on, but a lot of laypeople do not, is the idea of appropriately normalizing parameters. For instance dividing something by the mass of the body, or the population of a nation, to do comparisons between individuals/nations of different sizes.
People will often make bad comparisons where they don’t normalize properly. But hopefully most people reading this article are not at risk for that.