The causal structure is basically a chaotic system, which means that NewtonIan style differential equations aren’t much use, and big computerized models are. Ordinary weather forecasting uses big models, and I don’t see why climate change, which is essentially very long term forecasting would different.
The causal structure is basically a chaotic system, which means that NewtonIan style differential equations aren’t much use, and big computerized models are. Ordinary weather forecasting uses big models, and I don’t see why climate change, which is essentially very long term forecasting would different.
Climatological models and meteorological models are very different. If they weren’t, then “we can’t predict whether it will rain or not ten days from now” (which is mostly true) would be a slam-dunk argument against our ability to predict temperatures ten years from now. One underlying technical issue is that floating point arithmetic is only so precise, and this gives you an upper bound on the amount of precision you can expect from your simulation given the number of steps you run the model for. Thus climatological models have larger cells, larger step times, and so on, so that you can run the model for 50 model-years and still think the result that comes out might be reasonable.
(I also don’t think it’s right to say that Newtonian-style diffeqs aren’t much use; the underlying update rules for the cells are diffeqs like that.)
I’m not sure if I’m understanding you correctly, but the reason why climate forecasts and meterological forecasts have different temporal ranges of validity is not that the climate models are coarser, it’s that they’re asking different questions.
Climate is (roughly speaking) the attractor on which the weather chaotically meanders on short (e.g. weekly) timescales. On much longer (1-100+ years) this attractor itself shifts. Weather forecasts want to determine the future state of the system itself as it evolves chaotically, which is impossible in principle after ~14 days because the system is chaotic. Climate forecasts want to track the slow shifts of the attractor. To do this, they run ensembles with slightly different initial conditions and observe the statistics of the ensemble at some future date, which is taken (via an ergodic assumption) to reflect the attractor at that date. None of the ensemble members are useful as “weather predictions” for 2050 or whatever, but their overall statistics are (it is argued) reliable predictions about the attractor on which the weather will be constrained to move in 2050 (i.e. “the climate in 2050″).
It’s analogous to the way we can precisely characterize the attractor in the Lorenz system, even if we can’t predict the future of any given trajectory in that system because it’s chaotic. (For a more precise analogy, imagine a version of the Lorenz system in which the attractor slowly changes over long time scales)
A simple way to explain the difference is that you have no idea what the weather will be in any particular place on June 19, 2016, but you can be pretty sure that in the Northern Hemisphere it will be summer in June 2016. This has nothing to do with differences in numerical model properties (you aren’t running a numerical model in your head), it’s just a consequence of the fact that climate and weather are two different things.
Apologies if you know all this. It just wasn’t clear to me if you did from your comment, and I thought I might spell it out since it might be valuable to someone reading the thread.
Apologies if you know all this. It just wasn’t clear to me if you did from your comment, and I thought I might spell it out since it might be valuable to someone reading the thread.
I did know this, but thanks for spelling it out! One of the troubles with making short comments on this is that it doesn’t work, and adding detail can be problematic if you add details in the wrong order. Your description is much better at getting the order of details right than my description has been.
I will point out also that my non-expert understanding is that some suspect that the attractor dynamics are themselves chaotic, because it looks like it’s determined by a huge number of positive and negative feedback loops whose strength is dependent on the state of the system in possibly non-obvious ways. My impression is that informed people are optimistic or pessimistic about climate change based on whether the feedback loops that they think about are on net positive or negative. (As extremes, consider people who reason by analogy from Venus representing the positive feedback loop view and people who think geoengineering will be sufficient to avoid disaster representing the negative feedback loop view.)
There are a number of different mechanisms which can trigger bifurcations. Finite precision is one of them. Another is that the measurements used to initialize the simulation have much more limited precision and accuracy, and that they do not sample the entire globe (so further approximations must be made to fill in the gaps). There also are numerical errors from the approximations used in converting differential equations to algebraic equations and algebraic errors whenever approximations to the solution of a large linear algebraic system are made. Etc. Any these can trigger bifurcations and make prediction of a certain realization (say, what happens in reality) impossible beyond a certain time.
The good news is that none of these models try to solve for a particular realization. Usually they try to solve for the ensemble mean or some other statistic. Basically, let’s say you have a collection of nominally equivalent initial conditions for the system*. Let’s say you evolve these fields in time, and average the results overall realizations at each time. That’s your ensemble average. If you decompose the fields to be solved into an ensemble mean and a fluctuation, you can then apply an averaging operator and get differential equations which are better behaved (in terms of resolution requirements; I assume they are less chaotic as well), but have unclosed terms which require models. This is turbulence modeling. (To be absolutely clear, what I’ve written is somewhat inaccurate, as from what I understand most climate and weather models use large eddy simulation which is a spatial filtering rather than ensemble averaging. You can ignore this for now.)
One could argue that the ensemble mean is more useful in some areas than others. Certainly, if you just want to calculate drag on a wing (a time-averaged quantity), the ensemble mean is great in that it allows you to jump directly to that. But if you want something which varies in time (as climate and weather models do) then you might not expect this approach to work so well. (But what else can you do?)
nostalgebraist is right, but a fair bit abstract. I never really liked the language of attractors when speaking about fluid dynamics. (Because you can’t visualize what the “attractor” is for a vector field so easily.) A much easier way to understand what he is saying is that there are multiple time scales, say, a slow and a fast one. Hopefully it’s not necessary to accurately predict or model the fast one (weather) to accurately predict the slow one (climate). You can make similar statements about spatial scales. This is not always true, but there are reasons to believe it is true in many circumstances in fluid dynamics.
In terms of accumulation of numerical error causing the problems, I don’t think that’s quite right. I think it’s more right to say that uncertainty grows in time due to both accumulation of numerical error and chaos, but It’s not clear to me which is more significant. This is assuming that climate models use some sort of turbulence model, which they do. It’s also assuming that an appropriate numerical method was used. For example, in combustion simulations, if you use a numerical method which has considerable dispersion errors, the entire result can go to garbage very quickly if this type of error causes the temperature to unphysically rise above the ignition temperature. Then you have flame propagation, etc., which might not happen if a better method was used.
* I have asked specifically about what this means from a technical standpoint, and am yet to get a satisfactory reply. My thinking is that the initial condition is the set of all possible initial conditions given the probability distribution of all the measurements. I have seen some weather models use what looks like Monte Carlo sampling to get average storm trajectories, for example, so someone must have formalized this.
One underlying technical issue is that floating point arithmetic is only so precise, and this gives you an upper bound on the amount of precision you can expect from your simulation given the number of steps you run the model for.
I don’t believe that in reality the precision of floats is a meaningful limit on the accuracy of climate forecasts. I would probably say that people who think so drastically underestimate the amount of uncertainty they have in their simulation.
I don’t believe that in reality the precision of floats is a meaningful limit on the accuracy of climate forecasts.
How much experience do you have with scientific computation?
I would probably say that people who think so drastically underestimate the amount of uncertainty they have in their simulation.
Disagreed. The more uncertainty you incorporate into your model (i.e., tracking distributions over temperatures in cells instead of tracking point estimates of temperatures in cells), the more arithmetic you need to do, and thus the sooner calculation noise raises its ugly head.
How much experience do you have with scientific computation?
Enough to worry about the precision of floats when inverting certain matrices, for example.
The more uncertainty you incorporate into your model (i.e., tracking distributions over temperatures in cells instead of tracking point estimates of temperatures in cells), the more arithmetic you need to do, and thus the sooner calculation noise raises its ugly head.
We continue to disagree :-) Doing arithmetic is not a problem (if your values are scaled properly and that’s an easy thing to do). What you probably mean is that if you run a very large number of cycles feeding the output of the previous into the next, your calculation noise accumulates and starts to cause problems. I would suggest that as your calculation noise accumulates, so do does the uncertainty you have about the starting values (and your model uncertainty accumulates with cycling, too), and by the time you start to care about the precision of floats, all the rest of the accumulated uncertainty makes the output garbage anyway.
Things are somewhat different in hard physics where the uncertainty can get very very very small, but climate science is not that.
To return to my original point, the numerical precision limits due to floating-point arithmetic was an illustrative example that upper bounds the fidelity of climate models. Climate isn’t my field (but numerical methods, broadly speaking, is), and so I expect my impressions to often be half-formed and/or out of date. While I’ve read discussions and papers about the impact of numerical precision on the reproducibility and fidelity of climate models, I don’t have those archived anywhere I can find them easily (and even if I did remember where to find them, there would be ‘beware the man of one study’ concerns).
I called it an upper bound specifically to avoid the claim that it’s the binding constraint on climate modeling; my impression is that cells are the volume they are because of the computational costs (in both time and energy) involved. So why focus on a constraint that’s not material? Because it might be easier to explain or understand, and knowing that there is an upper bound, and that it’s low enough that it might be relevant, can be enough to guide action.
As an example of that sort of reasoning, I’m thinking here of the various semiconductor people who predicted that CPUs would stop getting faster because of speed of light and chip size concerns—that turned out to not be the constraint that actually killed increasing CPU speed (energy consumption / heat dissipation was), but someone planning around that constraint would have had a much better time than someone who wasn’t. (Among other things, it helps you predict that parallel processing will become increasingly critical once speed gains can no longer be attained by doing things serially faster.)
I would suggest that as your calculation noise accumulates, so do does the uncertainty you have about the starting values, and by the time you start to care about the precision of floats, all the rest of the accumulated uncertainty makes the output garbage anyway.
I don’t agree, but my views may be idiosyncratic. There’s a research area called “uncertainty propagation,” which deals with the challenge of creating good posterior distributions over model outputs given model inputs. I might have some distribution over the parameters of my model, some distribution over the boundary conditions of my model (i.e. the present measurements of climatological data, etc.), and want to somehow push both of those uncertainties through my model to get an uncertainty over outputs at the end that takes everything into account.
If the model calculation process is deterministic (i.e. the outputs of the model can be an object that describes some stochastic phenomenon, like a wavefunction, but which wavefunction the model outputs can’t be stochastic), then this problem has at least one conceptually straightforward solution (sample from the input distribution, run the model, generate an empirical output distribution) and a number of more sophisticated solutions. If the model calculation is “smooth,” the final posterior becomes even easier to calculate; there are situations where you can just push Gaussian distributions on inputs through your model and get Gaussian distributions on your outputs.
Calculation noise seems separate from parameter input uncertainty to me because it enters into this process separately. I can come up with some sampling lattice over my model parameter possibilities, but it may be significantly more difficult to come up with some sampling lattice over the calculation noise in the same way. (Yes, I can roll everything together into “noise,” and when it comes to actually making a decision that’s how this shakes out, but from computational theory point of view there seems to be value in separating the two.)
In particular, climate as a chaotic system is not “smooth.” The famous Lorenz quote is relevant:
Chaos: When the present determines the future, but the approximate present does not approximately determine the future.
When we only have the approximate present, we can see how various possibilities would propagate forward and get a distribution over what the future would look like. But with calculation noise and the underlying topological mixing in the structure, we no longer have the guarantee that the present determines the future! (That is, we are not guaranteed that “our model” will generate the same outputs given the same inputs, as its behavior may be determined by low-level implementation details.)
the numerical precision limits due to floating-point arithmetic was an illustrative example that upper bounds the fidelity of climate models
Yes, this is technically correct but I struggle to find this meaningful. Any kind of model or even of a calculation which uses real numbers (and therefore floating-point values) is subject to the same upper bounds.
knowing that there is an upper bound, and that it’s low enough that it might be relevant, can be enough to guide action.
Well, of course there is an upper bound. What I contest is that the bound imposed by the floating-point precision is relevant here. I am also not sure what kind of guide do you expect it to be.
this problem has at least one conceptually straightforward solution
In reality things are considerably more complicated. First, you assume that you can arbitrarily reduce the input uncertainty by sufficient sampling from the input distribution. The problem is that you don’t know the true input distribution. Instead you have an estimate which itself is a model and as such is different from the underlying reality. Repeated sampling from this estimated distribution can get you arbitrarily close to your estimate, but it won’t get you arbitrarily close to the underlying true values because you don’t know what they are.
Second, there are many sources of uncertainty. Let me list some.
The process stability. When you model some process you typically assume that certain characteristcs of it are stable, that is, they do not change over either your fit period or your forecasting period. That is not necessarily true but is a necessary assumption to build a reasonable model.
The sample. Normally you don’t have exhaustive data over the lifetime of the process you’re trying to model. You have a sample and then you estimate things (like distributions) from the sample that you have. The estimates are, of course, subject to some error.
The model uncertainty. All models are wrong in that they are not a 1:1 match to reality. The goal of modeling is to make the “wrongness” of the model acceptably low, but it will never go away completely. This is actually a biggie when you cycle your model—the model error accumulates at each iteration.
Black swan events. The fact something didn’t occur in the history visible to you is not a guarantee that it won’t occur in the future—but your ability to model the impact of such an event is very limited.
Calculation noise seems separate from parameter input uncertainty to me because it enters into this process separately.
This is true. My contention is in most modeling (climate models, certainly) other sources of noise completely dominate over the calculation noise.
we no longer have the guarantee that the present determines the future
You don’t have such a guarantee to start with. Specifically, there is no guarantee whatsoever that your model if run with infinite-precision calculations will adquately represent the future.
This is true. My contention is in most modeling (climate models, certainly) other sources of noise completely dominate over the calculation noise.
The more I think about this, the less sure I am about how true this is. I was initially thinking that the input and model uncertainties are very large. But I think Vaniver is right and this depends on the particulars of the implementation. The differences between different simulation codes for nominally identical inputs can be surprising. Both are large. (I am thinking in particular about fluid dynamics here, but it’s basically the same equations as in weather and climate modeling, so I assume my conclusions carry over as well.)
One weird idea that comes from this: You could use an approach like MILES in fluid dynamics where you treat the numerical error as a model, which could reduce uncertainty. This only makes sense in turbulence modeling and would take more time than I have to explain.
I was initially thinking that the input and model uncertainties are very large. But I think Vaniver is right and this depends on the particulars of the implementation.
I am not a climatologist, but I have a hard time imagining how the input and model uncertainties in a climate model can be driven down to the magnitudes where floating-point precision starts to matter.
If I’m reading Vaniver correctly (or possibly I’m steelmanning his argument without realizing it), he’s using round-off error (as it’s called in scientific computing) as an example of one of several numerical errors, e.g., discretization and truncation. There are further subcategories like dispersion and dissipation (the latter is the sort of “model” MILES provides for turbulent dissipation). I don’t think round-off error usually is the dominant factor, but the other numerical errors can be, and this might often be the case in fluid flow simulations on more modest hardware.
Round-off error can accumulate to dominate the numerical error if you do things wrong. See figure 38.5 for a representative illustration of the total numerical error as a function of time step. If the time step becomes very small, total numerical error actually increases due to build-up of round-off error. As I said, this only happens if you do things wrong, but it can happen.
Yes, I understand all that, but this isn’t the issue. The issue is how much all the assorted calculation errors matter in comparison to the rest of the uncertainty in the model.
I don’t think we disagree too much. If I had to pick one, I’d agree with you that the rest of the uncertainty is likely larger in most cases, but I think you substantially underestimate how inaccurate these numerical methods can be. Many commercial computational fluid dynamics codes use quite bad numerical methods along with large grid cells and time steps, so it seems possible to me that those errors can exceed the uncertainties in the other parameters. I can think of one case in particular in my own work where the numerical errors likely exceed the other uncertainties.
Even single-precision floating point gives you around 7 decimal digits of accuracy. If (as is the case for both weather and climate modelling) the inputs are not known with anything like that amount of precision, surely input uncertainty will overwhelm calculation noise? Calculation noise enters at every step, of course, but even so, there must be diminishing returns from increased precision.
See the second half of this cousin comment. But a short summary (with a bit of additional info):
First, I see a philosophical difference between input uncertainty and calculation noise; the mathematical tools you need to attack each problem are different. The first can be solved through sampling (or a number of other different ways); the second can be solved with increased precision (or a number of other different ways). Importantly, sampling does not seem to me to be a promising approach to solving the calculation noise problem, because the errors may be systematic instead of random. In chaotic systems, this problem seems especially important.
Second, it seems common for both weather models and climate models to use simulation time steps of about 10 minutes. If you want to predict 6 days ahead, that’s 864 time steps. If you want to predict 60 years ahead, that’s over 3 million time steps.
(i.e., tracking distributions over temperatures in cells instead of tracking point estimates of temperatures in cells)
Many combustion modeling approaches do precisely this. Look into prescribed PDF methods, for example. You can see the necessity of this by recognizing that ignition can occur if the temperature anywhere in a cell is above the ignition temperature.
(There is also the slightly confusing issue that these distributions are not the same thing as the distribution of possible realizations.)
The differences between climate and meteorological models are reasons that should only increase someone’s confidence in the relative capabilities of climate science, so the analogy seems apt despite these differences.
I am not sure what you mean by “causal structure” in this context. I was attempting to provide some intuition as to why ordinary weather forecasting and climate change modeling would be different, since you stated that you didn’t see what the essential difference between them is.
But it was a short comment, and so many things were only left as implications. For example, the cell update laws (i.e. the differential equations guiding the system) will naturally be different for weather forecasting and climate forecasting because the cells are physically different beasts. You’ll model cloud dynamics very differently depending on whether or not clouds are bigger or smaller than a model cell, and it’s not necessarily the case that a fine-grained model will be more accurate than a coarse-grained model, for many reasons.
Understanding causal structure seems to be something that is kind of shiny and impressive sounding, connotationally, but doesn’t mean much, at least not much that is new, denotationally. And it comes up because I thought was replying to DVH, who brought it up.
I don’t think CC modelling and weather forecasting are all that essentially different, or at least not as different as Causal Structure is supposed to be from either.
The pattern “the experts in X are actually incompetent fools, because they are not doing Y” is frequent in LessWrong Classic, even if it hasn’t been applied to climate change before.
Model bias is not a joke. If your model is severely biased, it is giving you garbage. I am not sure in what sense a model that outputs garbage is better than no model at all. The former just gives you a false sense of confidence, because math was used.
If you think there are [reasons] where the model isn’t completely garbage, or we can put bounds on garbage, or something, then that is a useful conversation to have.
If you set up the conversation where it’s the garbage model or no science at all, then you are engaged in rhetoric, not science.
I don’t suppose public policy is based on a single model.
If you read back, nothing has been said about any specific model, so no such claim needs defending.
If you read back, it has been suggested that there is a much better way of doing climate .science than modelling of any kind....but details are lacking.
Understanding causal structure seems to be something that is kind of shiny and impressive sounding,
connotationally, but doesn’t mean much
No, it means a whole lot. You need to get the causal structure right, or at least reasonably close, or your model is garbage for policy. See also: “irrational policy of managing the news.”
I fight this fight, along with my colleagues, in much simpler settings than weather. And it is still difficult.
The whole “normative sociology” concept has its origins in a joke that Robert Nozick made, in Anarchy, State and Utopia, where he claimed, in an offhand way, that “Normative sociology, the study of what the causes of problems ought to be, greatly fascinates us all”(247). Despite the casual manner in which he made the remark, the observation is an astute one. Often when we study social problems, there is an almost irresistible temptation to study what we would like the cause of those problems to be (for whatever reason), to the neglect of the actual causes. When this goes uncorrected, you can get the phenomenon of “politically correct” explanations for various social problems – where there’s no hard evidence that A actually causes B, but where people, for one reason or another, think that A ought to be the explanation for B. This can lead to a situation in which denying that A is the cause of B becomes morally stigmatized, and so people affirm the connection primarily because they feel obliged to, not because they’ve been persuaded by any evidence.
So what’s the right way to predict the future?
What exactly do you mean by that? Because the obvious answer is to figure out the causal structure of things, but I don’t think that helps here.
The causal structure is basically a chaotic system, which means that NewtonIan style differential equations aren’t much use, and big computerized models are. Ordinary weather forecasting uses big models, and I don’t see why climate change, which is essentially very long term forecasting would different.
Climatological models and meteorological models are very different. If they weren’t, then “we can’t predict whether it will rain or not ten days from now” (which is mostly true) would be a slam-dunk argument against our ability to predict temperatures ten years from now. One underlying technical issue is that floating point arithmetic is only so precise, and this gives you an upper bound on the amount of precision you can expect from your simulation given the number of steps you run the model for. Thus climatological models have larger cells, larger step times, and so on, so that you can run the model for 50 model-years and still think the result that comes out might be reasonable.
(I also don’t think it’s right to say that Newtonian-style diffeqs aren’t much use; the underlying update rules for the cells are diffeqs like that.)
I’m not sure if I’m understanding you correctly, but the reason why climate forecasts and meterological forecasts have different temporal ranges of validity is not that the climate models are coarser, it’s that they’re asking different questions.
Climate is (roughly speaking) the attractor on which the weather chaotically meanders on short (e.g. weekly) timescales. On much longer (1-100+ years) this attractor itself shifts. Weather forecasts want to determine the future state of the system itself as it evolves chaotically, which is impossible in principle after ~14 days because the system is chaotic. Climate forecasts want to track the slow shifts of the attractor. To do this, they run ensembles with slightly different initial conditions and observe the statistics of the ensemble at some future date, which is taken (via an ergodic assumption) to reflect the attractor at that date. None of the ensemble members are useful as “weather predictions” for 2050 or whatever, but their overall statistics are (it is argued) reliable predictions about the attractor on which the weather will be constrained to move in 2050 (i.e. “the climate in 2050″).
It’s analogous to the way we can precisely characterize the attractor in the Lorenz system, even if we can’t predict the future of any given trajectory in that system because it’s chaotic. (For a more precise analogy, imagine a version of the Lorenz system in which the attractor slowly changes over long time scales)
A simple way to explain the difference is that you have no idea what the weather will be in any particular place on June 19, 2016, but you can be pretty sure that in the Northern Hemisphere it will be summer in June 2016. This has nothing to do with differences in numerical model properties (you aren’t running a numerical model in your head), it’s just a consequence of the fact that climate and weather are two different things.
Apologies if you know all this. It just wasn’t clear to me if you did from your comment, and I thought I might spell it out since it might be valuable to someone reading the thread.
I did know this, but thanks for spelling it out! One of the troubles with making short comments on this is that it doesn’t work, and adding detail can be problematic if you add details in the wrong order. Your description is much better at getting the order of details right than my description has been.
I will point out also that my non-expert understanding is that some suspect that the attractor dynamics are themselves chaotic, because it looks like it’s determined by a huge number of positive and negative feedback loops whose strength is dependent on the state of the system in possibly non-obvious ways. My impression is that informed people are optimistic or pessimistic about climate change based on whether the feedback loops that they think about are on net positive or negative. (As extremes, consider people who reason by analogy from Venus representing the positive feedback loop view and people who think geoengineering will be sufficient to avoid disaster representing the negative feedback loop view.)
There are a number of different mechanisms which can trigger bifurcations. Finite precision is one of them. Another is that the measurements used to initialize the simulation have much more limited precision and accuracy, and that they do not sample the entire globe (so further approximations must be made to fill in the gaps). There also are numerical errors from the approximations used in converting differential equations to algebraic equations and algebraic errors whenever approximations to the solution of a large linear algebraic system are made. Etc. Any these can trigger bifurcations and make prediction of a certain realization (say, what happens in reality) impossible beyond a certain time.
The good news is that none of these models try to solve for a particular realization. Usually they try to solve for the ensemble mean or some other statistic. Basically, let’s say you have a collection of nominally equivalent initial conditions for the system*. Let’s say you evolve these fields in time, and average the results overall realizations at each time. That’s your ensemble average. If you decompose the fields to be solved into an ensemble mean and a fluctuation, you can then apply an averaging operator and get differential equations which are better behaved (in terms of resolution requirements; I assume they are less chaotic as well), but have unclosed terms which require models. This is turbulence modeling. (To be absolutely clear, what I’ve written is somewhat inaccurate, as from what I understand most climate and weather models use large eddy simulation which is a spatial filtering rather than ensemble averaging. You can ignore this for now.)
One could argue that the ensemble mean is more useful in some areas than others. Certainly, if you just want to calculate drag on a wing (a time-averaged quantity), the ensemble mean is great in that it allows you to jump directly to that. But if you want something which varies in time (as climate and weather models do) then you might not expect this approach to work so well. (But what else can you do?)
nostalgebraist is right, but a fair bit abstract. I never really liked the language of attractors when speaking about fluid dynamics. (Because you can’t visualize what the “attractor” is for a vector field so easily.) A much easier way to understand what he is saying is that there are multiple time scales, say, a slow and a fast one. Hopefully it’s not necessary to accurately predict or model the fast one (weather) to accurately predict the slow one (climate). You can make similar statements about spatial scales. This is not always true, but there are reasons to believe it is true in many circumstances in fluid dynamics.
In terms of accumulation of numerical error causing the problems, I don’t think that’s quite right. I think it’s more right to say that uncertainty grows in time due to both accumulation of numerical error and chaos, but It’s not clear to me which is more significant. This is assuming that climate models use some sort of turbulence model, which they do. It’s also assuming that an appropriate numerical method was used. For example, in combustion simulations, if you use a numerical method which has considerable dispersion errors, the entire result can go to garbage very quickly if this type of error causes the temperature to unphysically rise above the ignition temperature. Then you have flame propagation, etc., which might not happen if a better method was used.
* I have asked specifically about what this means from a technical standpoint, and am yet to get a satisfactory reply. My thinking is that the initial condition is the set of all possible initial conditions given the probability distribution of all the measurements. I have seen some weather models use what looks like Monte Carlo sampling to get average storm trajectories, for example, so someone must have formalized this.
I don’t believe that in reality the precision of floats is a meaningful limit on the accuracy of climate forecasts. I would probably say that people who think so drastically underestimate the amount of uncertainty they have in their simulation.
Yeah, you can get arbitrary precision libraries.
You can, and what you’ll discover is that they are abysmally slow.
How much experience do you have with scientific computation?
Disagreed. The more uncertainty you incorporate into your model (i.e., tracking distributions over temperatures in cells instead of tracking point estimates of temperatures in cells), the more arithmetic you need to do, and thus the sooner calculation noise raises its ugly head.
Enough to worry about the precision of floats when inverting certain matrices, for example.
We continue to disagree :-) Doing arithmetic is not a problem (if your values are scaled properly and that’s an easy thing to do). What you probably mean is that if you run a very large number of cycles feeding the output of the previous into the next, your calculation noise accumulates and starts to cause problems. I would suggest that as your calculation noise accumulates, so do does the uncertainty you have about the starting values (and your model uncertainty accumulates with cycling, too), and by the time you start to care about the precision of floats, all the rest of the accumulated uncertainty makes the output garbage anyway.
Things are somewhat different in hard physics where the uncertainty can get very very very small, but climate science is not that.
To return to my original point, the numerical precision limits due to floating-point arithmetic was an illustrative example that upper bounds the fidelity of climate models. Climate isn’t my field (but numerical methods, broadly speaking, is), and so I expect my impressions to often be half-formed and/or out of date. While I’ve read discussions and papers about the impact of numerical precision on the reproducibility and fidelity of climate models, I don’t have those archived anywhere I can find them easily (and even if I did remember where to find them, there would be ‘beware the man of one study’ concerns).
I called it an upper bound specifically to avoid the claim that it’s the binding constraint on climate modeling; my impression is that cells are the volume they are because of the computational costs (in both time and energy) involved. So why focus on a constraint that’s not material? Because it might be easier to explain or understand, and knowing that there is an upper bound, and that it’s low enough that it might be relevant, can be enough to guide action.
As an example of that sort of reasoning, I’m thinking here of the various semiconductor people who predicted that CPUs would stop getting faster because of speed of light and chip size concerns—that turned out to not be the constraint that actually killed increasing CPU speed (energy consumption / heat dissipation was), but someone planning around that constraint would have had a much better time than someone who wasn’t. (Among other things, it helps you predict that parallel processing will become increasingly critical once speed gains can no longer be attained by doing things serially faster.)
I don’t agree, but my views may be idiosyncratic. There’s a research area called “uncertainty propagation,” which deals with the challenge of creating good posterior distributions over model outputs given model inputs. I might have some distribution over the parameters of my model, some distribution over the boundary conditions of my model (i.e. the present measurements of climatological data, etc.), and want to somehow push both of those uncertainties through my model to get an uncertainty over outputs at the end that takes everything into account.
If the model calculation process is deterministic (i.e. the outputs of the model can be an object that describes some stochastic phenomenon, like a wavefunction, but which wavefunction the model outputs can’t be stochastic), then this problem has at least one conceptually straightforward solution (sample from the input distribution, run the model, generate an empirical output distribution) and a number of more sophisticated solutions. If the model calculation is “smooth,” the final posterior becomes even easier to calculate; there are situations where you can just push Gaussian distributions on inputs through your model and get Gaussian distributions on your outputs.
Calculation noise seems separate from parameter input uncertainty to me because it enters into this process separately. I can come up with some sampling lattice over my model parameter possibilities, but it may be significantly more difficult to come up with some sampling lattice over the calculation noise in the same way. (Yes, I can roll everything together into “noise,” and when it comes to actually making a decision that’s how this shakes out, but from computational theory point of view there seems to be value in separating the two.)
In particular, climate as a chaotic system is not “smooth.” The famous Lorenz quote is relevant:
When we only have the approximate present, we can see how various possibilities would propagate forward and get a distribution over what the future would look like. But with calculation noise and the underlying topological mixing in the structure, we no longer have the guarantee that the present determines the future! (That is, we are not guaranteed that “our model” will generate the same outputs given the same inputs, as its behavior may be determined by low-level implementation details.)
Yes, this is technically correct but I struggle to find this meaningful. Any kind of model or even of a calculation which uses real numbers (and therefore floating-point values) is subject to the same upper bounds.
Well, of course there is an upper bound. What I contest is that the bound imposed by the floating-point precision is relevant here. I am also not sure what kind of guide do you expect it to be.
In reality things are considerably more complicated. First, you assume that you can arbitrarily reduce the input uncertainty by sufficient sampling from the input distribution. The problem is that you don’t know the true input distribution. Instead you have an estimate which itself is a model and as such is different from the underlying reality. Repeated sampling from this estimated distribution can get you arbitrarily close to your estimate, but it won’t get you arbitrarily close to the underlying true values because you don’t know what they are.
Second, there are many sources of uncertainty. Let me list some.
The process stability. When you model some process you typically assume that certain characteristcs of it are stable, that is, they do not change over either your fit period or your forecasting period. That is not necessarily true but is a necessary assumption to build a reasonable model.
The sample. Normally you don’t have exhaustive data over the lifetime of the process you’re trying to model. You have a sample and then you estimate things (like distributions) from the sample that you have. The estimates are, of course, subject to some error.
The model uncertainty. All models are wrong in that they are not a 1:1 match to reality. The goal of modeling is to make the “wrongness” of the model acceptably low, but it will never go away completely. This is actually a biggie when you cycle your model—the model error accumulates at each iteration.
Black swan events. The fact something didn’t occur in the history visible to you is not a guarantee that it won’t occur in the future—but your ability to model the impact of such an event is very limited.
This is true. My contention is in most modeling (climate models, certainly) other sources of noise completely dominate over the calculation noise.
You don’t have such a guarantee to start with. Specifically, there is no guarantee whatsoever that your model if run with infinite-precision calculations will adquately represent the future.
The more I think about this, the less sure I am about how true this is. I was initially thinking that the input and model uncertainties are very large. But I think Vaniver is right and this depends on the particulars of the implementation. The differences between different simulation codes for nominally identical inputs can be surprising. Both are large. (I am thinking in particular about fluid dynamics here, but it’s basically the same equations as in weather and climate modeling, so I assume my conclusions carry over as well.)
One weird idea that comes from this: You could use an approach like MILES in fluid dynamics where you treat the numerical error as a model, which could reduce uncertainty. This only makes sense in turbulence modeling and would take more time than I have to explain.
I am not a climatologist, but I have a hard time imagining how the input and model uncertainties in a climate model can be driven down to the magnitudes where floating-point precision starts to matter.
If I’m reading Vaniver correctly (or possibly I’m steelmanning his argument without realizing it), he’s using round-off error (as it’s called in scientific computing) as an example of one of several numerical errors, e.g., discretization and truncation. There are further subcategories like dispersion and dissipation (the latter is the sort of “model” MILES provides for turbulent dissipation). I don’t think round-off error usually is the dominant factor, but the other numerical errors can be, and this might often be the case in fluid flow simulations on more modest hardware.
Round-off error can accumulate to dominate the numerical error if you do things wrong. See figure 38.5 for a representative illustration of the total numerical error as a function of time step. If the time step becomes very small, total numerical error actually increases due to build-up of round-off error. As I said, this only happens if you do things wrong, but it can happen.
Yes, I understand all that, but this isn’t the issue. The issue is how much all the assorted calculation errors matter in comparison to the rest of the uncertainty in the model.
I don’t think we disagree too much. If I had to pick one, I’d agree with you that the rest of the uncertainty is likely larger in most cases, but I think you substantially underestimate how inaccurate these numerical methods can be. Many commercial computational fluid dynamics codes use quite bad numerical methods along with large grid cells and time steps, so it seems possible to me that those errors can exceed the uncertainties in the other parameters. I can think of one case in particular in my own work where the numerical errors likely exceed the other uncertainties.
Even single-precision floating point gives you around 7 decimal digits of accuracy. If (as is the case for both weather and climate modelling) the inputs are not known with anything like that amount of precision, surely input uncertainty will overwhelm calculation noise? Calculation noise enters at every step, of course, but even so, there must be diminishing returns from increased precision.
See the second half of this cousin comment. But a short summary (with a bit of additional info):
First, I see a philosophical difference between input uncertainty and calculation noise; the mathematical tools you need to attack each problem are different. The first can be solved through sampling (or a number of other different ways); the second can be solved with increased precision (or a number of other different ways). Importantly, sampling does not seem to me to be a promising approach to solving the calculation noise problem, because the errors may be systematic instead of random. In chaotic systems, this problem seems especially important.
Second, it seems common for both weather models and climate models to use simulation time steps of about 10 minutes. If you want to predict 6 days ahead, that’s 864 time steps. If you want to predict 60 years ahead, that’s over 3 million time steps.
Many combustion modeling approaches do precisely this. Look into prescribed PDF methods, for example. You can see the necessity of this by recognizing that ignition can occur if the temperature anywhere in a cell is above the ignition temperature.
(There is also the slightly confusing issue that these distributions are not the same thing as the distribution of possible realizations.)
The differences between climate and meteorological models are reasons that should only increase someone’s confidence in the relative capabilities of climate science, so the analogy seems apt despite these differences.
What’s that got to do with causal structure?
I am not sure what you mean by “causal structure” in this context. I was attempting to provide some intuition as to why ordinary weather forecasting and climate change modeling would be different, since you stated that you didn’t see what the essential difference between them is.
But it was a short comment, and so many things were only left as implications. For example, the cell update laws (i.e. the differential equations guiding the system) will naturally be different for weather forecasting and climate forecasting because the cells are physically different beasts. You’ll model cloud dynamics very differently depending on whether or not clouds are bigger or smaller than a model cell, and it’s not necessarily the case that a fine-grained model will be more accurate than a coarse-grained model, for many reasons.
Understanding causal structure seems to be something that is kind of shiny and impressive sounding, connotationally, but doesn’t mean much, at least not much that is new, denotationally. And it comes up because I thought was replying to DVH, who brought it up.
I don’t think CC modelling and weather forecasting are all that essentially different, or at least not as different as Causal Structure is supposed to be from either.
The pattern “the experts in X are actually incompetent fools, because they are not doing Y” is frequent in LessWrong Classic, even if it hasn’t been applied to climate change before.
I think one reasonable complaint is that you should not use predictive models to guide policy because of the usual issues with confounding.
Unguided policy is better?
No? What do you think my position is?
Model bias is not a joke. If your model is severely biased, it is giving you garbage. I am not sure in what sense a model that outputs garbage is better than no model at all. The former just gives you a false sense of confidence, because math was used.
If you think there are [reasons] where the model isn’t completely garbage, or we can put bounds on garbage, or something, then that is a useful conversation to have.
If you set up the conversation where it’s the garbage model or no science at all, then you are engaged in rhetoric, not science.
I don’t suppose public policy is based on a single model.
If you read back, nothing has been said about any specific model, so no such claim needs defending.
If you read back, it has been suggested that there is a much better way of doing climate .science than modelling of any kind....but details are lacking.
If I read back I also read things like this:
No, it means a whole lot. You need to get the causal structure right, or at least reasonably close, or your model is garbage for policy. See also: “irrational policy of managing the news.”
I fight this fight, along with my colleagues, in much simpler settings than weather. And it is still difficult.
Related. Sample:
Getting causal structure right in that sense is not an alternative to modelling, it is part of getting modelling right.
If you don’t want to talk in binary black-or-white terms, perhaps you shouldn’t lead with a set-up where a model outputs either truth or garbage ;-)