This is a topic I frequently see misunderstood, and as a programmer who has built simple physics simulations I have some expertise on the topic, so perhaps I should elaborate.
If you have a simple, linear system involving math that isn’t too CPU-intensive you can build an accurate computer simulation of it with a relatively modest amount of testing. Your initial attempt will be wrong due to simple bugs, which you can probably detect just by comparing simulation data with a modest set of real examples.
But if you have a complex, non-linear system, or just one that’s too big to simulate in complete detail, this is no longer the case. Getting a useful simulation then requires that you make a lot of educated guesses about what factors to include in your simulation, and how to approximate effects you can’t calculate in any detail. The probability of getting these guesses right the first time is essentially zero—you’re lucky if the behavior of your initial model has even a hazy resemblance to anything real, and it certainly isn’t going to come within an order of magnitude of being correct.
The way you get to a useful model is through a repeated cycle of running the simulator, comparing the (wrong) results to reality, making an educated guess about what caused the difference, and trying again. With something relatively simple like, say, turbulent fluid dynamics, you might need a few hundred to a few thousand test runs to tweak your model enough that it generates accurate results over the domain of input parameters that you’re interested in.
If you can’t run real-world experiments to generate the phenomena you’re interested in, you might be able to substitute a huge data set of observations of natural events. Astronomy has had some success with this, for example. But you need a data set big enough to encompass a representative sample of all the possible behaviors of the system you’re trying to simulate, or else you’ll just gets a ‘simulator’ that always predicts the few examples you fed it.
So, can you see the problem with the nuclear winter simulations now? You can’t have a nuclear war to test the simulation, and our historical data set of real climate changes doesn’t include anything similar (and doesn’t collect anywhere near as many data points as a simulator needs, anyway). But global climate is a couple of orders of magnitude more complex than your typical physics or chemistry sims, so the need for testing would be correspondingly greater.
The point non-programmers tend to miss here is that lack of testing doesn’t just mean the model is a a little off. It means the model has no connection at all to reality, and either outputs garbage or echoes whatever result the programmer told it to give. Any programmer who claims such a model means something is committing fraud, plain and simple.
There are particulate events in the climate record that a model of nuclear winter could be calibrated against—any major volcanic eruption, for example. Some have even approached the level of severity predicted for a mild nuclear winter: the “Year Without A Summer” following the 1815 Tambora eruption is the first one I can think of.
This isn’t perfect: volcanoes mainly release fine rock ash instead of the wood and hydrocarbon soot that we’d expect from burning cities, which behaves differently in the atmosphere, and while we can get some idea of the difference from looking at events like large-scale forest fires there are limits on how far we can extrapolate. But we should have enough to at least put some bounds on what we could expect.
Different components in the model can be tested separately. How stratospheric gases disperse can be tested. How black soot rises in the atmosphere, in a variety of heat conditions, can be tested. How black soot affects absorption of the solar radiation can be simulated in laboratory, and tested in indirect ways (as Nornagest mentioned, by comparing with volcanic eruptions).
Yes, and that’s why you can even attempt to build a computer model. But you seem to be assuming that a climate model can actually simulate all those processes on a relatively fundamental level, and that isn’t the case.
When you set out to build a model of a large, non-linear system you’re confronted with a list of tens of thousands of known processes that might be important. Adding them all to your model would take millions of man-hours, and make it so big no computer could possibly run it. But you can’t just take the most important-looking processes and ignore the rest, because the behavior of any non-linear system tends to be dominated by unexpected interactions between obscure parts of the system that seem unrelated at first glance.
So what actually happens is you implement rough approximations of the effects the specialists in the field think are important, and get a model that outputs crazy nonsense. If you’re honest, the next step is a long process of trying to figure out what you missed, adding things to the model, comparing the output to reality, and then going back to the drawing board again. There’s no hard, known-to-be-accurate physics modeling involved here, because that would take far more CPU power than any possible system could provide. Instead it’s all rules of thumb and simplified approximations, stuck together with arbitrary kludges that seem to give reasonable results.
Or you can take that first, horribly broken model, slap on some arbitrary fudge factors to make it spit out results the specialists agree look reasonable, and declare your work done. Then you get paid, the scientists can proudly show off their new computer model, and the media will credulously believe whatever predictions you make because they came out of a computer. But in reality all you’ve done is build an echo chamber—you can easily adjust such a model to give any result you want, so it provides no additional evidence.
In the case of nuclear winter there was no preexisting body of climate science that predicted a global catastrophe. There was just a couple of scientists who thought it would happen, and built a model to echo their prediction.
The point non-programmers tend to miss here is that lack of testing doesn’t just mean the model is a a little off. It means the model has no connection at all to reality, and either outputs garbage or echoes whatever result the programmer told it to give. Any programmer who claims such a model means something is committing fraud, plain and simple.
This really is a pretty un-bayesian way of thinking—the idea that we should totally ignore incomplete evidence. And by extension that we should chose to believe an alternative hypothesis (″no nuclear winter’) with even less evidence merely because it is assumed for unstated reasons to be the ‘default belief’.
An uncalibrated sim will typically give crazy results like ‘increasing atmospheric CO2 by 1% raises surface temperatures by 300 degrees’ or ‘one large forest fire will trigger a permanent ice age’. If you see an uncalibrated sim giving results that seem even vaguely plausible, this means the programmer has tinkered with its internal mechanisms to make it give those results. Doing that is basically equivalent to just typing up the desired output by hand—it provides evidence about the beliefs of the programmer, but nothing else.
This is a topic I frequently see misunderstood, and as a programmer who has built simple physics simulations I have some expertise on the topic, so perhaps I should elaborate.
If you have a simple, linear system involving math that isn’t too CPU-intensive you can build an accurate computer simulation of it with a relatively modest amount of testing. Your initial attempt will be wrong due to simple bugs, which you can probably detect just by comparing simulation data with a modest set of real examples.
But if you have a complex, non-linear system, or just one that’s too big to simulate in complete detail, this is no longer the case. Getting a useful simulation then requires that you make a lot of educated guesses about what factors to include in your simulation, and how to approximate effects you can’t calculate in any detail. The probability of getting these guesses right the first time is essentially zero—you’re lucky if the behavior of your initial model has even a hazy resemblance to anything real, and it certainly isn’t going to come within an order of magnitude of being correct.
The way you get to a useful model is through a repeated cycle of running the simulator, comparing the (wrong) results to reality, making an educated guess about what caused the difference, and trying again. With something relatively simple like, say, turbulent fluid dynamics, you might need a few hundred to a few thousand test runs to tweak your model enough that it generates accurate results over the domain of input parameters that you’re interested in.
If you can’t run real-world experiments to generate the phenomena you’re interested in, you might be able to substitute a huge data set of observations of natural events. Astronomy has had some success with this, for example. But you need a data set big enough to encompass a representative sample of all the possible behaviors of the system you’re trying to simulate, or else you’ll just gets a ‘simulator’ that always predicts the few examples you fed it.
So, can you see the problem with the nuclear winter simulations now? You can’t have a nuclear war to test the simulation, and our historical data set of real climate changes doesn’t include anything similar (and doesn’t collect anywhere near as many data points as a simulator needs, anyway). But global climate is a couple of orders of magnitude more complex than your typical physics or chemistry sims, so the need for testing would be correspondingly greater.
The point non-programmers tend to miss here is that lack of testing doesn’t just mean the model is a a little off. It means the model has no connection at all to reality, and either outputs garbage or echoes whatever result the programmer told it to give. Any programmer who claims such a model means something is committing fraud, plain and simple.
There are particulate events in the climate record that a model of nuclear winter could be calibrated against—any major volcanic eruption, for example. Some have even approached the level of severity predicted for a mild nuclear winter: the “Year Without A Summer” following the 1815 Tambora eruption is the first one I can think of.
This isn’t perfect: volcanoes mainly release fine rock ash instead of the wood and hydrocarbon soot that we’d expect from burning cities, which behaves differently in the atmosphere, and while we can get some idea of the difference from looking at events like large-scale forest fires there are limits on how far we can extrapolate. But we should have enough to at least put some bounds on what we could expect.
Different components in the model can be tested separately. How stratospheric gases disperse can be tested. How black soot rises in the atmosphere, in a variety of heat conditions, can be tested. How black soot affects absorption of the solar radiation can be simulated in laboratory, and tested in indirect ways (as Nornagest mentioned, by comparing with volcanic eruptions).
Yes, and that’s why you can even attempt to build a computer model. But you seem to be assuming that a climate model can actually simulate all those processes on a relatively fundamental level, and that isn’t the case.
When you set out to build a model of a large, non-linear system you’re confronted with a list of tens of thousands of known processes that might be important. Adding them all to your model would take millions of man-hours, and make it so big no computer could possibly run it. But you can’t just take the most important-looking processes and ignore the rest, because the behavior of any non-linear system tends to be dominated by unexpected interactions between obscure parts of the system that seem unrelated at first glance.
So what actually happens is you implement rough approximations of the effects the specialists in the field think are important, and get a model that outputs crazy nonsense. If you’re honest, the next step is a long process of trying to figure out what you missed, adding things to the model, comparing the output to reality, and then going back to the drawing board again. There’s no hard, known-to-be-accurate physics modeling involved here, because that would take far more CPU power than any possible system could provide. Instead it’s all rules of thumb and simplified approximations, stuck together with arbitrary kludges that seem to give reasonable results.
Or you can take that first, horribly broken model, slap on some arbitrary fudge factors to make it spit out results the specialists agree look reasonable, and declare your work done. Then you get paid, the scientists can proudly show off their new computer model, and the media will credulously believe whatever predictions you make because they came out of a computer. But in reality all you’ve done is build an echo chamber—you can easily adjust such a model to give any result you want, so it provides no additional evidence.
In the case of nuclear winter there was no preexisting body of climate science that predicted a global catastrophe. There was just a couple of scientists who thought it would happen, and built a model to echo their prediction.
This really is a pretty un-bayesian way of thinking—the idea that we should totally ignore incomplete evidence. And by extension that we should chose to believe an alternative hypothesis (″no nuclear winter’) with even less evidence merely because it is assumed for unstated reasons to be the ‘default belief’.
An uncalibrated sim will typically give crazy results like ‘increasing atmospheric CO2 by 1% raises surface temperatures by 300 degrees’ or ‘one large forest fire will trigger a permanent ice age’. If you see an uncalibrated sim giving results that seem even vaguely plausible, this means the programmer has tinkered with its internal mechanisms to make it give those results. Doing that is basically equivalent to just typing up the desired output by hand—it provides evidence about the beliefs of the programmer, but nothing else.