The Simulators post repeatedly alludes to the loss function on which GPTs are trained corresponding to a “simulation objective”, but I don’t really see why that would be true. It is technically true that a GPT that perfectly simulates earth, including the creation of its own training data set, can use that simulation to get perfect training loss. But actually doing so would require enormous amounts of compute and we of course know that nothing close to that is going on inside of GPT-4.
I think a lot of what is causing confusion here is the word ‘simulation’. People often talk colloquially about “running a weather simulation” or “simulating an aircraft’s wing under stress”. This is a common misnomer, technically the correct word they should be using there is ‘emulation’. If you are running a detailed analysis of each subprocess that matters and combining the all their interactions together to produce a detail prediction, then you are ‘emulating’ something. On the other hand, if you’re doing something that more resembles a machine learning model pragmatically leaning its behavior (what one could even call a stochastic parrot), trained to predict the same outcomes over some large set of sample situations, then you’re running a ‘simulation’.
As janus writes:
Self-supervised ML can create “behavioral” simulations of impressive semantic fidelity. Whole brain emulation is not necessary to construct convincing and useful virtual humans; it is conceivable that observations of human behavioral traces (e.g. text) are sufficient to reconstruct functionally human-level virtual intelligence.
So he is clearly and explicitly making this distinction between the words ‘simulation’ and ‘emulation’, and evidently understands the correct usage of each of them. To pick a specific example, the weather models that most government’s meteorological departments run are emulations that divide the entire atmosphere (or the part near that country) into a great many small cells and emultate the entire system (except at the level of the smallest cells, where they fall back on simulation since they cannot afford to further subdivide the problem, as the physics of turbulence would otherwise require); whereas the (vastly more computationally efficient) GraphCast system that DeepMind recently built is a simulation. It basically relies on the weather continuing to act in the future in ways it has in the past (so potentially could be thrown off by effects like global warming). So Simulator Theory is saying “LLMS work like GraphCast makes weather predictions” not “LLMs work like detailed models of the atmosphere split into a vast number of tiny cells make weather predictions”.
[The fact that this is even possible in non-linear systems is somewhat surprising, as janus is expressing in the quote above, but then Science has often managed to find useful regularities in the behavior of very large systems, ones that that do not require mechanistically breaking their behavior down all the way to individual fundamental particles to model them. Most behavior most of the time is not in fact NP-complete, and has Lyapunov times much longer than the periods between interactions of its constituent fundamental particles — so clearly often a lot of the fine details wash out. Apparently this is also true of the human brain, unlike the case for computers]
So the “Simulator Theory” is not an “Emulator Theory”. Janus is explicitly not claiming that an LLM “perfectly [emulates] earth, including the creation of its own training data set”. Any fan of Simulator Theory who make claims like that has not correctly understood it (most likely due to this common confusion over the meaning of the word ‘simulate’). The claim in the Simulation Thesis is that the ML model finds and learns regularities in its training set, and them reapplies them in a way that makes (quite good) predictions, without doing a detailed emulation of the process it is predicting, in just the same way that GraphCast makes weather predictions without (and far more computationally cheaply than) emulating the entire atmosphere. (Note that this claim is entirely uncontroversial: that’s exactly what machine learning models always do when they work.) So the LLM has internal world models, but they are models of the behavior of parts of the world, not of the detailed underlying physical process that produces that behavior. Also note that while such models can sometimes correctly extrapolate outside the training distribution, this requires luck: specifically that no new phenomena become important to the behavior outside the training distribution that weren’t learnable from the behavior inside it. The risk of this being false increases the more complex the underlying system and further you attempt to extrapolate outside the training distribution.
I think a lot of what is causing confusion here is the word ‘simulation’. People often talk colloquially about “running a weather simulation” or “simulating an aircraft’s wing under stress”. This is a common misnomer, technically the correct word they should be using there is ‘emulation’. If you are running a detailed analysis of each subprocess that matters and combining the all their interactions together to produce a detail prediction, then you are ‘emulating’ something. On the other hand, if you’re doing something that more resembles a machine learning model pragmatically leaning its behavior (what one could even call a stochastic parrot), trained to predict the same outcomes over some large set of sample situations, then you’re running a ‘simulation’.
As janus writes:
So he is clearly and explicitly making this distinction between the words ‘simulation’ and ‘emulation’, and evidently understands the correct usage of each of them. To pick a specific example, the weather models that most government’s meteorological departments run are emulations that divide the entire atmosphere (or the part near that country) into a great many small cells and emultate the entire system (except at the level of the smallest cells, where they fall back on simulation since they cannot afford to further subdivide the problem, as the physics of turbulence would otherwise require); whereas the (vastly more computationally efficient) GraphCast system that DeepMind recently built is a simulation. It basically relies on the weather continuing to act in the future in ways it has in the past (so potentially could be thrown off by effects like global warming). So Simulator Theory is saying “LLMS work like GraphCast makes weather predictions” not “LLMs work like detailed models of the atmosphere split into a vast number of tiny cells make weather predictions”.
[The fact that this is even possible in non-linear systems is somewhat surprising, as janus is expressing in the quote above, but then Science has often managed to find useful regularities in the behavior of very large systems, ones that that do not require mechanistically breaking their behavior down all the way to individual fundamental particles to model them. Most behavior most of the time is not in fact NP-complete, and has Lyapunov times much longer than the periods between interactions of its constituent fundamental particles — so clearly often a lot of the fine details wash out. Apparently this is also true of the human brain, unlike the case for computers]
So the “Simulator Theory” is not an “Emulator Theory”. Janus is explicitly not claiming that an LLM “perfectly [emulates] earth, including the creation of its own training data set”. Any fan of Simulator Theory who make claims like that has not correctly understood it (most likely due to this common confusion over the meaning of the word ‘simulate’). The claim in the Simulation Thesis is that the ML model finds and learns regularities in its training set, and them reapplies them in a way that makes (quite good) predictions, without doing a detailed emulation of the process it is predicting, in just the same way that GraphCast makes weather predictions without (and far more computationally cheaply than) emulating the entire atmosphere. (Note that this claim is entirely uncontroversial: that’s exactly what machine learning models always do when they work.) So the LLM has internal world models, but they are models of the behavior of parts of the world, not of the detailed underlying physical process that produces that behavior. Also note that while such models can sometimes correctly extrapolate outside the training distribution, this requires luck: specifically that no new phenomena become important to the behavior outside the training distribution that weren’t learnable from the behavior inside it. The risk of this being false increases the more complex the underlying system and further you attempt to extrapolate outside the training distribution.