Overall, I agree with most of this post, thanks for writing it.
The term “Simulator” has a potentially dangerous connotation of precision and reliability
I agree with your discussion of the importance of having the right vocabulary. However, I feel that the term “simulator” that you propose has a nagging flaw: that is, it invokes the connotation of “precision simulation” in people with a computer engineering background, so perhaps in most alignment researchers (rather than, I guess, the main connotation invoked in the general public, as in “Alice simulated illness to skip classes”, which is actually closer to what GPT does). Additionally, the simulation hypothesis sometimes (though not always) assumes a “precision simulation”, not an “approximate simulation” a.k.a. prediction, which GPT really does and will do.
To me, it’s obvious that GPT-like AIs will always be “predictors”, not “precision simulators” because of computation boundedness and context (prompt, window) boundedness.
Why this false connotation of precision is bad? Because it seems to lead to over-estimation of simulacra rolled out by GPT. Such as in the following sentence:
Simulators like GPT give us methods of instantiating intelligent processes, including goal-directed agents, with methods other than optimizing against a reward function.
At the very least, the statement that GPT can “instantiate intelligent processes, including goal-directed agents” should be proven (it’s very far from obvious for me that what is produced by GPT in such cases can be called an intelligent agent), and I feel there is much more nuance to it than thinking of GPT as a “simulator” tempts you to claim.
What term do I propose instead? To me, it’s somewhere in the conceptual cloud of meaning between the verbs “simulate”, “imagine”, “predict”, “generate”, “reason”, and “sample”. Perhaps, we should better coin a new term.
If we are accustomed to thinking of AI systems as corresponding to agents, it is natural to interpret behavior produced by GPT – say, answering questions on a benchmark test, or writing a blog post – as if it were a human that produced it. We say “GPT answered the question {correctly|incorrectly}” or “GPT wrote a blog post claiming X”, and in doing so attribute the beliefs, knowledge, and intentions revealed by those actions to the actor, GPT (unless it has ‘deceived’ us).
But when grading tests in the real world, we do not say “the laws of physics got this problem wrong” and conclude that the laws of physics haven’t sufficiently mastered the course material. If someone argued this is a reasonable view since the test-taker was steered by none other than the laws of physics, we could point to a different test where the problem was answered correctly by the same laws of physics propagating a different configuration. The “knowledge of course material” implied by test performance is a property of configurations, not physics.
This analogy is rather confusing because there is no information about the right answers to the test stored in the laws of physics, but there is a lot of information stored in the “GPT rule”, which is (and will be) far from pure “laws of physics” simulator, but full of factual knowledge and heuristics. This is because of the issue discussed elsewhere in the post: prompts under-determine the simulacra, but GPT has to generalise from that nonetheless.
The implication of this observation is that, I think, in a certain sense, it’s reasonable to say that “GPT wrote a blog post”, much more reasonable than “laws of physics wrote a blog post”, though perhaps less reasonable than “an agent wrote a blog post”. But I think it’s not right to declare (which seems to me, you do) that people make a semantic mistake when they say “GPT wrote a blog post”.
In the simulation ontology, I say that GPT and its output-instances correspond respectively to the simulator and simulacra. GPT is to a piece of text output by GPT as quantum physics is to a person taking a test, or as transition rules of Conway’s Game of Life are to glider. The simulator is a time-invariant law which unconditionally governs the evolution of all simulacra.
Putting quantum physics and a person taking a test in this row of analogies is problematic for at least two reasons: 1) quantum physics is not all the physics there is (at least as long as there is no theory of quantum gravity), 2) ontologically, seeing the laws of physics as a simulator and the reality around us as its simulacra is just one interpretation of what the laws of physics really are (roughly—realism). But there are also non-realist views.
Learned simulations can be partially observed and lazily-rendered, and still work. A couple of pages of text severely underdetermines the real-world process that generated text, so GPT simulations are likewise underdetermined. A “partially observed” simulation is more efficient to compute because the state can be much smaller, but can still have the effect of high fidelity as details can be rendered as needed.
Invoking the phrase “partially observed” is very confusing here, because we are not talking about some ground state of the world and the observation window into it (as in partially observed Markov decision process), but of something very different from that.
The tradeoff is that it requires the simulator to model semantics – human imagination does this, for instance – which turns out not to be an issue for big models.
Overall, I agree with most of this post, thanks for writing it.
The term “Simulator” has a potentially dangerous connotation of precision and reliability
I agree with your discussion of the importance of having the right vocabulary. However, I feel that the term “simulator” that you propose has a nagging flaw: that is, it invokes the connotation of “precision simulation” in people with a computer engineering background, so perhaps in most alignment researchers (rather than, I guess, the main connotation invoked in the general public, as in “Alice simulated illness to skip classes”, which is actually closer to what GPT does). Additionally, the simulation hypothesis sometimes (though not always) assumes a “precision simulation”, not an “approximate simulation” a.k.a. prediction, which GPT really does and will do.
To me, it’s obvious that GPT-like AIs will always be “predictors”, not “precision simulators” because of computation boundedness and context (prompt, window) boundedness.
Why this false connotation of precision is bad? Because it seems to lead to over-estimation of simulacra rolled out by GPT. Such as in the following sentence:
At the very least, the statement that GPT can “instantiate intelligent processes, including goal-directed agents” should be proven (it’s very far from obvious for me that what is produced by GPT in such cases can be called an intelligent agent), and I feel there is much more nuance to it than thinking of GPT as a “simulator” tempts you to claim.
What term do I propose instead? To me, it’s somewhere in the conceptual cloud of meaning between the verbs “simulate”, “imagine”, “predict”, “generate”, “reason”, and “sample”. Perhaps, we should better coin a new term.
This analogy is rather confusing because there is no information about the right answers to the test stored in the laws of physics, but there is a lot of information stored in the “GPT rule”, which is (and will be) far from pure “laws of physics” simulator, but full of factual knowledge and heuristics. This is because of the issue discussed elsewhere in the post: prompts under-determine the simulacra, but GPT has to generalise from that nonetheless.
The implication of this observation is that, I think, in a certain sense, it’s reasonable to say that “GPT wrote a blog post”, much more reasonable than “laws of physics wrote a blog post”, though perhaps less reasonable than “an agent wrote a blog post”. But I think it’s not right to declare (which seems to me, you do) that people make a semantic mistake when they say “GPT wrote a blog post”.
Putting quantum physics and a person taking a test in this row of analogies is problematic for at least two reasons: 1) quantum physics is not all the physics there is (at least as long as there is no theory of quantum gravity), 2) ontologically, seeing the laws of physics as a simulator and the reality around us as its simulacra is just one interpretation of what the laws of physics really are (roughly—realism). But there are also non-realist views.
Invoking the phrase “partially observed” is very confusing here, because we are not talking about some ground state of the world and the observation window into it (as in partially observed Markov decision process), but of something very different from that.
What do you mean by “modelling semantics” here?