The big misconception that the simulcra post tries to correct is that, while GPT-N can simulate processes that are agentic, GPT-N is not agentic itself. Calling GPT-Ns “optimizers” in any sense of the word is the sort of basic category mistake that seems extremely silly in retrospect.
This is a misconception itself. GPT is an agent that learns to be a good simulator, an “actor”. It doesn’t mean that the actor itself is not there—that would be physically incoherent. The strength of that agency is a different question, as well as the goals of that agent: might not actually go beyond the goal of “being a good actor” (and this is just what we want, presumably, as of now). But that goal probably should be there if we want the system to be robust. (How good progress RLHF does towards that goal is a different question still.) Compare: human actors do have goals beyond “being a good actor”, e.g., being a good citizen, a good child, etc.
Fortunately, a “pure simulator” is neither physically coherent nor would it be desirable if it was physically possible. Because we don’t want actors in the theatre when playing Nero actually just “become Nero”, we want them to “be actors” in the back of their minds. This is a part of acting as a profession and what good actors are paid for—being able to go in and out of roles.
This is a misconception itself. GPT is an agent that learns to be a good simulator, an “actor”. It doesn’t mean that the actor itself is not there—that would be physically incoherent. The strength of that agency is a different question, as well as the goals of that agent: might not actually go beyond the goal of “being a good actor” (and this is just what we want, presumably, as of now). But that goal probably should be there if we want the system to be robust. (How good progress RLHF does towards that goal is a different question still.) Compare: human actors do have goals beyond “being a good actor”, e.g., being a good citizen, a good child, etc.
Fortunately, a “pure simulator” is neither physically coherent nor would it be desirable if it was physically possible. Because we don’t want actors in the theatre when playing Nero actually just “become Nero”, we want them to “be actors” in the back of their minds. This is a part of acting as a profession and what good actors are paid for—being able to go in and out of roles.