The thing is that, when you ask this to ChatGPT, it is still the simulacrum ChatGPT that is going to answer, not an oracle prediction (like you can see in base models). If you want to know the capability of the underlying simulator with chat models, you need to sample sufficiently enough simulacra to be sure that the mistakes comes from the simulator lack of capability and not the simulacra preferences (or modes as Janus call them). For math, it is often not important to check different simulacrum, because each simulacrum tends to share the math ability (unless you use some other weird languages, @Ethan Edwards might be able to jump in here). But for other capability (like biology or cooking), changing the simulacrum with which you interact with does have a big impact on the performance of the model. You can see that in GPT-4′s technical report, languages impact performance a lot. Using another language is one of the way to modulate the simulacrum you are interacting with. I’ll showcase in the next batch of post how you can take control a bit more accurately. Tell me if you need more precision
The thing is that, when you ask this to ChatGPT, it is still the simulacrum ChatGPT that is going to answer, not an oracle prediction (like you can see in base models). If you want to know the capability of the underlying simulator with chat models, you need to sample sufficiently enough simulacra to be sure that the mistakes comes from the simulator lack of capability and not the simulacra preferences (or modes as Janus call them). For math, it is often not important to check different simulacrum, because each simulacrum tends to share the math ability (unless you use some other weird languages, @Ethan Edwards might be able to jump in here). But for other capability (like biology or cooking), changing the simulacrum with which you interact with does have a big impact on the performance of the model. You can see that in GPT-4′s technical report, languages impact performance a lot. Using another language is one of the way to modulate the simulacrum you are interacting with.
I’ll showcase in the next batch of post how you can take control a bit more accurately.
Tell me if you need more precision