Well, why didn’t GPT3.5 also copy it if it was in the training data? As well, I’ve never seen the specification of ‘print hello world’ in a quine before, and checking the first 5 hits for python quine print(source_code.format(source_code)), out of the dozen or so Python quines, none of them look blatantly like this example nor do any of them print out hello-world optionally.
Why didn’t GPT-3.5 also copy it if it was in the training data?
Two possible answers:
The quine wasn’t in the training data of GPT-3.5 but was in the training data of GPT-4
GPT-4 is better at “retrieving” answers from the training data
That being said, I also briefly tried to search for this quine online and couldn’t find anything. So I agree, it probably does exhibit this new ability. The reason I was suspicious at first is because the quine prompt seemed generic enough that it could have existed before, but I see that’s not the case.
Sure, but the point is that those theories are much less likely than if GPT-3.5 had done it too.
I too was a bit surprised. Critch should probably have emphasized the hello-world twist a bit more: I don’t spend much time reading quines or recreational programming, so I was assuming it could’ve been memorized and wasn’t sure that that was ‘novel’ (there are lots of quine ‘genres’, like multilingual quines or ‘radiation-hardened’ quines) until I’d look through a bunch of results and noticed none of them had that. So his point is not that quines are somehow incredibly amazing & impossible to write hitherto, but that it’s gotten good enough at code-writing that it can meaningful modify & adapt quines.
Surely one should look for ones that are like “Quine given argument 1, output [something else] given argument 2”. The presence or absence of this sort of already very modular template being in the data would give better context.
If you have a very large training dataset and the phenomenon of interest is sparsely represented in that training data, it’s well known that as we increase the number of parameters of the model, its ability to accurately handle those cases increases. Unless there is any evidence against that simple explanation, it seems most natural to just think that the GPT4 has the required model complexity to consistently handle this somewhat rare coding phenomenon—and that GPT3.5 did not. However, I would be surprised if after poking at GPT3.5 to do quines repeatedly, that we could not get it to do something similar. In fact, having just tried myself, it gave me a perfectly elegant quine:
Yes, I know what a quine is! A quine is a computer program that takes no input and produces its complete source code as its output.
Here is an example of a Python quine:
s='s=%r;print(s%%s)';print(s%s)
When this code is executed, it assigns the string ‘s=%r;print(s%%s)’ to the variable s, then prints the string formatted with itself using the % operator. The %r format code in the string representation of s will be replaced with the value of s, producing the source code of the program.
Note that the x = 'x = {!r};print(x.format(x))';print(x.format(x)) pattern is described on the Rosetta Code page for quines. It’s possible that the trick is well known and that GPT4 was able to reach for it.
(I don’t know that I would call the resulting code copied. If I were given this prompt, the extra requirements would make me interpret it as a “show me you can fulfill these specific requirements” exercise rather than an exercise specifically about finding the trick. So, reaching for a pattern seems the appropriately level of lazy, the trick feels sufficiently less central to the question that I like the ‘applying a pattern’ label better than copying)
Depends on what you mean by “copy”. It can probably do any string, not just “Hello World” and get the right answer. The general structure of the solution it did copy obviously.
How do we know it didn’t copy this code from somewhere on the internet?
Well, why didn’t GPT3.5 also copy it if it was in the training data? As well, I’ve never seen the specification of ‘print hello world’ in a quine before, and checking the first 5 hits for
python quine print(source_code.format(source_code)), out of the dozen or so Python quines, none of them look blatantly like this example nor do any of them print out hello-world optionally.Two possible answers:
The quine wasn’t in the training data of GPT-3.5 but was in the training data of GPT-4
GPT-4 is better at “retrieving” answers from the training data
That being said, I also briefly tried to search for this quine online and couldn’t find anything. So I agree, it probably does exhibit this new ability. The reason I was suspicious at first is because the quine prompt seemed generic enough that it could have existed before, but I see that’s not the case.
Sure, but the point is that those theories are much less likely than if GPT-3.5 had done it too.
I too was a bit surprised. Critch should probably have emphasized the hello-world twist a bit more: I don’t spend much time reading quines or recreational programming, so I was assuming it could’ve been memorized and wasn’t sure that that was ‘novel’ (there are lots of quine ‘genres’, like multilingual quines or ‘radiation-hardened’ quines) until I’d look through a bunch of results and noticed none of them had that. So his point is not that quines are somehow incredibly amazing & impossible to write hitherto, but that it’s gotten good enough at code-writing that it can meaningful modify & adapt quines.
Surely one should look for ones that are like “Quine given argument 1, output [something else] given argument 2”. The presence or absence of this sort of already very modular template being in the data would give better context.
GPT-4 could also be trained for more epochs, letting it “see” this example multiple times.
If you have a very large training dataset and the phenomenon of interest is sparsely represented in that training data, it’s well known that as we increase the number of parameters of the model, its ability to accurately handle those cases increases. Unless there is any evidence against that simple explanation, it seems most natural to just think that the GPT4 has the required model complexity to consistently handle this somewhat rare coding phenomenon—and that GPT3.5 did not. However, I would be surprised if after poking at GPT3.5 to do quines repeatedly, that we could not get it to do something similar. In fact, having just tried myself, it gave me a perfectly elegant quine:
Github code searches for “python quine” and “python quine format” also don’t throw up things I’d call similar.
Note that the
x = 'x = {!r};print(x.format(x))';print(x.format(x))pattern is described on the Rosetta Code page for quines. It’s possible that the trick is well known and that GPT4 was able to reach for it.(I don’t know that I would call the resulting code copied. If I were given this prompt, the extra requirements would make me interpret it as a “show me you can fulfill these specific requirements” exercise rather than an exercise specifically about finding the trick. So, reaching for a pattern seems the appropriately level of lazy, the trick feels sufficiently less central to the question that I like the ‘applying a pattern’ label better than copying)
Depends on what you mean by “copy”. It can probably do any string, not just “Hello World” and get the right answer. The general structure of the solution it did copy obviously.