Firstly, there is one clear standout from these: Sample 6. This must be GPT4. It could come straight out of the original story. None of the others give off that smell so intensely. While they could all be written by humans, I am very suspicious of some of them.
Here are my detailed criticisms, unassisted by any consultation of the Internet. I’m writing based on the assumption that the human-written extracts (if any) are from commercially published (and not self-published) works.
Sample 1: Probably human, on account of the coherence and lack of slack writing. There is one red flag, the “tightly-wound and elaborately crafted drinking costumes” that are apparently worn every weekend, not just on some special occasion. This ethnographic detail is just weird, given that nothing else in the passage suggests that this is fantastic fiction. However, the passage as a whole has better coherence than GPT4 usually manages, and the successful exact parallelism in “Pick a Friday or a Saturday, any Friday or Saturday of the year” is something I have not seen GPT4 do.
Sample 2: Probably GPT4. Each sentence on its own is innocuous enough, but on the larger scale it seems to veer from talking about the book and the movie to telling the Pinocchio story. What is this “halter”? Why mention the dowels? Or course there would be dowels in a puppet’s joints, and they wouldn’t just “seem” to be there.
Sample 3: Doubtful, but I lean towards GPT. “Frozen in the grasp of time” is cliched. If the story has already established this anomalous city, there’s no reason to describe it here with that phrase. It is also inconsistent with the motion of time implied by “fading sunlight”. When Finn’s fall is stilled, the shadows of buildings are described as statues. The shadows of buildings do not look like statues. The movement of shadows is not ordinarily perceptible anyway.
Sample 4: Human. I’m not aware that GPT4 can spontaneously use italics. Leaving that aside, the italicized paragraph is not something that GPT4 would write. The first and last paragraphs have something of its thump, thump repetitiveness, but human writers do sometimes do that, especially at the start of a story to establish the scene.
Sample 5: I’m about 60-40 on this one, more likely GPT than not. It could be from near the start of a Ray Bradbury story, but the writing just isn’t tight enough. For example, “blinked out of existence just as quickly as it had appeared” is redundant. A blink is as fast as anything could appear or disappear.
Sample 6: As I said, this instantly stands out as GPT4. It’s junk. Even if every other sample is human, this one is not.
Sample 7: GPT4. Land vehicles don’t have sterns. The scene describes the vehicle sinking into a sudden subsidence, but that is a hazard that any vehicle would be susceptible to, given a big enough subsidence. The surprise should not be that the vehicle sank, but that the subsidence happened at all. “Steam”, escaping from a Mars vehicle? No. The extract ends in the middle of a sentence, but that might just be a copy-paste error.
Sample 8: Likely human. It’s the sort of writing that single-mindedly is exactly what it is, but it has a ring to it without repetitions, that I don’t find in GPT4′s bells of lead.
I’ve marked all of these quite sternly, but if I’ve wrongly accused any real authors of writing like GPT4, I stand by the criticisms I’ve made of what they’ve written.
Of course, real writers, even stars of the literary firmament, do nod, and lesser-known published writers are often lesser-known for a good reason. David Langford’s science fiction newsletter Ansible has a regular item called Thog’s Masterclass, exhibiting examples of “differently good” actually published writing. Dare the Thog-o-Matic to see some random examples. ETA: or look at any Perry Rhodan novel.
You got 4 of 8 right. In two cases you failed to recognize humans, and in another two—GPT4.
It was a weakly adversarial test:
I took a few less-known but obviously talented writers from the top of my head, and copied the excerpts from the first pages.
For GPT4, I’ve used several prompts from the competition, and then selected the parts for their stylistic diversity.
I suspect that a test with longer excerpts would be much easier for you, as the vanilla GPT4 is indeed often easy to detect due to its repetitiveness etc (I haven’t tried the APIs yet).
If GPT4 already can fool some of us science fiction junkies, I can’t wait to read the fiction by GPT5.
David Langford’s science fiction newsletter Ansible has a regular item called Thog’s Masterclass, exhibiting examples of “differently good” actually published writing. Dare the Thog-o-Matic to see some random examples. ETA: or look at any Perry Rhodan novel.
Thank you!
BTW, have you read “Appleseed” by John Clute? I have a feeling you may be one of the few people on Earth who can enjoy it. A representative sample:
They passed the iron-grey portcullis that sealed off the inferno of drive country. A dozen ceremonial masks, mourning the hardened goblin eidolons of KathKirtt that died hourly inside drive country, hung within their tile embrasure above the frowning portal. The masks were simplified versions of the flyte gorgon. Their single eyes shut in unison at the death of one of the goblin eidolons, who spent their brief spans liaising with the quasi-sentient engine brother that drove the ship through the demonic rapturous ftl maze of wormholes. Even for eidolons with hardened carapaces, to liaise was to burn and die. When Tile Dance plunged through the ashen caltraps of ftl at full thrust, the engine brother howling out something like anguish or joy all the while, its entirely imaginary ‘feet’ pounding the turns of the maze, goblins lived no longer than mayflies.
Very interesting. My accuracy was the same as Richard’s: 4/8*. I think you probably used my prompt for one of the ones I got right, which is probably why I got it right (the tone and structure are very familiar to me after so much experimentation).
To those who think the current crop of AIs aren’t capable of writing great novellas (18-40k words): Do you think your opinion will change in the next 5 years?
* I originally reported a score of 1⁄8 by mistake.
I think you probably used my prompt for the one I got right, which is probably why I got it right (the tone and structure are very familiar to me after so much experimentation).
Nope, this one. But their prompt does incorporate some ideas from your prompt.
Firstly, there is one clear standout from these: Sample 6. This must be GPT4. It could come straight out of the original story. None of the others give off that smell so intensely. While they could all be written by humans, I am very suspicious of some of them.
Here are my detailed criticisms, unassisted by any consultation of the Internet. I’m writing based on the assumption that the human-written extracts (if any) are from commercially published (and not self-published) works.
Sample 1: Probably human, on account of the coherence and lack of slack writing. There is one red flag, the “tightly-wound and elaborately crafted drinking costumes” that are apparently worn every weekend, not just on some special occasion. This ethnographic detail is just weird, given that nothing else in the passage suggests that this is fantastic fiction. However, the passage as a whole has better coherence than GPT4 usually manages, and the successful exact parallelism in “Pick a Friday or a Saturday, any Friday or Saturday of the year” is something I have not seen GPT4 do.
Sample 2: Probably GPT4. Each sentence on its own is innocuous enough, but on the larger scale it seems to veer from talking about the book and the movie to telling the Pinocchio story. What is this “halter”? Why mention the dowels? Or course there would be dowels in a puppet’s joints, and they wouldn’t just “seem” to be there.
Sample 3: Doubtful, but I lean towards GPT. “Frozen in the grasp of time” is cliched. If the story has already established this anomalous city, there’s no reason to describe it here with that phrase. It is also inconsistent with the motion of time implied by “fading sunlight”. When Finn’s fall is stilled, the shadows of buildings are described as statues. The shadows of buildings do not look like statues. The movement of shadows is not ordinarily perceptible anyway.
Sample 4: Human. I’m not aware that GPT4 can spontaneously use italics. Leaving that aside, the italicized paragraph is not something that GPT4 would write. The first and last paragraphs have something of its thump, thump repetitiveness, but human writers do sometimes do that, especially at the start of a story to establish the scene.
Sample 5: I’m about 60-40 on this one, more likely GPT than not. It could be from near the start of a Ray Bradbury story, but the writing just isn’t tight enough. For example, “blinked out of existence just as quickly as it had appeared” is redundant. A blink is as fast as anything could appear or disappear.
Sample 6: As I said, this instantly stands out as GPT4. It’s junk. Even if every other sample is human, this one is not.
Sample 7: GPT4. Land vehicles don’t have sterns. The scene describes the vehicle sinking into a sudden subsidence, but that is a hazard that any vehicle would be susceptible to, given a big enough subsidence. The surprise should not be that the vehicle sank, but that the subsidence happened at all. “Steam”, escaping from a Mars vehicle? No. The extract ends in the middle of a sentence, but that might just be a copy-paste error.
Sample 8: Likely human. It’s the sort of writing that single-mindedly is exactly what it is, but it has a ring to it without repetitions, that I don’t find in GPT4′s bells of lead.
I’ve marked all of these quite sternly, but if I’ve wrongly accused any real authors of writing like GPT4, I stand by the criticisms I’ve made of what they’ve written.
Of course, real writers, even stars of the literary firmament, do nod, and lesser-known published writers are often lesser-known for a good reason. David Langford’s science fiction newsletter Ansible has a regular item called Thog’s Masterclass, exhibiting examples of “differently good” actually published writing. Dare the Thog-o-Matic to see some random examples. ETA: or look at any Perry Rhodan novel.
The key[1].
You got 4 of 8 right. In two cases you failed to recognize humans, and in another two—GPT4.
It was a weakly adversarial test:
I took a few less-known but obviously talented writers from the top of my head, and copied the excerpts from the first pages.
For GPT4, I’ve used several prompts from the competition, and then selected the parts for their stylistic diversity.
I suspect that a test with longer excerpts would be much easier for you, as the vanilla GPT4 is indeed often easy to detect due to its repetitiveness etc (I haven’t tried the APIs yet).
If GPT4 already can fool some of us science fiction junkies, I can’t wait to read the fiction by GPT5.
Thank you!
BTW, have you read “Appleseed” by John Clute? I have a feeling you may be one of the few people on Earth who can enjoy it. A representative sample:
Only one, two, and seven are human: from “Ra” by qntm, “Contact” by Sagan, “Noon: 22nd Century” by Strugatsky. The rest is GPT4Very interesting. My accuracy was the same as Richard’s: 4/8*. I think you probably used my prompt for one of the ones I got right, which is probably why I got it right (the tone and structure are very familiar to me after so much experimentation).
To those who think the current crop of AIs aren’t capable of writing great novellas (18-40k words): Do you think your opinion will change in the next 5 years?
* I originally reported a score of 1⁄8 by mistake.
Nope, this one. But their prompt does incorporate some ideas from your prompt.