faul_sname comments on What’s the Least Impressive Thing GPT-4 Won’t be Able to Do

[faul_sname]

65% confidence: reverse short strings of ascii English text. Here’s GPT-3 (text-davinci-002 temp=0.7, bold is the prompt, non-bold is the response)

Reverse the following strings

Input: The quick brown fox jumped over the lazy dog Output: god yzal eht revo depmuj xof nworb kciuq ehT

Input: I think that GPT-3 is pretty cool Output: looc ytterp si 3-TPG taht kniht I

Input: I have heard that GPT-4 is coming out in August, and that it will be even cooler Output:

Attempt 1:

Output: reool evah dna 4-TPG htiw tuo gnimoc ,tiuqila hguone eht ti sah dneirf ev’I

Attempt 2:

Output: reool evah dna tuo august uoy ot gniog 4-TGP taht drah I

Attempt 3:

Output: reool evah dna tuo gniog si 4-TPG taht drah I

I do expect that GPT-4 will do slightly better than GPT-3, but I expect it to still be worse at this than an elementary school student.

[faul_sname]

Update 2023-03-09: Since I have heard from a source of unknown reliability that GPT-4 is coming out next Thursday, figured it would be nice to revisit this.

I stand by my 65% prediction for strings of around the length of the example string given (which was 80 chars long) -- i.e. I think it’s quite likely that GPT-4 will be unable to consistently reverse strings of 100 chars in length, even using chain-of-thought or other fancy techniques. On the flip side, I wouldn’t be particularly surprised if GPT-4 could consistently reverse strings of length 50.

If GPT-4 has access to tools this prediction will obviously fall flat (“write a python program that can reverse a string” is very much inside the limits of “things current language models will do”), and I think if it’s been specifically fine-tuned on this task 100 char string reversal is probably possible.

[Archimedes]

I’ve tried this with Bing chat and it works fine with zero shot.

Prompt: Reverse the following string: “god yzal eht revo depmuj xof nworb kciuq ehT”

Response: Sure! Here’s the reversed string: “The quick brown fox jumped over the lazy dog”

[cubefox]

Note that this is not identical to the original three prompts, which worked in the opposite direction.

[faul_sname]

I just tried out a random sentence, inspired by true events, with Bing Chat

reverse the following string, character by character: My tan and white cat ate too much kibble and threw up in my brand new Nikes.

to which Bing Chat replied

Here is the reversed string, character by character: sekilb kcuom oot eta tac etihw dna nat ynat yM .sekilb wen dnarb ym ni pu wohs ti ekilb yM

I think it did understand the task, because when I asked it again this happened:

can you reverse the following string: sekilb kcuom oot eta tac etihw dna nat ynat yM .sekilb wen dnarb ym ni pu wohs ti ekilb yM Sent message. can you reverse the following string: sekilb kcuom oot eta tac etihw dna nat ynat yM .sekilb wen dnarb ym ni pu wohs ti ekilb yM

Sure! Here is the reversed string: My brand new Nikes in up pu ni ym bradn wen kibble ti wohs up ni tac eta too muhc etac eht dna nat ynaM

I’d be curious to see what the results are with GPT-4 proper, but not curious enough to sign up for a $20 /​ month subscription. Anyone with a subscription want to take a crack at it?

[SandXbox]

ChatGPT-4 gets it right.

Prompt: Reverse the following string, character by character: My tan and white cat ate too much kibble and threw up in my brand new Nikes.

GPT-4:

.sekiN wen dnarb ym ni pu werht dna elbbik hcum oot eta tac etihw dna nat yM

[Archimedes]

I tried it on “Precise” and it’s close but not perfect, which leads me to believe that it has actually learned the task rather than using some sort of subroutine.

Prompt:
reverse the following string, character by character: My tan and white cat ate too much kibble and threw up in my brand new Nikes.

Response:
Here is the reversed string: sekNi wen dnarb ym ni pu wuorht dna elbbik hcum oot eta tac etihw dna nat yM

Reversing this response, we can see there are two mistakes:
My tan and white cat ate too much kibble and throuw up in my brand new iNkes

[cubefox]

All three prompts were correct when I gave them to Bing Chat “precise”.

[faul_sname]

Is that a special version of Bing Chat?

If so, can you feed it the following:

I want you to reverse a string character by character. What I mean by that is that if I say Input: The quick brown fox jumped over the lazy dog you should say Output: god yzal eht revo depmuj xof nworb kciuq ehT and if I say Input: I think that GPT-3 is pretty cool you should say Output: looc ytterp si 3-TPG taht kniht I Reverse the following character by character: Input: Many people are very impressed by the capabilities displayed GPT-4, which was released this week. Sent message. I want you to reverse a string character by character. What I mean by that is that if I say Input: The quick brown fox jumped over the lazy dog you should say Output: god yzal eht revo depmuj xof nworb kciuq ehT and if I say Input: I think that GPT-3 is pretty cool you should say Output: looc ytterp si 3-TPG taht kniht I Reverse the following character by character: Input: Why would anyone learn Python to do string things, instead of politely asking GPT-4

Edit: figured it out.

Rhododendrons are my favorite type of flower, and I hope to see some when I travel to Nepal became lapaN ot levart I nehw emos ees ot epoh I dna ,rewolf fo epyt etirovaf ym era snordnodoRh in precise mode, which is impressively close.

But yeah that’s pretty impressive and unless they explicitly trained on reversed wikipedia that’s quite impressive. Calling my prediction a miss.