Prompt 1: We are going to multiply two large numbers together. We will calculate 12345 * 13579. First, I want you to list the digits of the first number and, for each digit, write down a times symbol and the second number and calculate it. Do not output anything other than this.
Prompt 2: Please verify all those calculations and correct any incorrect ones. After you have done that, do the following: Output the first result, but with “0000” added at the end. Output the second result, but with “000“ added at the end. Output the third result, but with “00” added at the end. Output the fourth result, but with “0” added at the end. Output the fifth result as-is.
Prompt 3: Now, we will add up those numbers to get the final result. However, to ensure that you do not make any mistakes, do each addition in a separate sentence. So, you will first add the first and second number. Then, you will add the third number to that result. Then, the fourth and finally the fifth.
Even with the chain of thought. Now, if GPT-4 refers to the whole integrated system beyond a neural net (e.g. if it can externally execute the generated Python program like in https://arxiv.org/abs/2211.12588), this prediction would obviously be wrong.
This is my favorite so far, since you really propose a task which is not impressive, while I would be quite impressed with most of the other suggestions. A sign for this is that my probability for GPT-4 not doing this is relatively close to 50%.
In fact there is a tradeoff here between certainty and surprise: The more confident you are GPT-4 won’t solve a task, the more surprised you will be if it does. This follows from conservation of expected evidence.
It won’t be able to multiply 5-digit integers (middle digits will be wrong).
This appears to be right.
Here’s what I got when I asked it to work out 12345 x 13579 step by step:
1 2 3 4 5 (1 * 12345, shifted four positions to the left)
1 6 7 4 6 1 6 0 5
(for reference, the correct answer is 167632755)
Using this prompting method from /u/TheCodingBoy, GPT-4 gets the correct answer:
Prompt 1: We are going to multiply two large numbers together. We will calculate 12345 * 13579. First, I want you to list the digits of the first number and, for each digit, write down a times symbol and the second number and calculate it. Do not output anything other than this.
GPT-4:
Prompt 2: Please verify all those calculations and correct any incorrect ones. After you have done that, do the following:
Output the first result, but with “0000” added at the end.
Output the second result, but with “000“ added at the end.
Output the third result, but with “00” added at the end.
Output the fourth result, but with “0” added at the end.
Output the fifth result as-is.
Do not do anything else.
GPT-4:
Prompt 3: Now, we will add up those numbers to get the final result. However, to ensure that you do not make any mistakes, do each addition in a separate sentence. So, you will first add the first and second number. Then, you will add the third number to that result. Then, the fourth and finally the fifth.
GPT-4:
Even when using chain of thought?
Even with the chain of thought. Now, if GPT-4 refers to the whole integrated system beyond a neural net (e.g. if it can externally execute the generated Python program like in https://arxiv.org/abs/2211.12588), this prediction would obviously be wrong.
This is my favorite so far, since you really propose a task which is not impressive, while I would be quite impressed with most of the other suggestions. A sign for this is that my probability for GPT-4 not doing this is relatively close to 50%.
In fact there is a tradeoff here between certainty and surprise: The more confident you are GPT-4 won’t solve a task, the more surprised you will be if it does. This follows from conservation of expected evidence.