I’m trying this with GPT-J and so far after at least ten tries it still hasn’t gotten the answer correct. Admittedly I’m using a more complicated example than the one in the image, because I thought “surely if it works so well with a simple multiplication problem it will work with something more complicated”—I probably should have checked the simple multiplication problem, because the following failed miserably every time:
Q: James drove on a full tank of gas at a speed of fifty miles an hour from his home to his workplace. His car gets a gas mileage of thirty miles per gallon. Gas costs $2.40 per gallon, and after getting to work he filled up the tank again, which cost him six dollars. How long did it take him to get to work? A: Let’s think step by step.
Oh gosh. I tried switching to the example in the post and it still can’t get it right. It said 375 once but its chain of reasoning was… well, no reasoning at all, it just made up three numbers and added them together. Clearly GPT-J is not as good at math as GPT-3 is!
I’m trying this with GPT-J and so far after at least ten tries it still hasn’t gotten the answer correct. Admittedly I’m using a more complicated example than the one in the image, because I thought “surely if it works so well with a simple multiplication problem it will work with something more complicated”—I probably should have checked the simple multiplication problem, because the following failed miserably every time:
Oh gosh. I tried switching to the example in the post and it still can’t get it right. It said 375 once but its chain of reasoning was… well, no reasoning at all, it just made up three numbers and added them together. Clearly GPT-J is not as good at math as GPT-3 is!