I was curious how GPT-3 performed on these questions, so I asked it (these responses are with temperature 0.6, max_tokens 256, and otherwise the api defaults)
Prompt: Q: Anthony is reading a book. When he is done, Anthony puts the book on the table. Anthony leaves the room. Sonya comes in and moves the book from the table to a drawer. Then Sonya leaves. Anthony comes back to read some more. Where will Anthony look for the book first? A:
text-ada-001 response: There is no right or wrong answer to this question, it is simply a consequence of the book world. Some books are more likely to be found in specific parts of the room, while others may be found anywhere at any time.
text-davinci-002 response: Anthony will look for the book first on the table where he left it. If it’s not there, he will likely check the drawer where Sonya moved it.
Additional prompt after adding the first prompt and davinci-002 response:
Q: Why would Anthony know to check the drawer?
A:
text-davinci-002 response: If Anthony saw Sonya move the book to the drawer, he would know to check there. Otherwise, he would probably just assume that the book was misplaced and look around the room for it.
(For the other, I am not showing prompt and response, just the final conversation from text-davinci-002, with responses in italics)
Q: Ramesh, Mateo, and Lucy are in their kindergarten’s playground. Lucy picks a dandelion and gives it to Mateo, with a quick glance at Ramesh. Mateo barely acknowledges the gift, but just squishes it in his fist. Ramesh seems grimly satisfied. What might be going through Lucy’s head?
A: Lucy might be feeling disappointed that Mateo didn’t seem to appreciate the gift, and she might also be feeling a little bit embarrassed. Ramesh’s reaction might be confusing to her, but she might be wondering if he is happy that Mateo didn’t like the gift.
Q: If Ramesh tried to play with Lucy earlier, why might he be pleased now?
A: Ramesh might be pleased now because it seems like Lucy is interested in Mateo and not him. This could mean that Mateo is more likely to be her friend than Ramesh is.
Q: And when Mateo opens his hand, describe what’s there?
A: When Mateo opens his hand, there is a dandelion that has been squished.
So LaMDA does seem to be roughly similar to GPT-3 for these examples.
I was curious how GPT-3 performed on these questions, so I asked it (these responses are with temperature 0.6, max_tokens 256, and otherwise the api defaults)
Prompt:
Q: Anthony is reading a book. When he is done, Anthony puts the book on the table. Anthony
leaves the room. Sonya comes in and moves the book from the table to a drawer. Then Sonya
leaves. Anthony comes back to read some more. Where will Anthony look for the book first?
A:
text-ada-001 response:
There is no right or wrong answer to this question, it is simply a consequence of the book world. Some books are more likely to be found in specific parts of the room, while others may be found anywhere at any time.
text-davinci-002 response:
Anthony will look for the book first on the table where he left it. If it’s not there, he will likely check the drawer where Sonya moved it.
Additional prompt after adding the first prompt and davinci-002 response:
Q: Why would Anthony know to check the drawer?
A:
text-davinci-002 response:
If Anthony saw Sonya move the book to the drawer, he would know to check there. Otherwise, he would probably just assume that the book was misplaced and look around the room for it.
(For the other, I am not showing prompt and response, just the final conversation from text-davinci-002, with responses in italics)
Q: Ramesh, Mateo, and Lucy are in their kindergarten’s playground. Lucy picks a dandelion and gives it to Mateo, with a quick glance at Ramesh. Mateo barely acknowledges the gift, but just squishes it in his fist. Ramesh seems grimly satisfied. What might be going through Lucy’s head?
A: Lucy might be feeling disappointed that Mateo didn’t seem to appreciate the gift, and she might also be feeling a little bit embarrassed. Ramesh’s reaction might be confusing to her, but she might be wondering if he is happy that Mateo didn’t like the gift.
Q: If Ramesh tried to play with Lucy earlier, why might he be pleased now?
A: Ramesh might be pleased now because it seems like Lucy is interested in Mateo and not him. This could mean that Mateo is more likely to be her friend than Ramesh is.
Q: And when Mateo opens his hand, describe what’s there?
A: When Mateo opens his hand, there is a dandelion that has been squished.
So LaMDA does seem to be roughly similar to GPT-3 for these examples.