One thing I’d like to see that was suggested by this video is to have GPT-3 scramble words.
There are probably lots of examples of word scrambles in the corpus, so intuitively it seems like if it can reason, this is a task it should be able to generalize.
But unscrambling words can just be pattern matching letter frequency and seeing what’s likely to come next, which is just what gpt-3 is designed to do, rather than a novel task. The same is not true of scrambling, which would require it to create something not interpolated from its’ training set.
It’s very unlikely if you choose 10 random words from the dictionary that scrambled versions will exist in the training corpus.
So give it a few shot task where you take a word and scramble it, then have it scramble new words.
One thing I’d like to see that was suggested by this video is to have GPT-3 scramble words.
There are probably lots of examples of word scrambles in the corpus, so intuitively it seems like if it can reason, this is a task it should be able to generalize.
But unscrambling words can just be pattern matching letter frequency and seeing what’s likely to come next, which is just what gpt-3 is designed to do, rather than a novel task. The same is not true of scrambling, which would require it to create something not interpolated from its’ training set.
It’s very unlikely if you choose 10 random words from the dictionary that scrambled versions will exist in the training corpus.
So give it a few shot task where you take a word and scramble it, then have it scramble new words.
http://gwern.net/GPT-3#anagrams
I haven’t been able to get it to do that well with my contexts. The best I got was: