Something you learn pretty quickly in academia: don’t trust the demos. Systems never work as well when you select the inputs freely (and, if they do, expect thorough proof). So, I wouldn’t read too deeply into this yet; we don’t know how good it actually is.
They claim beating records on a range of standard tests (such as the Winograd schema), which is not something you can cheat by cherry-picking, assuming they are honest about the results.
Something you learn pretty quickly in academia: don’t trust the demos. Systems never work as well when you select the inputs freely (and, if they do, expect thorough proof). So, I wouldn’t read too deeply into this yet; we don’t know how good it actually is.
Vis-a-vis selecting inputs freely: OpenAI also included a large dump of unconditioned text generation in their github repo.
They claim beating records on a range of standard tests (such as the Winograd schema), which is not something you can cheat by cherry-picking, assuming they are honest about the results.
https://transformer.huggingface.co/ is a nice demonstration of GPT2 that allows you to select the inputs freely.