A “compiler” is anything that translates a program from one representation to another. Usually this translation is from a high-level language (like Java) to a lower-level language (like JVM bytecode), but you can also have e.g. a Python → Javascript compiler that takes in Python code and produces Javascript. A “natural language compiler”, then, is one that takes in ordinary English-or-whatever sentences and emits something executable. I think this is a pretty fair way to talk about Codex: it’s the world’s first natural language compiler that’s any good at all. I call it “stochastic” because its output is not consistent: if you give Codex the same input twice, you won’t necessarily get the same result.
So if I’m writing Python with Codex, let’s say, then whenever I start to write a function I have a choice of implementation languages: Python, or Codex prompt. Codex is helpful when the prompt is easier to write than the Python implementation would have been. The question isn’t just “can I write a prompt that will elicit good-enough Python?”, it’s “is writing the prompt easier, or harder, than writing the Python?” This is why I am not excited by very low-level prompts like “takes the difference between the set s and the set t”; once you understand your problem well enough to write a prompt like that, writing the actual code is only hard if you don’t know Python well. And if you don’t know Python well, using Codex to generate it is a little risky, since then you won’t be able to catch its silly mistakes.
I will write about the context window thing too, since I think it’s close to the heart of our disagreement, but for now I’m out of time.
Thanks for natural language stochastic compiler explanation, makes a lot of sense. I broadly get a sense of what you mean by “context window” since people have been mentioning that quite a lot when talking about GPT-3. As for whether it makes sense to write docstrings for trivial things, I think this is only pointing at the Codex demo examples where people write docstrings and get results, but for most of my use cases, and when it gets really interesting, is when it auto-completes 1) while I’m writing 2) when I’m done writing and it guesses the next line 3) when I start a line by “return ” or “x = ” and wait for his auto-completion. Here, I would have no idea how to formulate it in the docstring, I just generally trust its ability to follow the logic of the code that precedes it (and I find it useful most of the time).
A “compiler” is anything that translates a program from one representation to another. Usually this translation is from a high-level language (like Java) to a lower-level language (like JVM bytecode), but you can also have e.g. a Python → Javascript compiler that takes in Python code and produces Javascript. A “natural language compiler”, then, is one that takes in ordinary English-or-whatever sentences and emits something executable. I think this is a pretty fair way to talk about Codex: it’s the world’s first natural language compiler that’s any good at all. I call it “stochastic” because its output is not consistent: if you give Codex the same input twice, you won’t necessarily get the same result.
So if I’m writing Python with Codex, let’s say, then whenever I start to write a function I have a choice of implementation languages: Python, or Codex prompt. Codex is helpful when the prompt is easier to write than the Python implementation would have been. The question isn’t just “can I write a prompt that will elicit good-enough Python?”, it’s “is writing the prompt easier, or harder, than writing the Python?” This is why I am not excited by very low-level prompts like “takes the difference between the set s and the set t”; once you understand your problem well enough to write a prompt like that, writing the actual code is only hard if you don’t know Python well. And if you don’t know Python well, using Codex to generate it is a little risky, since then you won’t be able to catch its silly mistakes.
I will write about the context window thing too, since I think it’s close to the heart of our disagreement, but for now I’m out of time.
Thanks for natural language stochastic compiler explanation, makes a lot of sense. I broadly get a sense of what you mean by “context window” since people have been mentioning that quite a lot when talking about GPT-3. As for whether it makes sense to write docstrings for trivial things, I think this is only pointing at the Codex demo examples where people write docstrings and get results, but for most of my use cases, and when it gets really interesting, is when it auto-completes 1) while I’m writing 2) when I’m done writing and it guesses the next line 3) when I start a line by “return ” or “x = ” and wait for his auto-completion. Here, I would have no idea how to formulate it in the docstring, I just generally trust its ability to follow the logic of the code that precedes it (and I find it useful most of the time).