Thanks for writing this! I appreciate it and hope you share more things that you write faster without totally polishing everything.
One word of caution I’d share is: beware of spending too much effort running experiments on toy examples. I think toy examples are useful to gain conceptual clarity. However, if your idea is primarily empirical (such as an improvement to a deep neural network architecture), then I would recommend spending basically zero time running toy experiments.
With deep learning, it’s often the case that improvements on toy examples don’t scale to being improvements on real examples. In my experience, lots of papers in reinforcement learning don’t actually work because the authors only tried out the method on toy examples. (Or, they tried out the method on more complex examples, but they didn’t publish those experiments because the method didn’t work.) So trying out a new empirical method on a toy example provides little information about how valuable the empirical method will be on real examples.
The flipside of this warning is advice: for empirical projects, test your idea on as diverse and complex a set of tasks as is possible. The good empirical ideas are few, and extensive empirical testing is the best way a researcher can determine if their idea will stand the test of time.
When running diverse and complex experiments, it is still important to design the simplest possible experiment that will be informative, as Lawrence describes in the section “Mock or simplify difficult components.” I suggest being simple (such as Lawrence’s example of using text-davinci-003 instead of finetuning one’s own model) rather than being toy (using a tiny or hard-coded language model).
Thanks for writing this! I appreciate it and hope you share more things that you write faster without totally polishing everything.
One word of caution I’d share is: beware of spending too much effort running experiments on toy examples. I think toy examples are useful to gain conceptual clarity. However, if your idea is primarily empirical (such as an improvement to a deep neural network architecture), then I would recommend spending basically zero time running toy experiments.
With deep learning, it’s often the case that improvements on toy examples don’t scale to being improvements on real examples. In my experience, lots of papers in reinforcement learning don’t actually work because the authors only tried out the method on toy examples. (Or, they tried out the method on more complex examples, but they didn’t publish those experiments because the method didn’t work.) So trying out a new empirical method on a toy example provides little information about how valuable the empirical method will be on real examples.
The flipside of this warning is advice: for empirical projects, test your idea on as diverse and complex a set of tasks as is possible. The good empirical ideas are few, and extensive empirical testing is the best way a researcher can determine if their idea will stand the test of time.
When running diverse and complex experiments, it is still important to design the simplest possible experiment that will be informative, as Lawrence describes in the section “Mock or simplify difficult components.” I suggest being simple (such as Lawrence’s example of using
text-davinci-003instead of finetuning one’s own model) rather than being toy (using a tiny or hard-coded language model).I think this is a good word of caution. I’ll edit in a link to this comment.