Thanks for the response! I see the approaches as more complimentary. Again, I think this is in keeping with standard/good ML practice.
A prototypical ML paper might first describe a motivating intuition, then formalize it via a formal model and demonstrate the intuition in that model (empirically or theoretically), then finally show the effect on real data.
The problem with only doing the real data (i.e. at scale) experiments is that it can be hard to isolate the phenomena you wish to study. And so a positive result does less to confirm the motivating intuition, as there are many other factors as play that might be responsible. We’ve seen this happen rather a lot in Deep Learning and Deep RL, in part because of the focus on empirical performance over a more scientific approach.
Thanks for the response!
I see the approaches as more complimentary.
Again, I think this is in keeping with standard/good ML practice.
A prototypical ML paper might first describe a motivating intuition, then formalize it via a formal model and demonstrate the intuition in that model (empirically or theoretically), then finally show the effect on real data.
The problem with only doing the real data (i.e. at scale) experiments is that it can be hard to isolate the phenomena you wish to study. And so a positive result does less to confirm the motivating intuition, as there are many other factors as play that might be responsible. We’ve seen this happen rather a lot in Deep Learning and Deep RL, in part because of the focus on empirical performance over a more scientific approach.