The algorithms that are used nowadays are basically the same as the algorithms that were known then, just with a bunch of tricks like dropout.
Suppose that you have 100 ideas that seem like they might work. You test them, and one of them does work. You then find a mathematical reason why it works. Is this insight of compute?
Even if most of the improvement is in compute, there could be much better algorithms that we just aren’t finding. I would be unsurprised if there exists an algorithm that would be really scary on vacuum tubes.
The algorithms that are used nowadays are basically the same as the algorithms that were known then, just with a bunch of tricks like dropout.
Suppose that you have 100 ideas that seem like they might work. You test them, and one of them does work. You then find a mathematical reason why it works. Is this insight of compute?
Even if most of the improvement is in compute, there could be much better algorithms that we just aren’t finding. I would be unsurprised if there exists an algorithm that would be really scary on vacuum tubes.