Algorithmic improvements are, on average, roughly similar in soeed to hardware improvements. In the area I f deep nets I believe they’re on average larger, although I haven’t looked deeply enough to say this with confidence or have a ref handy. So how much you can do is a function of how far in the future you’re talking about, on two fronts. The opportunities for algorithmic improvements go far beyond the parallelization and mixture of experts methods you mention.
The opportunities for algorithmic improvements go far beyond the parallelization and mixture of experts methods you mention.
I agree. I’d be very interested in anyone’s forecasts for how they might evolve.
I’ve been working with (very roughly) another ~10x or so improvement in “inference efficiency” by 2030 (or how to measure this and make sure it’s independent from other factors).
By this I mean that if we were able to train a model with 10^26 FLOP this year, achieving a fixed level of learning efficiency, it would require 10X FLOP to generate useful output, while by 2030 it would only require X FLOP to get the same output.
Algorithmic improvements are, on average, roughly similar in soeed to hardware improvements. In the area I f deep nets I believe they’re on average larger, although I haven’t looked deeply enough to say this with confidence or have a ref handy. So how much you can do is a function of how far in the future you’re talking about, on two fronts. The opportunities for algorithmic improvements go far beyond the parallelization and mixture of experts methods you mention.
I agree. I’d be very interested in anyone’s forecasts for how they might evolve.
I’ve been working with (very roughly) another ~10x or so improvement in “inference efficiency” by 2030 (or how to measure this and make sure it’s independent from other factors).
By this I mean that if we were able to train a model with 10^26 FLOP this year, achieving a fixed level of learning efficiency, it would require 10X FLOP to generate useful output, while by 2030 it would only require X FLOP to get the same output.