The more you accelerate something, the slower and more limiting all it’s other hidden dependencies become.
So by the time we get to AGI, regular ML research will have rapidly diminishing returns (and cuda low level software or hardware optimization will also have diminishing returns), general hardware improvement will be facing the end of moore’s law, etc etc.
I don’t see why that last sentence follows from the previous sentences. In fact I don’t think it does. What if we get to AGI next year? Then returns won’t have diminished as much & there’ll be lots of overhang to exploit.
Sure - if we got to AGI next year—but for that to actually occur you’d have to exploit most of the remaining optimization slack in both high level ML and low level algorithms. Then beyond that Moore’s law is already mostly ended or nearly so depending on who you ask, and most of the easy obvious hardware arch optimizations are now behind us.
No because of the generalized version of Amdhal’s law, which I explored in “Fast Minds and Slow Computers”.
The more you accelerate something, the slower and more limiting all it’s other hidden dependencies become.
So by the time we get to AGI, regular ML research will have rapidly diminishing returns (and cuda low level software or hardware optimization will also have diminishing returns), general hardware improvement will be facing the end of moore’s law, etc etc.
I don’t see why that last sentence follows from the previous sentences. In fact I don’t think it does. What if we get to AGI next year? Then returns won’t have diminished as much & there’ll be lots of overhang to exploit.
Sure - if we got to AGI next year—but for that to actually occur you’d have to exploit most of the remaining optimization slack in both high level ML and low level algorithms. Then beyond that Moore’s law is already mostly ended or nearly so depending on who you ask, and most of the easy obvious hardware arch optimizations are now behind us.