I enjoyed the discussion. My own take is that this view is likely wrong.
The “many ways to train that aren’t widely used” is evidence for alternatives which could substitute for a certain amount of hardware growth, but I don’t see it as evidence that hardware doesn’t drive growth.
My impression is that alternatives to grid search aren’t very popular because alternatives don’t really work reliably. Maybe this has changed and people haven’t picked up on it yet. Or maybe alternatives take more effort than they’re worth.
The fact that these things are fairly well known and still not used suggests that it is cheaper to pick up more compute rather than use them. You discuss these things as evidence that computing power is abundant. I’m not sure how to quantify that. It seems like you mean for “computing power is abundant” to be an argument against “computing power drives progress”.
“computing power is abundant” could mean that everyone can run whatever crazy idea they want, but the hard part is specifying something which does something interesting. This is quite relative, though. Computing power is certainly abundant compared to 20 years ago. But, the fact that people pay a lot for computing power to run large experiments means that it could be even more abundant than it is now. And, we can certainly write down interesting things which we can’t run, and which would produce more intelligent behavior if only we could.
“computing power is abundant” could mean that buying more computing power is cheaper in comparison to a lot of low-hanging-fruit optimization of what you’re running. This seems like what you’re providing evidence for (on my interpretation—I’m not imagining this is what you intend to be providing evidence for). This to me sounds like an argument that computing power drives progress: when people want to purchase capability progress, they often purchase computing power.
I do think that your observations suggest that computing power can be replaced by engineering, at least to a certain extent. So, slower progress on faster/cheaper computers doesn’t mean correspondingly slower AI progress; only somewhat slower.
I enjoyed the discussion. My own take is that this view is likely wrong.
The “many ways to train that aren’t widely used” is evidence for alternatives which could substitute for a certain amount of hardware growth, but I don’t see it as evidence that hardware doesn’t drive growth.
My impression is that alternatives to grid search aren’t very popular because alternatives don’t really work reliably. Maybe this has changed and people haven’t picked up on it yet. Or maybe alternatives take more effort than they’re worth.
The fact that these things are fairly well known and still not used suggests that it is cheaper to pick up more compute rather than use them. You discuss these things as evidence that computing power is abundant. I’m not sure how to quantify that. It seems like you mean for “computing power is abundant” to be an argument against “computing power drives progress”.
“computing power is abundant” could mean that everyone can run whatever crazy idea they want, but the hard part is specifying something which does something interesting. This is quite relative, though. Computing power is certainly abundant compared to 20 years ago. But, the fact that people pay a lot for computing power to run large experiments means that it could be even more abundant than it is now. And, we can certainly write down interesting things which we can’t run, and which would produce more intelligent behavior if only we could.
“computing power is abundant” could mean that buying more computing power is cheaper in comparison to a lot of low-hanging-fruit optimization of what you’re running. This seems like what you’re providing evidence for (on my interpretation—I’m not imagining this is what you intend to be providing evidence for). This to me sounds like an argument that computing power drives progress: when people want to purchase capability progress, they often purchase computing power.
I do think that your observations suggest that computing power can be replaced by engineering, at least to a certain extent. So, slower progress on faster/cheaper computers doesn’t mean correspondingly slower AI progress; only somewhat slower.