One important question is: how much do existing AI systems help with research and the development of new, more capable AI systems?
If you make it more of a productivity question like this, then you can ask how people actually do their work, and how many man hours it takes to do that work to get the same level of performance.
By that metric, things have improved a lot.
But if the metric is “effort to halve errors from existing levels”, you don’t see the same improvement.
Do you want improved algorithm effectiveness to count as “current AI systems”?
One important question is: how much do existing AI systems help with research and the development of new, more capable AI systems?
I think a lot. The increased computer power and system tools allow hyperparameter spaces to be searched automatically. A lot of brute force becomes practical, and the available applications support it. Also, consider the application support for import of data, and generation of new data.
So I’d like to be able to measure the share of AI R&D done by computers vs humans.
It’s kind of like asking how much of your driving is done by the tires, and how much by the transmission. Your out of luck if you don’t have both.
what we should expect progress to look like going forward.
If the goal is forecasting progress going forward, I’d recommend thinking of the problem in those terms and flowing from there instead of thinking of going directly to sub forecasts before you’ve clearly related the meaning of those sub forecasts to the main goal.
If you make it more of a productivity question like this, then you can ask how people actually do their work, and how many man hours it takes to do that work to get the same level of performance.
By that metric, things have improved a lot.
But if the metric is “effort to halve errors from existing levels”, you don’t see the same improvement.
Do you want improved algorithm effectiveness to count as “current AI systems”?
I think a lot. The increased computer power and system tools allow hyperparameter spaces to be searched automatically. A lot of brute force becomes practical, and the available applications support it. Also, consider the application support for import of data, and generation of new data.
It’s kind of like asking how much of your driving is done by the tires, and how much by the transmission. Your out of luck if you don’t have both.
If the goal is forecasting progress going forward, I’d recommend thinking of the problem in those terms and flowing from there instead of thinking of going directly to sub forecasts before you’ve clearly related the meaning of those sub forecasts to the main goal.