I like this idea. I’d guess that a real economist would phrase this problem as trying to measure productivity. This isn’t particularly useful though. Productivity is output (AI research) value over input (time), so this begs the question of how to measure the output half. (I mention it mainly just in case it’s a useful search term.)
I’m no economist, but I do have an idea for measuring the output. It’s very much a hacky KISS approach, but might suffice. I’d try and poll various researchers, and ask them to estimate how much longer it would take them to do their work by slide-rule. You could ask older generations of researchers the same thing about past work. You could also ask how much faster their work would have been if they could have done it on modern computers.
It would also probably be useful to know what fraction of researcher’s time is spent using a computer. Ideally you would know how much time was spent running AI-specific programs, versus things like typing notes/reports into Microsoft Word. (which could clearly be done on a typewriter or by hand.) Programs like RescueTime could monitor this going forward, but you’d have to rely on anecdotal data to get a historical trend line. However, anecdote is probably good enough to get an order-of-magnitude estimate.
You’d definitely want a control, though. People’s memories can blur together, especially over decades. Maybe find a related field for whom data actually does exist? (From renting time on old computers? There must be at least some records.) If there are old computer logs specific to AI researchers, it would be fantastic to be able to correlate something like citations/research paper or number of papers per researcher per year with computer purchases. (Did such-and-such universitys new punch-card machine actually increase productivity?) Publication rates in general are skyrocketing, and academic trends shift, so I suspect that publications is a hopelessly confounded metric on a timescale of decades, but might be able to show changes from one year to the next.
Another reason for good control group, if I remember correctly, is that productivity of industry as a whole didn’t actually improve much by computers; people just think it was. It might also be worth digging around in the Industrial-Organizational Psychology literature to see if you can find studies involving productivity that are specific to AI research, or even something more generic like Computer Science. (I did a quick search on Google Scholar, and determined that all my search terms were far too common to narrow things down the the oddly-specific target.)
As I understand it, Eliezer Yudkowski doesn’t do much coding, but mostly purely theoretical stuff. I think most of Superintelligence could have been written on a typewriter based on printed research. I also suspect that there are plenty of academic papers which could be written by hand.
However, as you point out, there are also clearly some cases where it would take much, much longer to do the same work by hand. I’d disagree that it would take infinite time, and that it can’t be done by hand, but that’s just me being pedantic and doesn’t get to the substance of the thing.
The questions that would be interesting to answer would be how much work falls into the first category and how much falls into the second. We might think of this as a continuum, ranging from 0 productivity gain from computers, to trillions of times more efficient. What sub-fields would and wouldn’t be possible without today’s computers? What types of AI research is enabled simply by faster computers, and which types are enabled by using existing AI?
Maybe I can type at 50 words a minute, but I sure as hell can’t code at 50 WPM. Including debugging time, I can write a line of code every couple minutes, if I’m lucky. Looking back on the past 2 things I wrote, one was ~50 lines of code and took me at least an hour or two if I recall, and the other was ~200 lines and took probably a day or two of solid work. I’m just starting to learn a new language, so I’m slower than in more familiar languages, but the point stands. This hints that, for me at least, the computer isn’t the limiting factor. It might take a little longer if I was using punch cards, and at worst maybe twice as long if I was drafting everything by hand, but the computer isn’t a huge productivity booster.
Maybe there’s an AI researcher out there who spends most of his or her day trying different machine learning algorithms to try and improve them. Even if not, It’d still take forever to crunch that type of algorithm by hand. It’d be a safe bet that anyone who spends a lot of time waiting for code to compile, or who rents time on a supercomputer, is doing work where the computer is the limiting factor. It seems valuable to know which areas might grow exponentially alongside Moore’s law, and which might grow based on AI improvements, as OP pointed out.
No, I don’t think I would be amazed. But do tell: how would you do AI by hand? The Chinese Room is fine as a thought experiment, but try implementing it in reality...
I like this idea. I’d guess that a real economist would phrase this problem as trying to measure productivity. This isn’t particularly useful though. Productivity is output (AI research) value over input (time), so this begs the question of how to measure the output half. (I mention it mainly just in case it’s a useful search term.)
I’m no economist, but I do have an idea for measuring the output. It’s very much a hacky KISS approach, but might suffice. I’d try and poll various researchers, and ask them to estimate how much longer it would take them to do their work by slide-rule. You could ask older generations of researchers the same thing about past work. You could also ask how much faster their work would have been if they could have done it on modern computers.
It would also probably be useful to know what fraction of researcher’s time is spent using a computer. Ideally you would know how much time was spent running AI-specific programs, versus things like typing notes/reports into Microsoft Word. (which could clearly be done on a typewriter or by hand.) Programs like RescueTime could monitor this going forward, but you’d have to rely on anecdotal data to get a historical trend line. However, anecdote is probably good enough to get an order-of-magnitude estimate.
You’d definitely want a control, though. People’s memories can blur together, especially over decades. Maybe find a related field for whom data actually does exist? (From renting time on old computers? There must be at least some records.) If there are old computer logs specific to AI researchers, it would be fantastic to be able to correlate something like citations/research paper or number of papers per researcher per year with computer purchases. (Did such-and-such universitys new punch-card machine actually increase productivity?) Publication rates in general are skyrocketing, and academic trends shift, so I suspect that publications is a hopelessly confounded metric on a timescale of decades, but might be able to show changes from one year to the next.
Another reason for good control group, if I remember correctly, is that productivity of industry as a whole didn’t actually improve much by computers; people just think it was. It might also be worth digging around in the Industrial-Organizational Psychology literature to see if you can find studies involving productivity that are specific to AI research, or even something more generic like Computer Science. (I did a quick search on Google Scholar, and determined that all my search terms were far too common to narrow things down the the oddly-specific target.)
The answer would be “infinity”—you can’t do AI work by slide-rule. What next?
As I understand it, Eliezer Yudkowski doesn’t do much coding, but mostly purely theoretical stuff. I think most of Superintelligence could have been written on a typewriter based on printed research. I also suspect that there are plenty of academic papers which could be written by hand.
However, as you point out, there are also clearly some cases where it would take much, much longer to do the same work by hand. I’d disagree that it would take infinite time, and that it can’t be done by hand, but that’s just me being pedantic and doesn’t get to the substance of the thing.
The questions that would be interesting to answer would be how much work falls into the first category and how much falls into the second. We might think of this as a continuum, ranging from 0 productivity gain from computers, to trillions of times more efficient. What sub-fields would and wouldn’t be possible without today’s computers? What types of AI research is enabled simply by faster computers, and which types are enabled by using existing AI?
Maybe I can type at 50 words a minute, but I sure as hell can’t code at 50 WPM. Including debugging time, I can write a line of code every couple minutes, if I’m lucky. Looking back on the past 2 things I wrote, one was ~50 lines of code and took me at least an hour or two if I recall, and the other was ~200 lines and took probably a day or two of solid work. I’m just starting to learn a new language, so I’m slower than in more familiar languages, but the point stands. This hints that, for me at least, the computer isn’t the limiting factor. It might take a little longer if I was using punch cards, and at worst maybe twice as long if I was drafting everything by hand, but the computer isn’t a huge productivity booster.
Maybe there’s an AI researcher out there who spends most of his or her day trying different machine learning algorithms to try and improve them. Even if not, It’d still take forever to crunch that type of algorithm by hand. It’d be a safe bet that anyone who spends a lot of time waiting for code to compile, or who rents time on a supercomputer, is doing work where the computer is the limiting factor. It seems valuable to know which areas might grow exponentially alongside Moore’s law, and which might grow based on AI improvements, as OP pointed out.
You’d be amazed what people can do “by hand”. Keep in mind, computer was originally an occupation.
No, I don’t think I would be amazed. But do tell: how would you do AI by hand? The Chinese Room is fine as a thought experiment, but try implementing it in reality...