Not at all—we already have recursive self-improvement, each new generation of computers is taking on more of the work of designing the next generation. But—as an observation of empirical fact, not just a conjecture—this does not produce foom, it produces steady exponential growth; as I explained, this is because recursive self-improvement is still subject to the curve of capability.
Whenever you have a process that requires multiple steps to complete, you can’t go any faster than the slowest step. Unless Intel’s R&D department actually does nothing but press the “make a new CPU design” button every few months, I think the limiting factor is still the step that involves unimproved human brains.
Elsewhere in this thread you talk about other bottlenecks, but as far as I know FOOM was never meant to imply unbounded speed of progress, only fast enough that humans have no hope of keeping up.
I saw a discussion somewhere with a link to a CPU design researcher discussing the level of innovation required to press that “make a new CPU design” button, and how much of the time is spent waiting for compiler results and bug testing new designs.
I’m getting confused trying to remember it off the top of my head, but here are some links to what I could dig up with a quick search.
Anyway I haven’t reread the whole discussion but the weak inside view seems to be that speeding up humans would not be quite as big of a gap as the outside view predicts—of course with a half dozen caveats that mean a FOOM could still happen. soreff’s comment below also seems related.
As long as nontrivial human reasoning is a necessary part of the process, even if it’s a small part, the process as a whole will stay at least somewhat “anchored” to a human-tractable time scale. Progress can’t speed up past human comprehension as long as human comprehension is required for progress. If the human bottleneck goes away there’s no guarantee that other bottlenecks will always conveniently appear in the right places to have the same effect.
I agree, but that is a factual and falsifiable claim that can be tested (albeit only in very weak ways) by looking at where current research is most bottlenecked by human comprehension.
where the bottleneck is outside the part being improved.
I work in CAD, and I can assure you that the users of my department’s code are very interested in performance improvements. The CAD software run time is a major bottleneck in designing the next generation of computers.
I work in CAD, and I can assure you that the users of my department’s code are very interested in performance improvements.
Might I suggest that the greatest improvements in CAD performance will come from R&D into CAD techniques and software? At least, that seems to be the case basically everywhere else. Not to belittle computing power but the software to use the raw power tends to be far more critical. If this wasn’t the case, and given that much of the CAD task can be parellelized, recursive self improvement in computer chip self improvement would result in an exponentially expanding number of exponentially improving supercomputer clusters. On the scale of “We’re using 30% of the chips we create to make new supercomputers instead of selling them. We’ll keep doing that until our processors are sufficiently ahead of ASUS that we can reinvest less of our production capacity and still be improving faster than all competitors”.
I think folk like yourself and the researchers into CAD are the greater bottleneck here. That’s a compliment to the importance of your work. I think. Or maybe an insult to your human frailty. I never can tell. :)
Jed Harris says similar things in the comments here, but this seems to make predictions that don’t seem born out to me (cf wedrifid). If serial runtime is a recursive bottleneck, then the break of exponentially increasing clockspeed should cause problems for the chip design process and then also break exponential transistor density. But if these processes can be parallelized, then they should have been parallelized long ago.
A way to reconcile some of these claims is that serial clockspeed has only recently become a bottleneck, as a result of the clockspeed plateau.
It is interesting to note that rw first expends effort to argue show something could kind of be considered to be recursive improvement so as to go on and show how weakly recursive it is. That’s not even ‘reference class tennis’… it’s reference class Aikido!
I’ll take that as a compliment :-) but to clarify, I’m not saying its weakly recursive. I’m saying it’s quite strongly recursive—and noting that recursion isn’t magic fairy dust, the curve of capability limits rate of progress even when you do have recursive self-improvement.
But there will always be a bottleneck outside the part being improved, if nothing else because the ultimate source of information is feedback from the real world.
(Well, that might stop being true if it turns out to be possible to Sublime into hyperspace or something like that. But it will remain true as long as we are talking about entities that exist in the physical universe.)
An AGI could FOOM and surpass us before it runs into that limit. It’s not like we are making observations anywhere near as fast as physically possible, nor are we drawing all the conclusions we ideally could from the data we do observe.
“That limit”? The mathematically ultimate limit is Solomonoff induction on an infinitely powerful computer, but that’s of no physical relevance. I’m talking about the observed bound on rates of progress, including rates of successive removal of bottlenecks. To be sure, there may—hopefully will! -- someday exist entities capable of making much better use of data than we can today; but there is no reason to believe the process of getting to that stage will be in any way discontinuous, and plenty of reason to believe it will not.
Yes, but you were the one who started talking about it as something you can “run into”, together with terms like “as fast as physically possible” and “ideally could from the data”—that last term in particular has previously been used in conversations like this to refer to Solomonoff induction on an infinitely powerful computer.
My point is that at any given moment an awful lot of things will be bottlenecks, including real-world data. The curve of capability is already observed in cases where you are free to optimize whatever variable is the lowest hanging fruit at the moment.
In other words, you are already “into” the current data limit; if you could get better performance by using less data and substituting e.g. more computation, you would already be doing it.
As time goes by, the amount of data you need for a given degree of performance will drop as you obtain more computing power, better algorithms etc. (But of course, better performance still will be obtainable by using more data.) However,
The amount of data needed won’t drop below some lower bound,
More to the point, the rate at which the amount needed drops, is itself bound by the curve of capability.
You seem to have completely glossed over the idea of recursive self-improvement.
Not at all—we already have recursive self-improvement, each new generation of computers is taking on more of the work of designing the next generation. But—as an observation of empirical fact, not just a conjecture—this does not produce foom, it produces steady exponential growth; as I explained, this is because recursive self-improvement is still subject to the curve of capability.
Whenever you have a process that requires multiple steps to complete, you can’t go any faster than the slowest step. Unless Intel’s R&D department actually does nothing but press the “make a new CPU design” button every few months, I think the limiting factor is still the step that involves unimproved human brains.
Elsewhere in this thread you talk about other bottlenecks, but as far as I know FOOM was never meant to imply unbounded speed of progress, only fast enough that humans have no hope of keeping up.
I saw a discussion somewhere with a link to a CPU design researcher discussing the level of innovation required to press that “make a new CPU design” button, and how much of the time is spent waiting for compiler results and bug testing new designs.
I’m getting confused trying to remember it off the top of my head, but here are some links to what I could dig up with a quick search.
Anyway I haven’t reread the whole discussion but the weak inside view seems to be that speeding up humans would not be quite as big of a gap as the outside view predicts—of course with a half dozen caveats that mean a FOOM could still happen. soreff’s comment below also seems related.
As long as nontrivial human reasoning is a necessary part of the process, even if it’s a small part, the process as a whole will stay at least somewhat “anchored” to a human-tractable time scale. Progress can’t speed up past human comprehension as long as human comprehension is required for progress. If the human bottleneck goes away there’s no guarantee that other bottlenecks will always conveniently appear in the right places to have the same effect.
I agree, but that is a factual and falsifiable claim that can be tested (albeit only in very weak ways) by looking at where current research is most bottlenecked by human comprehension.
No. We have only observed weak recursive self improvement, where the bottleneck is outside the part being improved.
I work in CAD, and I can assure you that the users of my department’s code are very interested in performance improvements. The CAD software run time is a major bottleneck in designing the next generation of computers.
Might I suggest that the greatest improvements in CAD performance will come from R&D into CAD techniques and software? At least, that seems to be the case basically everywhere else. Not to belittle computing power but the software to use the raw power tends to be far more critical. If this wasn’t the case, and given that much of the CAD task can be parellelized, recursive self improvement in computer chip self improvement would result in an exponentially expanding number of exponentially improving supercomputer clusters. On the scale of “We’re using 30% of the chips we create to make new supercomputers instead of selling them. We’ll keep doing that until our processors are sufficiently ahead of ASUS that we can reinvest less of our production capacity and still be improving faster than all competitors”.
I think folk like yourself and the researchers into CAD are the greater bottleneck here. That’s a compliment to the importance of your work. I think. Or maybe an insult to your human frailty. I never can tell. :)
Jed Harris says similar things in the comments here, but this seems to make predictions that don’t seem born out to me (cf wedrifid). If serial runtime is a recursive bottleneck, then the break of exponentially increasing clockspeed should cause problems for the chip design process and then also break exponential transistor density. But if these processes can be parallelized, then they should have been parallelized long ago.
A way to reconcile some of these claims is that serial clockspeed has only recently become a bottleneck, as a result of the clockspeed plateau.
It is interesting to note that rw first expends effort to argue show something could kind of be considered to be recursive improvement so as to go on and show how weakly recursive it is. That’s not even ‘reference class tennis’… it’s reference class Aikido!
I’ll take that as a compliment :-) but to clarify, I’m not saying its weakly recursive. I’m saying it’s quite strongly recursive—and noting that recursion isn’t magic fairy dust, the curve of capability limits rate of progress even when you do have recursive self-improvement.
I suppose ‘quite’ is a relative term. It’s improvement with a bottleneck that resides firmly in the human brain.
Of course it does. Which is why it matters so much how steep the curve of recursion is compared to the curve of capability. It is trivial maths.
But there will always be a bottleneck outside the part being improved, if nothing else because the ultimate source of information is feedback from the real world.
(Well, that might stop being true if it turns out to be possible to Sublime into hyperspace or something like that. But it will remain true as long as we are talking about entities that exist in the physical universe.)
An AGI could FOOM and surpass us before it runs into that limit. It’s not like we are making observations anywhere near as fast as physically possible, nor are we drawing all the conclusions we ideally could from the data we do observe.
“That limit”? The mathematically ultimate limit is Solomonoff induction on an infinitely powerful computer, but that’s of no physical relevance. I’m talking about the observed bound on rates of progress, including rates of successive removal of bottlenecks. To be sure, there may—hopefully will! -- someday exist entities capable of making much better use of data than we can today; but there is no reason to believe the process of getting to that stage will be in any way discontinuous, and plenty of reason to believe it will not.
Are you being deliberately obtuse? “That limit” refers to the thing you brought up: the rate at which observations can be made.
Yes, but you were the one who started talking about it as something you can “run into”, together with terms like “as fast as physically possible” and “ideally could from the data”—that last term in particular has previously been used in conversations like this to refer to Solomonoff induction on an infinitely powerful computer.
My point is that at any given moment an awful lot of things will be bottlenecks, including real-world data. The curve of capability is already observed in cases where you are free to optimize whatever variable is the lowest hanging fruit at the moment.
In other words, you are already “into” the current data limit; if you could get better performance by using less data and substituting e.g. more computation, you would already be doing it.
As time goes by, the amount of data you need for a given degree of performance will drop as you obtain more computing power, better algorithms etc. (But of course, better performance still will be obtainable by using more data.) However,
The amount of data needed won’t drop below some lower bound,
More to the point, the rate at which the amount needed drops, is itself bound by the curve of capability.