You make a number of assumptions here and you also ignore my previous comments regarding the following point: you assume that knowledge of one’s source code permits a fundamentally more powerful kind of recursive self improvement. This is a crucial assumption on which your argument rests… if this assumption is false (and it is certainly insubstantiated) then we have no reason to believe that a GAI can do any more than what a human can do, given full knowledge of the brain. And as we know, there are some serious limitations on what we can do with the brain. Thus the concept of recursive self improvement leading to super-human intelligence is equated to (essentially) the problem of drug and surgical treatment and expansion, which has a rightfully limiting sounding ring to it.
Furthermore, your assumptions consist in (for example) the idea that such a thing as the agent you describe can possibly exist. It is all well and good in theory to talk abstractly about a system (e.g., a human) improving its intelligence to improve its intelligence, but you seem to draw a kind of arbitrary distinction between this process and the more common processes involves in human activities, like piano playing.
In the piano example, you are just incorrect to claim there is no recursive self improvement (RSI, now) going on there. Just consider the following ideas.
Specifically, you point out that pianists don’t actively seek to recursively self-improve, which is true, as it would be hopelessly convoluted and they would never learn how to actually play anything. However, you neglect to consider the passive action of recursive self-improvement which takes part in the process. This action is clear from the simple observation that an experienced pianist can learn (i.e., sight read a new piece and play it) much better than a beginner pianist. Since learning pieces is exactly what makes you a better pianist, this is an empirical evidence of recursive self-improvement. It is besides the point that this may not be the same degree as your idealized RSI, which is an arbitrary, impractical, and undemonstrated mode, as pointed out above. It is also besides the point that the pianist doesn’t actively seek out this technique. (Even if a GAI didn’t actively seek out to RSI its “intelligence”, but it still did, we would achieve the same end results. )
And as we know, there are some serious limitations on what we can do with the brain.
Yes, but imagine not only that we have complete access to the brain’s source code, but that the brain is digitally implemented and any component can be changed at whim. What could we achieve? At the very least, some very helpful things, if not superintelligence:
We already have examples of drugs and diseases that boost cognitive performance. Personally, I’ve been hyperthyroid before. The cognitive boost at the peak was very pronounced. This can’t be sustained in wetware (at the moment) for various reasons. None of those reasons, as far as I’ve seen, would matter in silicon. Sustainable hyperthyroidism via alterations to my ‘source code’ alone would make me 5 times more productive.
Once a mechanism of action is understood, it’s likely it can be increased, at least a little. For instance, nootropes (such as piracetam, huperzine a, modafinil) work via chemical pathways. It seems reasonable to expect that bypassing the chemical aspect and directly tweaking the code should provide better results. If nothing else the quality and quantity of the dose can be regularized and optimized much more efficiently. This isn’t even mentioning all the drugs that can’t cross the blood-brain barrier but which could be directly ‘injected’ into individual neurons in a simulation, which is a tiny subset of all the ‘drugs’ that could be tried from directly changing the way the neuron works. Many nootropes, too, either diminish in effect over time (for chemical reasons) or tax the body in unsupportable ways, as with hyperthyroidism: neither of these would pose a problem for a silicon brain, which could be permanently pumped up on a whole cocktail of crazy drugs and mood modifiers without worrying about the damage being done to the endocrine system or any other fragile wetware.
In short, an uploaded human with access to its source code and an understanding of neurology and biochemistry, while probably falling short of superintelligence, would have a hell of an advantage over meatspace humans, even without hardware acceleration.
you assume that knowledge of one’s source code permits a fundamentally more powerful kind of recursive self improvement.
It’s not really a difference in kind so much as a radical difference in terms of efficiency.
If asked to improve a C program, do you think a C programmer would rather have a memory dump of the running program or the memory dump and the source code for the program? The source code is a huge help in understanding and improving the program, and this translates into an ability to make improvements at a rate that is orders of magnitude greater with the source code than without. There’s no reason to expect the case to be different for programs that are AGIs than for other kinds of programs, and no reason to expect it to be different for programmers that are AGIs than for human programmers. On the contrary, I think the advantage of having and understanding the source code increases as programs get larger and more complex, and is greater for programs that were artificially designed and are modular and highly compressed versus naturally evolved programs that have lots of redundancy and are non-modular.
Several posts in this thread seem to be confusing recursive self-improvement with merely iterative self-improvement.
If a human pianist practices to get better, and then practices some more to get even better, and then practices some more to get even better than that, then that is ISI: essentially linear growth.
RSI in humans would have to involve things like rationality training and “learning how to learn”: getting better at getting better.
ISI does not go foom. RSI can. (A human RSI foom would involve neurosurgery and transhumanism.)
Thanks for trying to clear that up but again, you’re not understanding the piano example. I’m not going to repeat it again as that would just be redundant, but if you read carefully in the example, you see that there is an empirical evidence of recursive self improvement. This isn’t a matter of confusion.
The pianist may seem like they are just practicing to get better, practing some more to get even better, as you say. However, if you look at the final product (the highly experienced pianist) he isn’t just better at playing—he is also much better at learning to play better. This is RSI, even though his intentions (as you correctly say) may not be explicitly set up to achieve RSI.
I reread it (here, right?) and I don’t see anything about recursion.
Yes, a master pianist can learn a new piece faster than a novice can, but this is merely… let’s call it concentric self-improvement. The master is (0) good at playing piano, (1) good at learning to do 0, (2) good at learning to do 1, etc., for finitely many levels in a strict, non-tangled hierarchy.
This is fundamentally different-in-kind from being (0) good at playing piano, and (1) good at learning to do 0 and 1. ISI grows linearly, CSI grows polynomially (of potentially very large degree), and RSI grows superexponentially.
You make a number of assumptions here and you also ignore my previous comments regarding the following point: you assume that knowledge of one’s source code permits a fundamentally more powerful kind of recursive self improvement. This is a crucial assumption on which your argument rests… if this assumption is false (and it is certainly insubstantiated) then we have no reason to believe that a GAI can do any more than what a human can do, given full knowledge of the brain. And as we know, there are some serious limitations on what we can do with the brain. Thus the concept of recursive self improvement leading to super-human intelligence is equated to (essentially) the problem of drug and surgical treatment and expansion, which has a rightfully limiting sounding ring to it.
Furthermore, your assumptions consist in (for example) the idea that such a thing as the agent you describe can possibly exist. It is all well and good in theory to talk abstractly about a system (e.g., a human) improving its intelligence to improve its intelligence, but you seem to draw a kind of arbitrary distinction between this process and the more common processes involves in human activities, like piano playing.
In the piano example, you are just incorrect to claim there is no recursive self improvement (RSI, now) going on there. Just consider the following ideas.
Specifically, you point out that pianists don’t actively seek to recursively self-improve, which is true, as it would be hopelessly convoluted and they would never learn how to actually play anything. However, you neglect to consider the passive action of recursive self-improvement which takes part in the process. This action is clear from the simple observation that an experienced pianist can learn (i.e., sight read a new piece and play it) much better than a beginner pianist. Since learning pieces is exactly what makes you a better pianist, this is an empirical evidence of recursive self-improvement. It is besides the point that this may not be the same degree as your idealized RSI, which is an arbitrary, impractical, and undemonstrated mode, as pointed out above. It is also besides the point that the pianist doesn’t actively seek out this technique. (Even if a GAI didn’t actively seek out to RSI its “intelligence”, but it still did, we would achieve the same end results. )
Yes, but imagine not only that we have complete access to the brain’s source code, but that the brain is digitally implemented and any component can be changed at whim. What could we achieve? At the very least, some very helpful things, if not superintelligence:
We already have examples of drugs and diseases that boost cognitive performance. Personally, I’ve been hyperthyroid before. The cognitive boost at the peak was very pronounced. This can’t be sustained in wetware (at the moment) for various reasons. None of those reasons, as far as I’ve seen, would matter in silicon. Sustainable hyperthyroidism via alterations to my ‘source code’ alone would make me 5 times more productive.
Once a mechanism of action is understood, it’s likely it can be increased, at least a little. For instance, nootropes (such as piracetam, huperzine a, modafinil) work via chemical pathways. It seems reasonable to expect that bypassing the chemical aspect and directly tweaking the code should provide better results. If nothing else the quality and quantity of the dose can be regularized and optimized much more efficiently. This isn’t even mentioning all the drugs that can’t cross the blood-brain barrier but which could be directly ‘injected’ into individual neurons in a simulation, which is a tiny subset of all the ‘drugs’ that could be tried from directly changing the way the neuron works. Many nootropes, too, either diminish in effect over time (for chemical reasons) or tax the body in unsupportable ways, as with hyperthyroidism: neither of these would pose a problem for a silicon brain, which could be permanently pumped up on a whole cocktail of crazy drugs and mood modifiers without worrying about the damage being done to the endocrine system or any other fragile wetware.
In short, an uploaded human with access to its source code and an understanding of neurology and biochemistry, while probably falling short of superintelligence, would have a hell of an advantage over meatspace humans, even without hardware acceleration.
It’s not really a difference in kind so much as a radical difference in terms of efficiency.
If asked to improve a C program, do you think a C programmer would rather have a memory dump of the running program or the memory dump and the source code for the program? The source code is a huge help in understanding and improving the program, and this translates into an ability to make improvements at a rate that is orders of magnitude greater with the source code than without. There’s no reason to expect the case to be different for programs that are AGIs than for other kinds of programs, and no reason to expect it to be different for programmers that are AGIs than for human programmers. On the contrary, I think the advantage of having and understanding the source code increases as programs get larger and more complex, and is greater for programs that were artificially designed and are modular and highly compressed versus naturally evolved programs that have lots of redundancy and are non-modular.
Several posts in this thread seem to be confusing recursive self-improvement with merely iterative self-improvement.
If a human pianist practices to get better, and then practices some more to get even better, and then practices some more to get even better than that, then that is ISI: essentially linear growth.
RSI in humans would have to involve things like rationality training and “learning how to learn”: getting better at getting better.
ISI does not go foom. RSI can. (A human RSI foom would involve neurosurgery and transhumanism.)
Thanks for trying to clear that up but again, you’re not understanding the piano example. I’m not going to repeat it again as that would just be redundant, but if you read carefully in the example, you see that there is an empirical evidence of recursive self improvement. This isn’t a matter of confusion.
The pianist may seem like they are just practicing to get better, practing some more to get even better, as you say. However, if you look at the final product (the highly experienced pianist) he isn’t just better at playing—he is also much better at learning to play better. This is RSI, even though his intentions (as you correctly say) may not be explicitly set up to achieve RSI.
I reread it (here, right?) and I don’t see anything about recursion.
Yes, a master pianist can learn a new piece faster than a novice can, but this is merely… let’s call it concentric self-improvement. The master is (0) good at playing piano, (1) good at learning to do 0, (2) good at learning to do 1, etc., for finitely many levels in a strict, non-tangled hierarchy.
This is fundamentally different-in-kind from being (0) good at playing piano, and (1) good at learning to do 0 and 1. ISI grows linearly, CSI grows polynomially (of potentially very large degree), and RSI grows superexponentially.