Reasonable interpretations of Recursive Self Improvement are either trivial, tautological or false?
(Trivial) AIs will do RSI by using more hardware—trivial form of RSI
(Tautological) Humans engage in a form of (R)SI when they engage in meta-cognition. i.e. therapy is plausibly a form of metacognition. Meta-cognition is plausible one of the remaining hallmarks of true general intelligence. See Vanessa Kosoy’s “Meta-Cognitive Agents”. In this view, AGIs will naturally engage in meta-cognition because they’re generally intelligent. They may (or may) not also engage in significantly more metacognition than humans but this isn’t qualitatively different from what the human cortical algorithm already engages in.
(False) It’s plausible that in many domains learning algorithms are already near a physical optimum. Given a fixed Bayesian prior of prior information and a data-set the Bayesian posterior is precise formal sense the ideal update. In practice Bayesian updating is intractable so we typically sample from the posterior using something SGD. It is plausible that something like SGD is already close to the optimum for a given amount of compute.
SGD finds algorithms. Before the DL revolution, science studied such algorithms. Now, the algorithms become inference without as much as a second glance. With sufficient abundance of general intelligence brought about by AGI, interpretability might get a lot out of studying the circuits SGD discovers. Once understood, the algorithms could be put to more efficient use, instead of remaining implicit in neural nets and used for thinking together with all the noise that remains from the search.
I think most interpretations of RSI aren’t useful.
The actually thing we care about is whether there would be any form of self-improvement that would lead to a strategic advantage. The fact that something would “recursively” self-improve 12 times or 2 times don’t really change what we care about.
With respect to your 3 points.
1) could happen by using more hardware, but better optimization of current hardware / better architecture is the actually scary part (which could lead to the discovery of “new physics” that could enable an escape even if the sandbox was good enough for the model before a few iterations of the RSI).
2) I don’t think what you’re talking about in terms of meta-cognition is relevant to the main problem. Being able to look at your own hardware or source code is though.
3) Cf. what I said at the beginning. The actual “limit” is I believe much higher than the strategic advantage threshold.
In practice Bayesian updating is intractable so we typically sample from the posterior using something SGD. It is plausible that something like SGD is already close to the optimum for a given amount of compute.
I give this view ~20%: There’s so much more info in some datapoints (curvature, third derivative of the function, momentum, see also Empirical Bayes-like SGD, the entire past trajectory through the space) that seems so available and exploitable!
I guess this is sorta about your 3, which I disbelieve (though algorithms for tasks other than learning are also important). Currently, Bayesian inference vs SGD is a question of how much data you have (where SGD wins except for very little data). For small to medium amounts of data, even without AGI, I expect SGD to lose eventually due to better inference algorithms. For many problems I have the intuition that it’s ~always possible to improve performance with more complicated algorithms (eg sat solvers). All that together makes me expect there to be inference algorithms that scale to very large amounts of data (that aren’t going to be doing full Bayesian inference but rather some complicated approximation).
When they do (like in Vanessa’s meta-MDPs) I think it’s plausible automated architecture search is a simply an instantiation of the algorithm for general intelligence (see 2.)
I think the AI will improve (itself) via better hardware and algorithms, and it will be a slog. The AI will frequently need to do narrow tasks where the general algorithm is very inefficient.
Reasonable interpretations of Recursive Self Improvement are either trivial, tautological or false?
(Trivial) AIs will do RSI by using more hardware—trivial form of RSI
(Tautological) Humans engage in a form of (R)SI when they engage in meta-cognition. i.e. therapy is plausibly a form of metacognition. Meta-cognition is plausible one of the remaining hallmarks of true general intelligence. See Vanessa Kosoy’s “Meta-Cognitive Agents”.
In this view, AGIs will naturally engage in meta-cognition because they’re generally intelligent. They may (or may) not also engage in significantly more metacognition than humans but this isn’t qualitatively different from what the human cortical algorithm already engages in.
(False) It’s plausible that in many domains learning algorithms are already near a physical optimum. Given a fixed Bayesian prior of prior information and a data-set the Bayesian posterior is precise formal sense the ideal update. In practice Bayesian updating is intractable so we typically sample from the posterior using something SGD. It is plausible that something like SGD is already close to the optimum for a given amount of compute.
SGD finds algorithms. Before the DL revolution, science studied such algorithms. Now, the algorithms become inference without as much as a second glance. With sufficient abundance of general intelligence brought about by AGI, interpretability might get a lot out of studying the circuits SGD discovers. Once understood, the algorithms could be put to more efficient use, instead of remaining implicit in neural nets and used for thinking together with all the noise that remains from the search.
I think most interpretations of RSI aren’t useful.
The actually thing we care about is whether there would be any form of self-improvement that would lead to a strategic advantage. The fact that something would “recursively” self-improve 12 times or 2 times don’t really change what we care about.
With respect to your 3 points.
1) could happen by using more hardware, but better optimization of current hardware / better architecture is the actually scary part (which could lead to the discovery of “new physics” that could enable an escape even if the sandbox was good enough for the model before a few iterations of the RSI).
2) I don’t think what you’re talking about in terms of meta-cognition is relevant to the main problem. Being able to look at your own hardware or source code is though.
3) Cf. what I said at the beginning. The actual “limit” is I believe much higher than the strategic advantage threshold.
:insightful reaction:
I give this view ~20%: There’s so much more info in some datapoints (curvature, third derivative of the function, momentum, see also Empirical Bayes-like SGD, the entire past trajectory through the space) that seems so available and exploitable!
What about specialized algorithms for problems (e.g. planning algorithms)?
What do you mean exactly? There are definitely domains in which humans have not yet come close to optimal algorithms.
I guess this is sorta about your 3, which I disbelieve (though algorithms for tasks other than learning are also important). Currently, Bayesian inference vs SGD is a question of how much data you have (where SGD wins except for very little data). For small to medium amounts of data, even without AGI, I expect SGD to lose eventually due to better inference algorithms. For many problems I have the intuition that it’s ~always possible to improve performance with more complicated algorithms (eg sat solvers). All that together makes me expect there to be inference algorithms that scale to very large amounts of data (that aren’t going to be doing full Bayesian inference but rather some complicated approximation).
What about automated architecture search?
Architectures mostly don’t seem to matter, see 3.
When they do (like in Vanessa’s meta-MDPs) I think it’s plausible automated architecture search is a simply an instantiation of the algorithm for general intelligence (see 2.)
I think the AI will improve (itself) via better hardware and algorithms, and it will be a slog. The AI will frequently need to do narrow tasks where the general algorithm is very inefficient.
As I state in the OP I don’t feel these examples are nontrivial examples of RSI.