So, this model of a takeoff scenario makes certain assumptions about how intelligence works, and these assumptions may or may not be correct. In particular, it assumes that the initial AI systems are very far from being algorithmically optimal. We don’t know whether or not this will be the case; that is what I am trying to highlight.
The task of extracting knowledge from data is a computational task, which has a certain complexity-theoretic hardness. We don’t know what that hardness is, but there is a lower bound on how efficiently this task can be done. Similarly for all the other tasks of intelligence (such as planning, etc).
Strong recursive self-improvement (given a fixed amount of resources) is only possible if the first AI systems are very far from being algorithmically optimal at all the relevant computational tasks. This is not a given; it could be true or false. For example, while you can optimise a SAT-solver in many ways, it will at the end of the day necessarily have a worst-case exponential runtime complexity (unless P = NP).
Therefore, the question of how much more intelligent AI systems will end up being compared to humans, depends on how close the human brain algorithm is to being (roughly) Pareto-optimal for all the relevant computational tasks. We don’t know the answer to this question. Strong, sustained recursive self-improvement is only possible if our initial AGI algorithm, and the human brain, both are very far from being Pareto-optimal.
Is this the case? You could point to the discrepancy between humans and animals, and argue that this demonstrates that there are cognitive algorithms that yield vastly different results given similar amounts of resources (in terms of data and parameters). However, the argument I’ve written casts doubt on whether or not this evidence is reliable. Given that, I think the case is no longer so clear; perhaps the human neural architecture is (within some small-ish polynomial factor of being) Pareto optimal for most relevant cognitive tasks.
Now, even if the human brain algorithm is roughly optimal, AI systems will almost certainly still end up with vastly more cognitive force (because they can be run faster, and given more memory and more data). However, I think that this scenario is different in relevant ways. In particular, without (strong) recursive self-improvement, you probably don’t get an uncontrollable, exponential growth in intelligence, but rather a growth that is bottle-necked by resources which you could control (such as memory, data, CPU cycles, and etc).
In particular, it assumes that the initial AI systems are very far from being algorithmically optimal. We don’t know whether or not this will be the case; that is what I am trying to highlight.
Strong recursive self-improvement (given a fixed amount of resources) is only possible if the first AI systems are very far from being algorithmically optimal at all the relevant computational tasks.
Right. But here we are talking about a specific class of tasks (which animals cannot do at all, they can’t write computer code or a mathematical paper; so, evolutionary speaking, human are new at that, and probably not anywhere close to any equilibrium because this class of tasks is very novel on the evolutionary timescale).
Moreover, we know a lot about human performance at those tasks, and it’s abysmal, even for top humans, and for AI research as a field. For example, there has been a paper in Nature in 2000 explaining that ReLU induce semantically meaningful sparseness. The field ignored this for over a decade, then rediscovered in 2009-2011 that ReLU were great, and by 2015 ReLU became dominant. That’s typical, there are plenty of examples like that (obvious discoveries not done for decades (e.g. the ReLU discovery in question was really due in early 1970-s), then further ignored for a decade or longer after the initial publication before being rediscovered and picked up). AI could simply try various things lost in the ignored old papers for a huge boost (there is a lot of actionable, pretty strong stuff in those, not everything gets picked up, like ReLU, a lot remains published, but ignored by the field).
Anyone who has attended an AutoML conference recently knows that the field of AutoML is simply too big to deal with (too many promising methods for neural architecture search, for hyperparameter optimization, for metalearning of optimization algorithms; so we have all these things in metalearning and in “AI-generating algorithms” which can provide a really large boost over status quo if done correctly, but too difficult to fully take advantage of because of human cognitive limitations, as the whole field is just “too large a mess to think effectively about”).
So it seems that, at least, there is quite a bit of room for a large initial boost over the current human-equivalent capacity. If one starts with a level of top AI researcher at a good company today, there are plenty of fixable things simply on the level of being able to consider everything that have been written in this field of study (there is also plenty of room for improvement in terms of being able to rapidly write correct computer code and rapidly try various small-scale ideas) .
It’s still quite possible that recursive self-improvement saturates after a while (it’s difficult to predict how far this would go, how soon it would start hitting difficult-to-overcome bottlenecks). But even the strongest human AI researchers (or strongest human software engineers) we have today are very, very improvable (perhaps, they can be improved by giving them more powerful tools, so that they might still remain in the loop, at least initially; the only thing people seem to be hoping for in this sense is that Copilot-like style of interaction with AIs would allow humans to keep participating in how it goes, so that the systems will remain human-AI hybrid systems, rather than AI-only).
which animals cannot do at all, they can’t write computer code or a mathematical paper
This is not obvious to me (at least not for some senses of the word “could”). Animals cannot be motivated into attempting to solve these tasks, and they cannot study maths or programming. If they could do those things, then it is not at all clear to me that they wouldn’t be able to write code or maths papers. To make this more specific; insofar as humans rely on a capacity for general problem-solving in order to do maths and programming, it would not surprise me if many animals also have this capacity to a sufficient extent, but that it cannot be directed in the right way. Note that animals even outperform humans at some general cognitive tasks. For example, chimps have a much better short-term memory than humans.
Moreover, we know a lot about human performance at those tasks, and it’s abysmal, even for top humans, and for AI research as a field.
Abysmal, compared to what? Yes, we can see that it is abysmal compared to what would in principle be information-theoretically possible. However, this doesn’t tell us very much about whether or not it is abysmal compared to what is computationally possible.
The problem of finding the minimal complexity hypothesis for a given set of data is not computationally tractable. For Kolmogorov complexity, it is uncomputable, but even for Boolean complexity, it is at least exponentially difficult (depending a bit on how exactly the problem is formalised). This means that in order to reason effectively about large amounts of data, it is (presumably) necessary to model most of it using low-fidelity methods, and then (potentially) use various heuristics in order to determine what pieces of information deserve more attention. I would therefore expect a “saturated” AI system to also frequently miss things that look obvious in hindsight.
So it seems that, at least, there is quite a bit of room for a large initial boost over the current human-equivalent capacity.
I agree that AI systems have many clear and obvious advantages, and that e.g. simply running them at a higher clock speed will give you a clear boost regardless of what assumptions we make about the “quality” of their cognition compared to that of humans. The question I’m concerned with is whether or not a takeoff scenario is better modeled as “AI quickly bootstraps to incomprehensible, Godlike intelligence through recursive self-improvement”, or whether it is better modeled as “economic growth suddenly goes up by a lot”. All the obvious advantages of AI systems are compatible with the latter.
So, this model of a takeoff scenario makes certain assumptions about how intelligence works, and these assumptions may or may not be correct. In particular, it assumes that the initial AI systems are very far from being algorithmically optimal. We don’t know whether or not this will be the case; that is what I am trying to highlight.
The task of extracting knowledge from data is a computational task, which has a certain complexity-theoretic hardness. We don’t know what that hardness is, but there is a lower bound on how efficiently this task can be done. Similarly for all the other tasks of intelligence (such as planning, etc).
Strong recursive self-improvement (given a fixed amount of resources) is only possible if the first AI systems are very far from being algorithmically optimal at all the relevant computational tasks. This is not a given; it could be true or false. For example, while you can optimise a SAT-solver in many ways, it will at the end of the day necessarily have a worst-case exponential runtime complexity (unless P = NP).
Therefore, the question of how much more intelligent AI systems will end up being compared to humans, depends on how close the human brain algorithm is to being (roughly) Pareto-optimal for all the relevant computational tasks. We don’t know the answer to this question. Strong, sustained recursive self-improvement is only possible if our initial AGI algorithm, and the human brain, both are very far from being Pareto-optimal.
Is this the case? You could point to the discrepancy between humans and animals, and argue that this demonstrates that there are cognitive algorithms that yield vastly different results given similar amounts of resources (in terms of data and parameters). However, the argument I’ve written casts doubt on whether or not this evidence is reliable. Given that, I think the case is no longer so clear; perhaps the human neural architecture is (within some small-ish polynomial factor of being) Pareto optimal for most relevant cognitive tasks.
Now, even if the human brain algorithm is roughly optimal, AI systems will almost certainly still end up with vastly more cognitive force (because they can be run faster, and given more memory and more data). However, I think that this scenario is different in relevant ways. In particular, without (strong) recursive self-improvement, you probably don’t get an uncontrollable, exponential growth in intelligence, but rather a growth that is bottle-necked by resources which you could control (such as memory, data, CPU cycles, and etc).
Right. But here we are talking about a specific class of tasks (which animals cannot do at all, they can’t write computer code or a mathematical paper; so, evolutionary speaking, human are new at that, and probably not anywhere close to any equilibrium because this class of tasks is very novel on the evolutionary timescale).
Moreover, we know a lot about human performance at those tasks, and it’s abysmal, even for top humans, and for AI research as a field. For example, there has been a paper in Nature in 2000 explaining that ReLU induce semantically meaningful sparseness. The field ignored this for over a decade, then rediscovered in 2009-2011 that ReLU were great, and by 2015 ReLU became dominant. That’s typical, there are plenty of examples like that (obvious discoveries not done for decades (e.g. the ReLU discovery in question was really due in early 1970-s), then further ignored for a decade or longer after the initial publication before being rediscovered and picked up). AI could simply try various things lost in the ignored old papers for a huge boost (there is a lot of actionable, pretty strong stuff in those, not everything gets picked up, like ReLU, a lot remains published, but ignored by the field).
Anyone who has attended an AutoML conference recently knows that the field of AutoML is simply too big to deal with (too many promising methods for neural architecture search, for hyperparameter optimization, for metalearning of optimization algorithms; so we have all these things in metalearning and in “AI-generating algorithms” which can provide a really large boost over status quo if done correctly, but too difficult to fully take advantage of because of human cognitive limitations, as the whole field is just “too large a mess to think effectively about”).
So it seems that, at least, there is quite a bit of room for a large initial boost over the current human-equivalent capacity. If one starts with a level of top AI researcher at a good company today, there are plenty of fixable things simply on the level of being able to consider everything that have been written in this field of study (there is also plenty of room for improvement in terms of being able to rapidly write correct computer code and rapidly try various small-scale ideas) .
It’s still quite possible that recursive self-improvement saturates after a while (it’s difficult to predict how far this would go, how soon it would start hitting difficult-to-overcome bottlenecks). But even the strongest human AI researchers (or strongest human software engineers) we have today are very, very improvable (perhaps, they can be improved by giving them more powerful tools, so that they might still remain in the loop, at least initially; the only thing people seem to be hoping for in this sense is that Copilot-like style of interaction with AIs would allow humans to keep participating in how it goes, so that the systems will remain human-AI hybrid systems, rather than AI-only).
This is not obvious to me (at least not for some senses of the word “could”). Animals cannot be motivated into attempting to solve these tasks, and they cannot study maths or programming. If they could do those things, then it is not at all clear to me that they wouldn’t be able to write code or maths papers. To make this more specific; insofar as humans rely on a capacity for general problem-solving in order to do maths and programming, it would not surprise me if many animals also have this capacity to a sufficient extent, but that it cannot be directed in the right way. Note that animals even outperform humans at some general cognitive tasks. For example, chimps have a much better short-term memory than humans.
Abysmal, compared to what? Yes, we can see that it is abysmal compared to what would in principle be information-theoretically possible. However, this doesn’t tell us very much about whether or not it is abysmal compared to what is computationally possible.
The problem of finding the minimal complexity hypothesis for a given set of data is not computationally tractable. For Kolmogorov complexity, it is uncomputable, but even for Boolean complexity, it is at least exponentially difficult (depending a bit on how exactly the problem is formalised). This means that in order to reason effectively about large amounts of data, it is (presumably) necessary to model most of it using low-fidelity methods, and then (potentially) use various heuristics in order to determine what pieces of information deserve more attention. I would therefore expect a “saturated” AI system to also frequently miss things that look obvious in hindsight.
I agree that AI systems have many clear and obvious advantages, and that e.g. simply running them at a higher clock speed will give you a clear boost regardless of what assumptions we make about the “quality” of their cognition compared to that of humans. The question I’m concerned with is whether or not a takeoff scenario is better modeled as “AI quickly bootstraps to incomprehensible, Godlike intelligence through recursive self-improvement”, or whether it is better modeled as “economic growth suddenly goes up by a lot”. All the obvious advantages of AI systems are compatible with the latter.