The model is this: assume that if an AI is created, it’s because one researcher, chosen at random from the pool of all researchers, has the key insight; and humanity survives if and only if that researcher is careful and takes safety seriously.
The “key insight” model seems deeply flawed. We know that the technical side of the problem involves performing inductive inference—which is a close cousin of stream compression. So, progress is very likely to look like progress with stream compression. Some low-hanging fruit—and then gradually diminishing returns. Rather like digging a big hole in the ground.
If you define an improvement of intelligence as being like optimizing a bunch of algorithms, such that you can do more, or the same, with fewer computes (approaching maximum compression) then the chances of a hard takeoff appear grim indeed.
Experience with optimizing systems such as genetics algorithms/programming suggests that the rate of improvement in performance decreases over time and becomes more and more tortuous with small improvements occurring less frequently. There may be occasional discoveries of new vistas on the fitness landscape, but these are not common as we march towards optimum compression.
Thus far nobody seems to have really been able to address this problem of a general optimizer which doesn’t run out of steam over time. To show that a hard takeoff is possible, at least in principle, it’s going to be necessary to demonstrate that you can devise an optimizer which doesn’t run out of steam, and in fact does the opposite.
How confident should we be that general AI involves solely hard work on existing problems like performing inductive inference? I agree that if there are no more Key Insights, and instead just a bunch of insights that some researcher will eventually have, then most of the gains from the proposal can’t be realized. Next steps: somehow estimate the probability that there are 0, 1, or several Key Insights remaining before general AI is “just” a matter of tons of hard research/experimentation, and estimate the gains from the 100-paper-strategy for the scenarios in which there are 0 or several Key Insights remaining.
How confident should we be that general AI involves solely hard work on existing problems like performing inductive inference?
I didn’t really claim that. There’s also the whole issue of what utility function to use—and some other things as well—tree pruning strategies, for instance. Just that inductive inference is the key technology for the technical side of the problem—the part not to do with values.
Much has been written about the link between induction and intelligence: Hutter. Mahoney. Me.
The “key insight” model seems deeply flawed. We know that the technical side of the problem involves performing inductive inference—which is a close cousin of stream compression. So, progress is very likely to look like progress with stream compression. Some low-hanging fruit—and then gradually diminishing returns. Rather like digging a big hole in the ground.
Here’s Bob Mottram making much the same point as I just made:
How confident should we be that general AI involves solely hard work on existing problems like performing inductive inference? I agree that if there are no more Key Insights, and instead just a bunch of insights that some researcher will eventually have, then most of the gains from the proposal can’t be realized. Next steps: somehow estimate the probability that there are 0, 1, or several Key Insights remaining before general AI is “just” a matter of tons of hard research/experimentation, and estimate the gains from the 100-paper-strategy for the scenarios in which there are 0 or several Key Insights remaining.
I didn’t really claim that. There’s also the whole issue of what utility function to use—and some other things as well—tree pruning strategies, for instance. Just that inductive inference is the key technology for the technical side of the problem—the part not to do with values.
Much has been written about the link between induction and intelligence: Hutter. Mahoney. Me.