I think Eliezer’s original analogy (which may or may not be right, but is a fun thing to think about mathematically) was more like “compound interest folded on itself”. Imagine you’re a researcher making progress at a fixed rate, improving computers by 10% per year. That’s modeled well by compound interest, since every year there’s a larger number to increase by 10%, and it gives your ordinary exponential curve. But now make an extra twist: imagine the computing advances are speeding up your research as well, maybe because your mind is running on a computer, or because of some less exotic effects. So the first 10% improvement happens after a year, the next after 11 months, and so on. This may not be obvious, but it changes the picture qualitatively: it gives not just a faster exponential, but a curve which has a vertical asymptote, going to infinity in finite time. The reason is that the descending geometrical progression—a year, plus 11 months, and so on—adds up to a finite amount of time, in the same way that 1+1/2+1/4… adds up to a finite amount.
Of course there’s no infinity in real life, but the point is that a situation where research makes research faster could be even more unstable (“gradual and then sudden”) than ordinary compound interest, which we already have trouble understanding intuitively.
I think Eliezer’s original analogy (which may or may not be right, but is a fun thing to think about mathematically) was more like “compound interest folded on itself”. Imagine you’re a researcher making progress at a fixed rate, improving computers by 10% per year. That’s modeled well by compound interest, since every year there’s a larger number to increase by 10%, and it gives your ordinary exponential curve. But now make an extra twist: imagine the computing advances are speeding up your research as well, maybe because your mind is running on a computer, or because of some less exotic effects. So the first 10% improvement happens after a year, the next after 11 months, and so on. This may not be obvious, but it changes the picture qualitatively: it gives not just a faster exponential, but a curve which has a vertical asymptote, going to infinity in finite time. The reason is that the descending geometrical progression—a year, plus 11 months, and so on—adds up to a finite amount of time, in the same way that 1+1/2+1/4… adds up to a finite amount.
Of course there’s no infinity in real life, but the point is that a situation where research makes research faster could be even more unstable (“gradual and then sudden”) than ordinary compound interest, which we already have trouble understanding intuitively.