I absolutely agree that there are usually multiple ways to do something, often one of them improves faster than current SOTA, and that the faster one often overtakes the slower improving one. I may be misunderstanding what you are taking away from the horses analogy. I don’t think this undermines my point (or at least I don’t yet see the connection).
I absolutely agree that there are usually multiple ways to do something, often one of them improves faster than current SOTA, and that the faster one often overtakes the slower improving one. I may be misunderstanding what you are taking away from the horses analogy.
The takeaway seems… really obvious to me? In fact, it seems to me that the bolded first sentence of your quote basically is the takeaway: that there are multiple ways of doing things, some of which are faster than others.
This really does seem to me to be all you need to argue for a “discontinuity”: just have your timeline play out such that faster way of doing things is discovered after the slower way, and then boom, you have a discontinuity in the rate of progress, located at precisely the point where people switch from the slower way to the faster way. The horses analogy establishes this idea perfectly well in my view, but it seems almost… unnecessary? Like, this is a really obvious point?
And so, like, the question from my perspective is, why would this not be relevant to the idea of slow versus fast takeoff? It continues to remain perfectly plausible to me that a “faster way” of improving “G” exists, and that once this “faster way” is discovered, it’s so much faster that it basically obsoletes existing methods in roughly the same way that aerospace & automobile technology obsoleted horses. You say that your model doesn’t forbid this, but from my perspective, that… really sounds like you’re just conceding the whole argument right there.
Of course, presumably you don’t think you’re conceding the argument. So what’s the remaining disagreement? Is it really just a question of what relative probabilities to assign to such scenarios versus alternative scenarios, i.e. what counts as “plausible”? I’ll be honest and say that, unless your probabilities on fast-takeoff-style scenarios are really low, this seems like a pretty pointless line of disagreement to take; conversely, if your probabilities are that low, that seems to me like it’d require positive knowledge of what future AI development will look like, in a low-level, detailed way that I’m pretty sure is not justified by looking at GDP curves or the like. (Also, this comment from you seems to make it pretty clear that your probabilities are not that low.)
I remain confused, even after reading pretty much everything you’ve publicly written on this topic.
Note that I say this multiple times in the dialog and I agree it’s obvious. It also happens all the time in continuous trajectories, so if you think it should lead to discontinuities with high probability it seems like you have a lot of retrodicting-reality to do.
There are ways of doing things that improve faster, but usually they start off worse. Then they get better, and at some point they overtake the slower-improving alternatives, after which subsequent progress is faster at least for a while.
Sometimes there are exceptions where once something is possible it is necessarily much better than the predecessors (e.g. if there is a fixed cost equal to a significant percentage of GDP, and you can’t trade off fixed costs vs marginal costs). But this doesn’t happen very much, which isn’t so surprising on paper.
I don’t think any of this leads to a fast takeoff in theory. Also the view “>30% of progress is stuff from left field that jumps over the competition” doesn’t seem at all plausible to me.
in roughly the same way that aerospace & automobile technology obsoleted horses.
I’m totally happy with some better form of future AI being like automobiles, and indeed I think it’s extremely likely that AI broadly will replace humans in an automobile-like way. It just seems to me like automobiles obsoleted horses slowly, with lots of crappy automobiles well before usable automobiles. (I don’t know much about this case in particular so open to correction, but it seems like common sense / folklore; e.g. see wikipedia.)
There are ways of doing things that improve faster, but usually they start off worse. Then they get better, and at some point they overtake the slower-improving alternatives, after which subsequent progress is faster at least for a while.
I don’t see how the faster-improving technology starting off worse doesn’t simply strengthen the case for fast takeoff. While it’s worse, fewer resources will be invested into it relative to the current best thing, which leaves more room for rapid improvement once a paradigm shift finally occurs and people start switching resources over.
This seems to basically be what happened with automobiles v horses; yes if you specifically look at a Wikipedia article titled “history of the automobile” you will find crappy historical precedents to the (modern) automobile, but the point is precisely that those crappy precedents received comparatively little attention, and therefore did not obsolete horses, until suddenly they became not-so-crappy and… well, did.
I’m not exactly sure what the line of reasoning is here that leads you to look at the existence of crappy historical precedents and conclude, “oh, I guess the fact that these existed means progress wasn’t so discontinuous after all!” instead of, “hm, these crappy historical precedents existed and did basically nothing for a long time, which certainly implies a discontinuity at the point where they suddenly took over”; but from the perspective of someone invested in (what gwern described as) the “horse economy”, the latter would probably be a much more relevant takeaway than the former.
Sometimes there are exceptions where once something is possible it is necessarily much better than the predecessors (e.g. if there is a fixed cost equal to a significant percentage of GDP, and you can’t trade off fixed costs vs marginal costs). But this doesn’t happen very much, which isn’t so surprising on paper.
I don’t think any of this leads to a fast takeoff in theory. Also the view “>30% of progress is stuff from left field that jumps over the competition” doesn’t seem at all plausible to me.
I don’t disagree with any of what you say here; it just doesn’t seem very relevant to me. As you say, something new coming from left field and jumping over the competition is a rare occurrence; certainly not anywhere near 30%. The problem is that the impact distribution of new technologies is heavy-tailed, meaning that you don’t need a >30% proportion of new technologies that do the whole “obsoleting” thing, to get a hugelyoutsized impact from the few that do. Like, it seems to me that the quoted argument could have been made almost word-for-word by someone invested in the “horse economy” in the late 1800s, and it would have nonetheless done nothing to prevent them from being blindsided by the automobile economy.
Which brings me back to the point about needing positive knowledge about the new technology in question, if you want to have any hope of not being blindsided. Without positive knowledge, you’re reduced to guessing based on base rates, which again puts you in the position of the horse investor. Fundamentally I don’t see an escape to this: you can’t draw conclusions about the “physics” of new technologies by looking at GDP curves on graphs; those curves don’t (and can’t) reflect phenomena that haven’t been discovered yet.
How about discontinuities like inventing algorithms? I think often performance on a task gets jumps of O(n^k) to O(n^k’) with k’ < k, or even O(k^n) to O(k’^n). I’d guess you’d say that these sorts of jumps would smooth out by being aggregrated. But I guess I don’t see why you think that the level at which jumps from algorithmic invention happen, is enough “lower” than the level at which meaningful progress towards TAI happens, for this smoothing out to happen.
(Or maybe, do you think jumps like this don’t happen (because in practice there’s intermediate bad versions of new algorithmic ideas), or don’t represent much discontinuity (because they’re invented when the task in question is in a regime where the performance is still comparable, or something), or aren’t similar to inventions of cognitive algorithms (e.g. because cognitive stuff is more like accumulating content or something)?)
What kind of example do you have in mind? Even for algorithmic problems with relatively small communities (and low $ invested) I think it’s still pretty rare to have big jumps like this (e..g measuring performance for values of n at the scale where it can be run on conventional communities). I’m thinking of domains like combinatorial optimization and constraint satisfaction, convex optimization, graph algorithms. In most cases I think you get to a pretty OK algorithm quite quickly and further progress is slow. Exceptions tend to be cases like “there was an algorithm that works well in practice but not the worst case” or “the old algorithm got an approximation ratio of 0.87 but the new one gets an approximation ratio of 0.92 and if you absolutely require 0.9 the new one is very much faster” or extremely niche problems. But my knowledge isn’t that deep.
(And maybe what I’d say quantitatively is that something like ~10% of the log-space progress on problems people care about comes from big jumps vs 90% from relatively smooth improvements, with the number getting lower for stricter notions of “people care about” and the step size for “relatively smooth improvement” being defined by how many people are working on it.)
(I’m sure you know more than I do about algorithms.)
What kind of example do you have in mind?
~10% of the log-space progress on problems people care about comes from big jumps vs 90% from relatively smooth improvements,
I’m thinking of the difference between insertion sort / bubble sort vs radix sort / merge sort.
(Knuth gives an interesting history here (Art of Programming Vol 3, section 5.5, p 383); apparently in 1890 the US census data was processed using the radix sorting algorithm running on a mechanical-electronic-human hybrid. There was an order-preserving card-stack merging machine in 1938. Then in 1945, von Neumann wrote down a merge sort, while independently Zuse wrote down an insertion sort.)
I guess we’re talking past each other because we’re answering different versions of “What is continuous in what?”. Performance on a task can be, and is, much more continuous in time than “ideas” are continuous in time, because translating ideas into performance on a task takes resources (money, work, more ideas). So I concede that what I said here:
I think often performance on a task gets jumps
was mostly incorrect, if we don’t count the part where
you get to a pretty OK algorithm quite quickly
So one question is, is TAI driven by ideas that will have a stage where they get to a pretty okay version quite quickly once the “idea” is there, or no, or what? Another question is, do you think “ideas” are discontinuous?
I absolutely agree that there are usually multiple ways to do something, often one of them improves faster than current SOTA, and that the faster one often overtakes the slower improving one. I may be misunderstanding what you are taking away from the horses analogy. I don’t think this undermines my point (or at least I don’t yet see the connection).
The takeaway seems… really obvious to me? In fact, it seems to me that the bolded first sentence of your quote basically is the takeaway: that there are multiple ways of doing things, some of which are faster than others.
This really does seem to me to be all you need to argue for a “discontinuity”: just have your timeline play out such that faster way of doing things is discovered after the slower way, and then boom, you have a discontinuity in the rate of progress, located at precisely the point where people switch from the slower way to the faster way. The horses analogy establishes this idea perfectly well in my view, but it seems almost… unnecessary? Like, this is a really obvious point?
And so, like, the question from my perspective is, why would this not be relevant to the idea of slow versus fast takeoff? It continues to remain perfectly plausible to me that a “faster way” of improving “G” exists, and that once this “faster way” is discovered, it’s so much faster that it basically obsoletes existing methods in roughly the same way that aerospace & automobile technology obsoleted horses. You say that your model doesn’t forbid this, but from my perspective, that… really sounds like you’re just conceding the whole argument right there.
Of course, presumably you don’t think you’re conceding the argument. So what’s the remaining disagreement? Is it really just a question of what relative probabilities to assign to such scenarios versus alternative scenarios, i.e. what counts as “plausible”? I’ll be honest and say that, unless your probabilities on fast-takeoff-style scenarios are really low, this seems like a pretty pointless line of disagreement to take; conversely, if your probabilities are that low, that seems to me like it’d require positive knowledge of what future AI development will look like, in a low-level, detailed way that I’m pretty sure is not justified by looking at GDP curves or the like. (Also, this comment from you seems to make it pretty clear that your probabilities are not that low.)
I remain confused, even after reading pretty much everything you’ve publicly written on this topic.
Note that I say this multiple times in the dialog and I agree it’s obvious. It also happens all the time in continuous trajectories, so if you think it should lead to discontinuities with high probability it seems like you have a lot of retrodicting-reality to do.
There are ways of doing things that improve faster, but usually they start off worse. Then they get better, and at some point they overtake the slower-improving alternatives, after which subsequent progress is faster at least for a while.
Sometimes there are exceptions where once something is possible it is necessarily much better than the predecessors (e.g. if there is a fixed cost equal to a significant percentage of GDP, and you can’t trade off fixed costs vs marginal costs). But this doesn’t happen very much, which isn’t so surprising on paper.
I don’t think any of this leads to a fast takeoff in theory. Also the view “>30% of progress is stuff from left field that jumps over the competition” doesn’t seem at all plausible to me.
I’m totally happy with some better form of future AI being like automobiles, and indeed I think it’s extremely likely that AI broadly will replace humans in an automobile-like way. It just seems to me like automobiles obsoleted horses slowly, with lots of crappy automobiles well before usable automobiles. (I don’t know much about this case in particular so open to correction, but it seems like common sense / folklore; e.g. see wikipedia.)
I don’t see how the faster-improving technology starting off worse doesn’t simply strengthen the case for fast takeoff. While it’s worse, fewer resources will be invested into it relative to the current best thing, which leaves more room for rapid improvement once a paradigm shift finally occurs and people start switching resources over.
This seems to basically be what happened with automobiles v horses; yes if you specifically look at a Wikipedia article titled “history of the automobile” you will find crappy historical precedents to the (modern) automobile, but the point is precisely that those crappy precedents received comparatively little attention, and therefore did not obsolete horses, until suddenly they became not-so-crappy and… well, did.
I’m not exactly sure what the line of reasoning is here that leads you to look at the existence of crappy historical precedents and conclude, “oh, I guess the fact that these existed means progress wasn’t so discontinuous after all!” instead of, “hm, these crappy historical precedents existed and did basically nothing for a long time, which certainly implies a discontinuity at the point where they suddenly took over”; but from the perspective of someone invested in (what gwern described as) the “horse economy”, the latter would probably be a much more relevant takeaway than the former.
I don’t disagree with any of what you say here; it just doesn’t seem very relevant to me. As you say, something new coming from left field and jumping over the competition is a rare occurrence; certainly not anywhere near 30%. The problem is that the impact distribution of new technologies is heavy-tailed, meaning that you don’t need a >30% proportion of new technologies that do the whole “obsoleting” thing, to get a hugely outsized impact from the few that do. Like, it seems to me that the quoted argument could have been made almost word-for-word by someone invested in the “horse economy” in the late 1800s, and it would have nonetheless done nothing to prevent them from being blindsided by the automobile economy.
Which brings me back to the point about needing positive knowledge about the new technology in question, if you want to have any hope of not being blindsided. Without positive knowledge, you’re reduced to guessing based on base rates, which again puts you in the position of the horse investor. Fundamentally I don’t see an escape to this: you can’t draw conclusions about the “physics” of new technologies by looking at GDP curves on graphs; those curves don’t (and can’t) reflect phenomena that haven’t been discovered yet.
How about discontinuities like inventing algorithms? I think often performance on a task gets jumps of O(n^k) to O(n^k’) with k’ < k, or even O(k^n) to O(k’^n). I’d guess you’d say that these sorts of jumps would smooth out by being aggregrated. But I guess I don’t see why you think that the level at which jumps from algorithmic invention happen, is enough “lower” than the level at which meaningful progress towards TAI happens, for this smoothing out to happen.
(Or maybe, do you think jumps like this don’t happen (because in practice there’s intermediate bad versions of new algorithmic ideas), or don’t represent much discontinuity (because they’re invented when the task in question is in a regime where the performance is still comparable, or something), or aren’t similar to inventions of cognitive algorithms (e.g. because cognitive stuff is more like accumulating content or something)?)
What kind of example do you have in mind? Even for algorithmic problems with relatively small communities (and low $ invested) I think it’s still pretty rare to have big jumps like this (e..g measuring performance for values of n at the scale where it can be run on conventional communities). I’m thinking of domains like combinatorial optimization and constraint satisfaction, convex optimization, graph algorithms. In most cases I think you get to a pretty OK algorithm quite quickly and further progress is slow. Exceptions tend to be cases like “there was an algorithm that works well in practice but not the worst case” or “the old algorithm got an approximation ratio of 0.87 but the new one gets an approximation ratio of 0.92 and if you absolutely require 0.9 the new one is very much faster” or extremely niche problems. But my knowledge isn’t that deep.
(And maybe what I’d say quantitatively is that something like ~10% of the log-space progress on problems people care about comes from big jumps vs 90% from relatively smooth improvements, with the number getting lower for stricter notions of “people care about” and the step size for “relatively smooth improvement” being defined by how many people are working on it.)
(I’m sure you know more than I do about algorithms.)
I’m thinking of the difference between insertion sort / bubble sort vs radix sort / merge sort.
(Knuth gives an interesting history here (Art of Programming Vol 3, section 5.5, p 383); apparently in 1890 the US census data was processed using the radix sorting algorithm running on a mechanical-electronic-human hybrid. There was an order-preserving card-stack merging machine in 1938. Then in 1945, von Neumann wrote down a merge sort, while independently Zuse wrote down an insertion sort.)
I guess we’re talking past each other because we’re answering different versions of “What is continuous in what?”. Performance on a task can be, and is, much more continuous in time than “ideas” are continuous in time, because translating ideas into performance on a task takes resources (money, work, more ideas). So I concede that what I said here:
was mostly incorrect, if we don’t count the part where
So one question is, is TAI driven by ideas that will have a stage where they get to a pretty okay version quite quickly once the “idea” is there, or no, or what? Another question is, do you think “ideas” are discontinuous?