Sorry, but I don’t think AGI development as a sequence of Bernoulli trials with constant probability is anything like a remotely sane model, and any conclusions drawn from it are worthless.
For one, it implies that after any “trial” after which AGI is not developed, we are in exactly the same state for future development of AGI as after any other “trial”. Even excluding all the ridiculous post-hoc dickering over what constitutes a “trial”, this is obvious nonsense.
Suppose that you were told by a trusted oracle that this model was absolutely correct and p=0.5. What would this actually mean for your understanding of reality?
For one, it implies that after any “trial” after which AGI is not developed, we are in exactly the same state for future development of AGI as after any other “trial”. Even excluding all the ridiculous post-hoc dickering over what constitutes a “trial”, this is obvious nonsense.
That obvious nonsense is a common feature of many models of technological development or combinatorial innovation and also scientific publishing (equal-odds rule): each paper, patent, or experiment is a lottery ticket with a similar probability of success.
“Success” in that context means something very different from “success” here: it refers to any impact whatsoever beyond the mere fact of being published, patented, or developed. For example, citations in future papers, an invention producing a profit, and other such measures.
I have no problem at all in supposing that (for example) any given AI researcher will have p chance this year of publishing a paper that is cited in some threshold of papers in the future, that the same chance p will apply to a randomly selected scientist’s paper next year of doing the same thing, and also applied to a researcher 30 years ago. That is not what this post’s model is saying.
This post refers to serial attempts to produce one specific thing that we preselected. The independent Bernoulli trial model does not work for that, except over the very shortest timescales, far shorter than the span of all attempts to make a thing.
For every specific invention, in hindsight it is clear that the earliest attempts had essentially zero chance of complete success, subsequent attempts had not much more, until eventually a relatively sharp threshold was reached due to meeting increasingly better understood prerequisites, and it became nearly inevitable.
This is not at all the same as a model that has constant p per trial right from the start. The only common factor is that if you retrospectively randomly sample a time point T between the start and the eventual success, you get E[T—start date] = E[success date—T]. However this is completely uninformative from a prediction point of view: it is true no matter where you set the start date.
This post refers to serial attempts to produce one specific thing that we preselected.
‘AI’ is a problem and a goal, not a specific thing. It’s what we label the thing which turns out to do what we want it to. “AI” is whatever works. We can no more ‘preselect’ the specific thing which will be the invention of ‘AI’ than Douglas Lenat could preselect Cyc as ‘AI’. (Because that didn’t work.)
For every specific invention, in hindsight it is clear that the earliest attempts had essentially zero chance of complete success, subsequent attempts had not much more, until eventually a relatively sharp threshold was reached due to meeting increasingly better understood prerequisites, and it became nearly inevitable. This is not at all the same as a model that has constant p per trial right from the start.
No it’s not. Innovation happens all the time by brute force, accident, trial-and-error, and serendipity. Literally the first example Clancy gives from Weitzman, of Edison testing lightbulb filaments, disproves that (an example one should remember from middle school history class). There was no ‘sharp threshold’ or ‘increasingly better understood prerequisites’. If Edison had tried the carbonized cotton thread first after his proof-of-concept using a metal filament, it would have worked; if he hadn’t eventually tried cotton after literally reaching in to grab thousands of other candidate plant materials, it wouldn’t’ve, and he would’ve gone on grabbing thousands of more materials with no greater chance of success in each trial until eventually he gave up or stumbled across another rare material which would work (bamboo apparently would’ve worked).
Attempts to produce a light bulb are exactly an example of what I mean.
Inventors in Ancient Greece would not have produced an electric light bulb, at all, even if thousands of them worked for hundreds of years. Early 18th century inventors could possibly have produced one if they had reason to put in immense effort. So rather than starting at Edison, this is where we should start that story. This is about the time period where p > 0 for a “trial” for the first time.
By the early 19th century the conditions were becoming ripe for the possibility of electric lighting, and some expensive and short-lived forms of electric light bulb were already being invented (Davy’s incandescent bulb and later arc lights). The techniques and materials still weren’t reliable or cheap enough at that time, and the theory and practice of electricity was still lacking, but people were attempting the task of making a long-lived, cheap and moderately efficient bulb and making progress with increasing degrees of success.
By the later 19th century there were dozens of successful incandescent light bulb designs, with the first patent granted about 40 years before Edison. Later versions were developed that had some narrow commercial use, and Edison’s bulb was an economically better incremental improvement that was sufficiently cheaper and longer-lived to go into mass production nearly 100 years after the first research prototype for incandescent lighting was developed.
I stated that the independent Bernoulli trial model does not work, except over the very shortest timescales, far shorter than the span of all attempts to make a thing. Edison’s attempts were indeed far shorter than the span of all attempts to make the thing.
The original post is talking about the span of all attempts to make the thing, and the constant-p model is not at all reasonable for that.
To make sure I’m understanding you correctly, do you think the largest problem comes from (1) thinking of AGI development as a sequence of Bernoulli trials, or (2) each Bernoulli trial having constant probability, or (3) both?
It’s not obvious to me that (1) is hugely problematic—isn’t Laplace’s rule of succession commonly applied to forecasting previously unseen events? Are you perhaps arguing that there’s something particular to AGI development such that thinking of it as a series of Bernoulli trials is completely invalid?
I’m more sympathetic to your criticism of (2), but I’ll note that Davidson actually relaxes this assumption in his model extensions, and further argues (in appendix 12) that the effect of the assumption is actually pretty small—most of the load of the model is carried by the first-trial probability and the reference classes used to generate it.
(1) seems reasonable as a model at this level of abstraction, absent quibbles about whether some outcome really is AGI or not, instead of some degree of AGI-ness.
(2) seems utterly wrong, and I don’t think it even makes sense to talk about a “first trial” as being a clear-cut thing, let alone having a sensible probability, and definitely not as something related to success of all future trials. I contend that it is not even “semi-informative”, it is useless.
Sorry, but I don’t think AGI development as a sequence of Bernoulli trials with constant probability is anything like a remotely sane model, and any conclusions drawn from it are worthless.
For one, it implies that after any “trial” after which AGI is not developed, we are in exactly the same state for future development of AGI as after any other “trial”. Even excluding all the ridiculous post-hoc dickering over what constitutes a “trial”, this is obvious nonsense.
Suppose that you were told by a trusted oracle that this model was absolutely correct and p=0.5. What would this actually mean for your understanding of reality?
That obvious nonsense is a common feature of many models of technological development or combinatorial innovation and also scientific publishing (equal-odds rule): each paper, patent, or experiment is a lottery ticket with a similar probability of success.
“Success” in that context means something very different from “success” here: it refers to any impact whatsoever beyond the mere fact of being published, patented, or developed. For example, citations in future papers, an invention producing a profit, and other such measures.
I have no problem at all in supposing that (for example) any given AI researcher will have p chance this year of publishing a paper that is cited in some threshold of papers in the future, that the same chance p will apply to a randomly selected scientist’s paper next year of doing the same thing, and also applied to a researcher 30 years ago. That is not what this post’s model is saying.
This post refers to serial attempts to produce one specific thing that we preselected. The independent Bernoulli trial model does not work for that, except over the very shortest timescales, far shorter than the span of all attempts to make a thing.
For every specific invention, in hindsight it is clear that the earliest attempts had essentially zero chance of complete success, subsequent attempts had not much more, until eventually a relatively sharp threshold was reached due to meeting increasingly better understood prerequisites, and it became nearly inevitable.
This is not at all the same as a model that has constant p per trial right from the start. The only common factor is that if you retrospectively randomly sample a time point T between the start and the eventual success, you get E[T—start date] = E[success date—T]. However this is completely uninformative from a prediction point of view: it is true no matter where you set the start date.
‘AI’ is a problem and a goal, not a specific thing. It’s what we label the thing which turns out to do what we want it to. “AI” is whatever works. We can no more ‘preselect’ the specific thing which will be the invention of ‘AI’ than Douglas Lenat could preselect Cyc as ‘AI’. (Because that didn’t work.)
No it’s not. Innovation happens all the time by brute force, accident, trial-and-error, and serendipity. Literally the first example Clancy gives from Weitzman, of Edison testing lightbulb filaments, disproves that (an example one should remember from middle school history class). There was no ‘sharp threshold’ or ‘increasingly better understood prerequisites’. If Edison had tried the carbonized cotton thread first after his proof-of-concept using a metal filament, it would have worked; if he hadn’t eventually tried cotton after literally reaching in to grab thousands of other candidate plant materials, it wouldn’t’ve, and he would’ve gone on grabbing thousands of more materials with no greater chance of success in each trial until eventually he gave up or stumbled across another rare material which would work (bamboo apparently would’ve worked).
Attempts to produce a light bulb are exactly an example of what I mean.
Inventors in Ancient Greece would not have produced an electric light bulb, at all, even if thousands of them worked for hundreds of years. Early 18th century inventors could possibly have produced one if they had reason to put in immense effort. So rather than starting at Edison, this is where we should start that story. This is about the time period where p > 0 for a “trial” for the first time.
By the early 19th century the conditions were becoming ripe for the possibility of electric lighting, and some expensive and short-lived forms of electric light bulb were already being invented (Davy’s incandescent bulb and later arc lights). The techniques and materials still weren’t reliable or cheap enough at that time, and the theory and practice of electricity was still lacking, but people were attempting the task of making a long-lived, cheap and moderately efficient bulb and making progress with increasing degrees of success.
By the later 19th century there were dozens of successful incandescent light bulb designs, with the first patent granted about 40 years before Edison. Later versions were developed that had some narrow commercial use, and Edison’s bulb was an economically better incremental improvement that was sufficiently cheaper and longer-lived to go into mass production nearly 100 years after the first research prototype for incandescent lighting was developed.
I stated that the independent Bernoulli trial model does not work, except over the very shortest timescales, far shorter than the span of all attempts to make a thing. Edison’s attempts were indeed far shorter than the span of all attempts to make the thing.
The original post is talking about the span of all attempts to make the thing, and the constant-p model is not at all reasonable for that.
To make sure I’m understanding you correctly, do you think the largest problem comes from (1) thinking of AGI development as a sequence of Bernoulli trials, or (2) each Bernoulli trial having constant probability, or (3) both?
It’s not obvious to me that (1) is hugely problematic—isn’t Laplace’s rule of succession commonly applied to forecasting previously unseen events? Are you perhaps arguing that there’s something particular to AGI development such that thinking of it as a series of Bernoulli trials is completely invalid?
I’m more sympathetic to your criticism of (2), but I’ll note that Davidson actually relaxes this assumption in his model extensions, and further argues (in appendix 12) that the effect of the assumption is actually pretty small—most of the load of the model is carried by the first-trial probability and the reference classes used to generate it.
(1) seems reasonable as a model at this level of abstraction, absent quibbles about whether some outcome really is AGI or not, instead of some degree of AGI-ness.
(2) seems utterly wrong, and I don’t think it even makes sense to talk about a “first trial” as being a clear-cut thing, let alone having a sensible probability, and definitely not as something related to success of all future trials. I contend that it is not even “semi-informative”, it is useless.