This just shifts the question to how you slotted FinalState into such a promising reference class? Conservatively, tens of academic research programs, tens of PhD dissertations, hundreds of hobbyist projects, hundreds of undergraduate term projects, and tens of business ventures have attempted something similar to AGI and none have succeeded.
As far as I can tell, the vast majority of academic projects (particularly those of undergrads) have worked on narrow AI, which this is supposedly not.
However, reading the post again, it doesn’t sound as though they have the support of any academic institution; I misread the bit around “academic network”. It sounds more as though this is a homebrew project, in which case I need to go two or three orders of magnitude lower.
As far as I can tell, the vast majority of academic projects (particularly those of undergrads) have worked on narrow AI, which this is supposedly not.
That’s definitely a reasonable assessment. I dialed all those estimates down by about an order of magnitude from when I started writing that point as I thought through just how unusual attempting general AI is. But over sixty years and hundreds of institutions where one might get a sufficiently solid background in CS to implement something big, there are going to be lots of unusual people trying things out.
The Rule of Succession, if I’m not mistaken, assumes a uniform prior from 0 to 1 for the probability of success. That seems unreasonable; it shouldn’t be extremely improbable (even before observing failure) that fewer than one in a thousand such claims result in a working AGI. So you have to adjust downward somewhat from there, but it’s hard to say how much.
(This is in addition to the point that user:othercriteria makes in the sibling comment.)
You’re correct, but where would I find a better prior? I’d rather be too conservative than resort to wild guessing (which it would be, since I’m not an expert on AGI).
(A variant of this is rhollerith_dot_com’s objection below, that I failed to take into account whatever the probability of working AGI leading to death is. Presumably that changes the prior as well.)
A. Many commonly used priors are listed in the Handbook of Chemistry and Physics.
Q. Where do priors originally come from?
A. Never ask that question.
Q. Uh huh. Then where do scientists get their priors?
A. Priors for scientific problems are established by annual vote of the AAAS. In recent years the vote has become fractious and controversial, with widespread acrimony, factional polarization, and several outright assassinations. This may be a front for infighting within the Bayes Council, or it may be that the disputants have too much spare time. No one is really sure.
Q. I see. And where does everyone else get their priors?
A. They download their priors from Kazaa.
Q. What if the priors I want aren’t available on Kazaa?
A. There’s a small, cluttered antique shop in a back alley of San Francisco’s Chinatown. Don’t ask about the bronze rat.
Isn’t the lesson of the Quantum Physics sequence that ordinary humans today should get their priors from the least complex (and falsifiable?) statements that aren’t inconsistent with empirical knowledge.
I don’t know where to get a good prior. I suppose you might look at past instances where someone claimed to be close to doing something that seemed about as difficult and confusing as AGI seems to be (before taking into account a history of promises that didn’t pan out, but after taking into account what we know about the confusingness of the problem, insofar as that knowledge doesn’t itself come from the fact of failed promises). I don’t know what that prior would look like, but it seems like it would assign (if you randomly selected a kind of feat) a substantially greater than 1⁄10 probability of seeing at least 10 failed predictions of achieving that feat for every successful such prediction, a substantially greater than 1⁄100 probability of seeing at least 100 failed predictions for every successful prediction, and so on.
Hmm yeah, I read the post again and, if it’s a troll, it’s a way-more-subtle-than-typical one. Still, my posterior probability assignment on him being serious/sincere is in the 0.40s (extraordinary claims require extraordinary evidence) -- though this means that the probability that he succeeds given that he’s serious is the same order of magnitude as the probability that he succeeds given everything I know.
If you know you probably would not have survived the sun’s having failed to rise, you cannot just apply the Rule of Succession to your knowledge of past sunrises to calculate the probability that the sun will rise tomorrow because that would be ignoring relevant information, namely the existence of a severe selection bias. (Sadly, I do not know how to modify the Rule of Succession to account for the selection bias.)
On the other hand, if you have so much background knowledge about the Sun that you can think about the selection effects involved, the Rule of Succession is a moot & incomplete analysis to begin with.
Regarding your second paragraph, Sir Gwern, if we switch the example to the question of whether the US and Russia will launch nukes at each other this year, I have at lot of information about the strength of the selection bias (including for example Carl Sagan’s work on nuclear winter) that I might put to good use if I knew how to account for selection effects, but I would be sorely tempted to use something like the Rule of Succession (modified to account for the selection bias and where the analog of a day in which the sun might or might not rise is the start of the part of the career of someone in the military or in politics during which he or she can influence whether or not an attempt at a first strike is made) because my causal model of the mental processes behind the decision to launch is so unsatisfactory.
This might be a good place for me to point out that I never bought into the common wisdom, which I have never seen anyone object to or distance themselves from in print, that the chances of a nuclear exchange between the US and Russia went down considerably after the collapse of the Soviet Union in 1991.
This might be a good place for me to point out that I never bought into the common wisdom, which I have never seen anyone object to or distance themselves from in print, that the chances of a nuclear exchange between the US and Russia went down considerably after the collapse of the Soviet Union in 1991.
Nuclear war isn’t the same situation, though. We can survive nuclear war at all sorts of levels of intensity, so the selection filter is not nearly the same as “the Sun going out”, which is ~100% fatal. Bostrom’s shadow paper might actually work for nuclear war, from the perspective of a revived civilization, but I’d have to reread it to see.
The selection filter does not have to be total or near total for my point to stand, namely, Rule-of-Succession-like calculations can be useful even when one has enough information to think about the selection effects involved (provided that Rule-of-Succession-like calculations are ever useful).
And parenthetically selection effects on observations about whether nuclear exchanges happened in the past can be very strong. Consider for example a family who has lived in Washington, D.C., for the last 5 decades: Washington, D.C., is such an important target that it is unlikely the family would have survived the launch of most or all of the Soviet/Russian arsenal at the U.S. So, although I agree with you that the human race as a whole would probably have survived almost any plausible nuclear exchange, that does not do the family in D.C. much good. More precisely, it does not do much good for the family’s ability to use historical data on whether or not nukes were launched at the U.S. in the past to refine their probability of launches in the future.
Laplace’s Rule of Succession, assuming around fifty failures under similar or more favorable circumstances.
This just shifts the question to how you slotted FinalState into such a promising reference class? Conservatively, tens of academic research programs, tens of PhD dissertations, hundreds of hobbyist projects, hundreds of undergraduate term projects, and tens of business ventures have attempted something similar to AGI and none have succeeded.
As far as I can tell, the vast majority of academic projects (particularly those of undergrads) have worked on narrow AI, which this is supposedly not.
However, reading the post again, it doesn’t sound as though they have the support of any academic institution; I misread the bit around “academic network”. It sounds more as though this is a homebrew project, in which case I need to go two or three orders of magnitude lower.
That’s definitely a reasonable assessment. I dialed all those estimates down by about an order of magnitude from when I started writing that point as I thought through just how unusual attempting general AI is. But over sixty years and hundreds of institutions where one might get a sufficiently solid background in CS to implement something big, there are going to be lots of unusual people trying things out.
Of those who attempted, fewer thought they were close, but fifty still seems very generous.
The Rule of Succession, if I’m not mistaken, assumes a uniform prior from 0 to 1 for the probability of success. That seems unreasonable; it shouldn’t be extremely improbable (even before observing failure) that fewer than one in a thousand such claims result in a working AGI. So you have to adjust downward somewhat from there, but it’s hard to say how much.
(This is in addition to the point that user:othercriteria makes in the sibling comment.)
You’re correct, but where would I find a better prior? I’d rather be too conservative than resort to wild guessing (which it would be, since I’m not an expert on AGI).
(A variant of this is rhollerith_dot_com’s objection below, that I failed to take into account whatever the probability of working AGI leading to death is. Presumably that changes the prior as well.)
http://yudkowsky.net/rational/bayes
Isn’t the lesson of the Quantum Physics sequence that ordinary humans today should get their priors from the least complex (and falsifiable?) statements that aren’t inconsistent with empirical knowledge.
I don’t know where to get a good prior. I suppose you might look at past instances where someone claimed to be close to doing something that seemed about as difficult and confusing as AGI seems to be (before taking into account a history of promises that didn’t pan out, but after taking into account what we know about the confusingness of the problem, insofar as that knowledge doesn’t itself come from the fact of failed promises). I don’t know what that prior would look like, but it seems like it would assign (if you randomly selected a kind of feat) a substantially greater than 1⁄10 probability of seeing at least 10 failed predictions of achieving that feat for every successful such prediction, a substantially greater than 1⁄100 probability of seeing at least 100 failed predictions for every successful prediction, and so on.
And why do you think FinalState is in such a circumstance, rather than just bullshitting us?
I was being charitable. Also, I misread the original post; see the comments below.
Hmm yeah, I read the post again and, if it’s a troll, it’s a way-more-subtle-than-typical one. Still, my posterior probability assignment on him being serious/sincere is in the 0.40s (extraordinary claims require extraordinary evidence) -- though this means that the probability that he succeeds given that he’s serious is the same order of magnitude as the probability that he succeeds given everything I know.
If you know you probably would not have survived the sun’s having failed to rise, you cannot just apply the Rule of Succession to your knowledge of past sunrises to calculate the probability that the sun will rise tomorrow because that would be ignoring relevant information, namely the existence of a severe selection bias. (Sadly, I do not know how to modify the Rule of Succession to account for the selection bias.)
Bostrom has made a stab at compensating, although I don’t think http://www.nickbostrom.com/papers/anthropicshadow.pdf works for the sun example.
On the other hand, if you have so much background knowledge about the Sun that you can think about the selection effects involved, the Rule of Succession is a moot & incomplete analysis to begin with.
Regarding your second paragraph, Sir Gwern, if we switch the example to the question of whether the US and Russia will launch nukes at each other this year, I have at lot of information about the strength of the selection bias (including for example Carl Sagan’s work on nuclear winter) that I might put to good use if I knew how to account for selection effects, but I would be sorely tempted to use something like the Rule of Succession (modified to account for the selection bias and where the analog of a day in which the sun might or might not rise is the start of the part of the career of someone in the military or in politics during which he or she can influence whether or not an attempt at a first strike is made) because my causal model of the mental processes behind the decision to launch is so unsatisfactory.
This might be a good place for me to point out that I never bought into the common wisdom, which I have never seen anyone object to or distance themselves from in print, that the chances of a nuclear exchange between the US and Russia went down considerably after the collapse of the Soviet Union in 1991.
What’s your line of thought?
Nuclear war isn’t the same situation, though. We can survive nuclear war at all sorts of levels of intensity, so the selection filter is not nearly the same as “the Sun going out”, which is ~100% fatal. Bostrom’s shadow paper might actually work for nuclear war, from the perspective of a revived civilization, but I’d have to reread it to see.
The selection filter does not have to be total or near total for my point to stand, namely, Rule-of-Succession-like calculations can be useful even when one has enough information to think about the selection effects involved (provided that Rule-of-Succession-like calculations are ever useful).
And parenthetically selection effects on observations about whether nuclear exchanges happened in the past can be very strong. Consider for example a family who has lived in Washington, D.C., for the last 5 decades: Washington, D.C., is such an important target that it is unlikely the family would have survived the launch of most or all of the Soviet/Russian arsenal at the U.S. So, although I agree with you that the human race as a whole would probably have survived almost any plausible nuclear exchange, that does not do the family in D.C. much good. More precisely, it does not do much good for the family’s ability to use historical data on whether or not nukes were launched at the U.S. in the past to refine their probability of launches in the future.
An interesting bracket style. How am I supposed to know where the parenthetical ends?