What probability would you give to FinalState’s assertion of having a working AGI?
0%, since it apparently isn’t finished yet.
Will it be finished in a year? 2%, as all other attempts that have reached the “final stages” have failed to build working AGI. The most credible of those attempts were done with groups; it appears FinalState is working alone.
That’s the kind of probability I would’ve assigned to EURISKO destroying the world back when Lenat was the first person ever to try to build anything self-improving. For a random guy on the Internet it’s off by… maybe five orders of magnitude? I would expect a pretty tiny fraction of all worlds to have the names of homebrew projects carved on their tombstones, and there are many random people on the Internet claiming to have AGI.
People like this are significant, not because of their chances of creating AGI, but because of what their inability to stop or take any serious precautions, despite their belief that they are about to create AGI, tells us about human nature.
Understanding “random guy on the Internet” to mean something like an Internet user all I know about whom is that they are interested in building AGI and willing to put some concerted effort into the project… hrm… yeah, I’ll accept e-7 as within my range.
My estimate for an actual random person on the Internet building AGI in, say, the next decade, has a ceiling of e-10 or so, but I don’t have a clue what its lower bound is.
That said, I’m not sure how well-correlated the willingness of a “random guy on the Internet” (meaning 1) to try to build AGI without taking precautions is to the willingness of someone whose chances are orders of magnitude higher to do so.
Then again, we have more compelling lines of evidence leading us to expect humans to not take precautions.
My estimate for an actual random person on the Internet building AGI in, say, the next decade, has a ceiling of e-10 or so, but I don’t have a clue what its lower bound is.
(I had to read that three times before getting why that number was 1000 times smaller than the other one, because I kept on misinterpreting “random person”. Try “randomly-chosen person”.)
I have no idea what you understood “random person” to mean, if not randomly chosen person. I’m also curious now as to whether whatever-that-is is what EY meant in the first place.
A stranger, esp. one behaving in weird ways; this appears to me to be the most common meaning of that word in 21st-century English when applied to a person. (Older speakers might be unfamiliar with it, but the median LWer is 25 years old, as of the latest survey.) And I also had taken the indefinite article to be an existential quantifier; hence, I had effectively interpreted the statement as “at least one actual strange person on the Internet building AGI in the next decade”, for which I thought such a low probability would be ridiculous.
but because of what their inability to stop or take any serious precautions, despite their belief that they are about to create AGI, tells us about human nature.
Are these in any way a representative sample of normal humans? In order to be in this category one generally needs to be pretty high on the crank scale along with some healthy Dunning-Kruger issues.
That’s always been the argument that future AGI scientists won’t be as crazy as the lunatics presently doing it—that the current crowd of researchers are self-selected for incaution—but I wouldn’t put too much weight on that; it seems like a very human behavior, some of the smarter ones with millions of dollars don’t seem of below-average competence in any other way, and the VCs funding them are similarly incapable of backing off even when they say they expect human-level AGI to be created.
Sorry, I’m confused. By “people like this” did you mean people like FinalState or did you mean professional AI researchers? I interpreted it as the first.
Before people downvote PM’s comment above, note that Eliezer’s comment prior to editing was a hierarchy of different AI researchers with lowest being people like FinalState, the second highest being professional AI researchers and the highest being “top AI researchers”.
With that out of the way, what do you think you are accomplishing with this remark? You have a variety of valid points to make, but I fail to see what is contained in this remark that does anything at all.
Me or Eliezer? I’m making some point by direct demonstration. It’s a popular ranking system, ya know? He used it on FinalState. A lot of people use it on him.
This just shifts the question to how you slotted FinalState into such a promising reference class? Conservatively, tens of academic research programs, tens of PhD dissertations, hundreds of hobbyist projects, hundreds of undergraduate term projects, and tens of business ventures have attempted something similar to AGI and none have succeeded.
As far as I can tell, the vast majority of academic projects (particularly those of undergrads) have worked on narrow AI, which this is supposedly not.
However, reading the post again, it doesn’t sound as though they have the support of any academic institution; I misread the bit around “academic network”. It sounds more as though this is a homebrew project, in which case I need to go two or three orders of magnitude lower.
As far as I can tell, the vast majority of academic projects (particularly those of undergrads) have worked on narrow AI, which this is supposedly not.
That’s definitely a reasonable assessment. I dialed all those estimates down by about an order of magnitude from when I started writing that point as I thought through just how unusual attempting general AI is. But over sixty years and hundreds of institutions where one might get a sufficiently solid background in CS to implement something big, there are going to be lots of unusual people trying things out.
The Rule of Succession, if I’m not mistaken, assumes a uniform prior from 0 to 1 for the probability of success. That seems unreasonable; it shouldn’t be extremely improbable (even before observing failure) that fewer than one in a thousand such claims result in a working AGI. So you have to adjust downward somewhat from there, but it’s hard to say how much.
(This is in addition to the point that user:othercriteria makes in the sibling comment.)
You’re correct, but where would I find a better prior? I’d rather be too conservative than resort to wild guessing (which it would be, since I’m not an expert on AGI).
(A variant of this is rhollerith_dot_com’s objection below, that I failed to take into account whatever the probability of working AGI leading to death is. Presumably that changes the prior as well.)
A. Many commonly used priors are listed in the Handbook of Chemistry and Physics.
Q. Where do priors originally come from?
A. Never ask that question.
Q. Uh huh. Then where do scientists get their priors?
A. Priors for scientific problems are established by annual vote of the AAAS. In recent years the vote has become fractious and controversial, with widespread acrimony, factional polarization, and several outright assassinations. This may be a front for infighting within the Bayes Council, or it may be that the disputants have too much spare time. No one is really sure.
Q. I see. And where does everyone else get their priors?
A. They download their priors from Kazaa.
Q. What if the priors I want aren’t available on Kazaa?
A. There’s a small, cluttered antique shop in a back alley of San Francisco’s Chinatown. Don’t ask about the bronze rat.
Isn’t the lesson of the Quantum Physics sequence that ordinary humans today should get their priors from the least complex (and falsifiable?) statements that aren’t inconsistent with empirical knowledge.
I don’t know where to get a good prior. I suppose you might look at past instances where someone claimed to be close to doing something that seemed about as difficult and confusing as AGI seems to be (before taking into account a history of promises that didn’t pan out, but after taking into account what we know about the confusingness of the problem, insofar as that knowledge doesn’t itself come from the fact of failed promises). I don’t know what that prior would look like, but it seems like it would assign (if you randomly selected a kind of feat) a substantially greater than 1⁄10 probability of seeing at least 10 failed predictions of achieving that feat for every successful such prediction, a substantially greater than 1⁄100 probability of seeing at least 100 failed predictions for every successful prediction, and so on.
Hmm yeah, I read the post again and, if it’s a troll, it’s a way-more-subtle-than-typical one. Still, my posterior probability assignment on him being serious/sincere is in the 0.40s (extraordinary claims require extraordinary evidence) -- though this means that the probability that he succeeds given that he’s serious is the same order of magnitude as the probability that he succeeds given everything I know.
If you know you probably would not have survived the sun’s having failed to rise, you cannot just apply the Rule of Succession to your knowledge of past sunrises to calculate the probability that the sun will rise tomorrow because that would be ignoring relevant information, namely the existence of a severe selection bias. (Sadly, I do not know how to modify the Rule of Succession to account for the selection bias.)
On the other hand, if you have so much background knowledge about the Sun that you can think about the selection effects involved, the Rule of Succession is a moot & incomplete analysis to begin with.
Regarding your second paragraph, Sir Gwern, if we switch the example to the question of whether the US and Russia will launch nukes at each other this year, I have at lot of information about the strength of the selection bias (including for example Carl Sagan’s work on nuclear winter) that I might put to good use if I knew how to account for selection effects, but I would be sorely tempted to use something like the Rule of Succession (modified to account for the selection bias and where the analog of a day in which the sun might or might not rise is the start of the part of the career of someone in the military or in politics during which he or she can influence whether or not an attempt at a first strike is made) because my causal model of the mental processes behind the decision to launch is so unsatisfactory.
This might be a good place for me to point out that I never bought into the common wisdom, which I have never seen anyone object to or distance themselves from in print, that the chances of a nuclear exchange between the US and Russia went down considerably after the collapse of the Soviet Union in 1991.
This might be a good place for me to point out that I never bought into the common wisdom, which I have never seen anyone object to or distance themselves from in print, that the chances of a nuclear exchange between the US and Russia went down considerably after the collapse of the Soviet Union in 1991.
Nuclear war isn’t the same situation, though. We can survive nuclear war at all sorts of levels of intensity, so the selection filter is not nearly the same as “the Sun going out”, which is ~100% fatal. Bostrom’s shadow paper might actually work for nuclear war, from the perspective of a revived civilization, but I’d have to reread it to see.
The selection filter does not have to be total or near total for my point to stand, namely, Rule-of-Succession-like calculations can be useful even when one has enough information to think about the selection effects involved (provided that Rule-of-Succession-like calculations are ever useful).
And parenthetically selection effects on observations about whether nuclear exchanges happened in the past can be very strong. Consider for example a family who has lived in Washington, D.C., for the last 5 decades: Washington, D.C., is such an important target that it is unlikely the family would have survived the launch of most or all of the Soviet/Russian arsenal at the U.S. So, although I agree with you that the human race as a whole would probably have survived almost any plausible nuclear exchange, that does not do the family in D.C. much good. More precisely, it does not do much good for the family’s ability to use historical data on whether or not nukes were launched at the U.S. in the past to refine their probability of launches in the future.
Me too. This value is several orders of magnitude above my own estimate.
That said, it depends on your definition of “finished”. For example, it is much more plausible (relatively speaking) that FinalState will fail to produce an AGI, but will nevertheless produce an algorithm that performs some specific task—such as character recognition, unit pathing, natural language processing, etc. -- better than the leading solutions. In this case, I suppose one could still make the argument that FinalState’s project was finished somewhat successfully.
0%, since it apparently isn’t finished yet.
Will it be finished in a year? 2%, as all other attempts that have reached the “final stages” have failed to build working AGI. The most credible of those attempts were done with groups; it appears FinalState is working alone.
2%?
Seriously?
I am curious as to why your estimate is so high.
That’s the kind of probability I would’ve assigned to EURISKO destroying the world back when Lenat was the first person ever to try to build anything self-improving. For a random guy on the Internet it’s off by… maybe five orders of magnitude? I would expect a pretty tiny fraction of all worlds to have the names of homebrew projects carved on their tombstones, and there are many random people on the Internet claiming to have AGI.
People like this are significant, not because of their chances of creating AGI, but because of what their inability to stop or take any serious precautions, despite their belief that they are about to create AGI, tells us about human nature.
Understanding “random guy on the Internet” to mean something like an Internet user all I know about whom is that they are interested in building AGI and willing to put some concerted effort into the project… hrm… yeah, I’ll accept e-7 as within my range.
My estimate for an actual random person on the Internet building AGI in, say, the next decade, has a ceiling of e-10 or so, but I don’t have a clue what its lower bound is.
That said, I’m not sure how well-correlated the willingness of a “random guy on the Internet” (meaning 1) to try to build AGI without taking precautions is to the willingness of someone whose chances are orders of magnitude higher to do so.
Then again, we have more compelling lines of evidence leading us to expect humans to not take precautions.
(I had to read that three times before getting why that number was 1000 times smaller than the other one, because I kept on misinterpreting “random person”. Try “randomly-chosen person”.)
I have no idea what you understood “random person” to mean, if not randomly chosen person. I’m also curious now as to whether whatever-that-is is what EY meant in the first place.
A stranger, esp. one behaving in weird ways; this appears to me to be the most common meaning of that word in 21st-century English when applied to a person. (Older speakers might be unfamiliar with it, but the median LWer is 25 years old, as of the latest survey.) And I also had taken the indefinite article to be an existential quantifier; hence, I had effectively interpreted the statement as “at least one actual strange person on the Internet building AGI in the next decade”, for which I thought such a low probability would be ridiculous.
Thanks for clarifying.
Are these in any way a representative sample of normal humans? In order to be in this category one generally needs to be pretty high on the crank scale along with some healthy Dunning-Kruger issues.
That’s always been the argument that future AGI scientists won’t be as crazy as the lunatics presently doing it—that the current crowd of researchers are self-selected for incaution—but I wouldn’t put too much weight on that; it seems like a very human behavior, some of the smarter ones with millions of dollars don’t seem of below-average competence in any other way, and the VCs funding them are similarly incapable of backing off even when they say they expect human-level AGI to be created.
Sorry, I’m confused. By “people like this” did you mean people like FinalState or did you mean professional AI researchers? I interpreted it as the first.
AGI researchers sound a lot like FinalState when they think they’ll have AGI cracked in two years.
Eliezer < anyone with actual notable accomplishments. edit: damn it you edited your message.
Over 140 posts and 0 total karma; that’s persistence.
private_messaging says he’s Dmytry, who has positive karma. It’s possible that the more anonymous-sounding name encourages worse behaviour though.
Before people downvote PM’s comment above, note that Eliezer’s comment prior to editing was a hierarchy of different AI researchers with lowest being people like FinalState, the second highest being professional AI researchers and the highest being “top AI researchers”.
With that out of the way, what do you think you are accomplishing with this remark? You have a variety of valid points to make, but I fail to see what is contained in this remark that does anything at all.
Me or Eliezer? I’m making some point by direct demonstration. It’s a popular ranking system, ya know? He used it on FinalState. A lot of people use it on him.
There’s got to be a level beyond “arguments as soldiers” to describe your current approach to ineffective contrarianism.
I volunteer “arguments as cannon fodder.”
Laplace’s Rule of Succession, assuming around fifty failures under similar or more favorable circumstances.
This just shifts the question to how you slotted FinalState into such a promising reference class? Conservatively, tens of academic research programs, tens of PhD dissertations, hundreds of hobbyist projects, hundreds of undergraduate term projects, and tens of business ventures have attempted something similar to AGI and none have succeeded.
As far as I can tell, the vast majority of academic projects (particularly those of undergrads) have worked on narrow AI, which this is supposedly not.
However, reading the post again, it doesn’t sound as though they have the support of any academic institution; I misread the bit around “academic network”. It sounds more as though this is a homebrew project, in which case I need to go two or three orders of magnitude lower.
That’s definitely a reasonable assessment. I dialed all those estimates down by about an order of magnitude from when I started writing that point as I thought through just how unusual attempting general AI is. But over sixty years and hundreds of institutions where one might get a sufficiently solid background in CS to implement something big, there are going to be lots of unusual people trying things out.
Of those who attempted, fewer thought they were close, but fifty still seems very generous.
The Rule of Succession, if I’m not mistaken, assumes a uniform prior from 0 to 1 for the probability of success. That seems unreasonable; it shouldn’t be extremely improbable (even before observing failure) that fewer than one in a thousand such claims result in a working AGI. So you have to adjust downward somewhat from there, but it’s hard to say how much.
(This is in addition to the point that user:othercriteria makes in the sibling comment.)
You’re correct, but where would I find a better prior? I’d rather be too conservative than resort to wild guessing (which it would be, since I’m not an expert on AGI).
(A variant of this is rhollerith_dot_com’s objection below, that I failed to take into account whatever the probability of working AGI leading to death is. Presumably that changes the prior as well.)
http://yudkowsky.net/rational/bayes
Isn’t the lesson of the Quantum Physics sequence that ordinary humans today should get their priors from the least complex (and falsifiable?) statements that aren’t inconsistent with empirical knowledge.
I don’t know where to get a good prior. I suppose you might look at past instances where someone claimed to be close to doing something that seemed about as difficult and confusing as AGI seems to be (before taking into account a history of promises that didn’t pan out, but after taking into account what we know about the confusingness of the problem, insofar as that knowledge doesn’t itself come from the fact of failed promises). I don’t know what that prior would look like, but it seems like it would assign (if you randomly selected a kind of feat) a substantially greater than 1⁄10 probability of seeing at least 10 failed predictions of achieving that feat for every successful such prediction, a substantially greater than 1⁄100 probability of seeing at least 100 failed predictions for every successful prediction, and so on.
And why do you think FinalState is in such a circumstance, rather than just bullshitting us?
I was being charitable. Also, I misread the original post; see the comments below.
Hmm yeah, I read the post again and, if it’s a troll, it’s a way-more-subtle-than-typical one. Still, my posterior probability assignment on him being serious/sincere is in the 0.40s (extraordinary claims require extraordinary evidence) -- though this means that the probability that he succeeds given that he’s serious is the same order of magnitude as the probability that he succeeds given everything I know.
If you know you probably would not have survived the sun’s having failed to rise, you cannot just apply the Rule of Succession to your knowledge of past sunrises to calculate the probability that the sun will rise tomorrow because that would be ignoring relevant information, namely the existence of a severe selection bias. (Sadly, I do not know how to modify the Rule of Succession to account for the selection bias.)
Bostrom has made a stab at compensating, although I don’t think http://www.nickbostrom.com/papers/anthropicshadow.pdf works for the sun example.
On the other hand, if you have so much background knowledge about the Sun that you can think about the selection effects involved, the Rule of Succession is a moot & incomplete analysis to begin with.
Regarding your second paragraph, Sir Gwern, if we switch the example to the question of whether the US and Russia will launch nukes at each other this year, I have at lot of information about the strength of the selection bias (including for example Carl Sagan’s work on nuclear winter) that I might put to good use if I knew how to account for selection effects, but I would be sorely tempted to use something like the Rule of Succession (modified to account for the selection bias and where the analog of a day in which the sun might or might not rise is the start of the part of the career of someone in the military or in politics during which he or she can influence whether or not an attempt at a first strike is made) because my causal model of the mental processes behind the decision to launch is so unsatisfactory.
This might be a good place for me to point out that I never bought into the common wisdom, which I have never seen anyone object to or distance themselves from in print, that the chances of a nuclear exchange between the US and Russia went down considerably after the collapse of the Soviet Union in 1991.
What’s your line of thought?
Nuclear war isn’t the same situation, though. We can survive nuclear war at all sorts of levels of intensity, so the selection filter is not nearly the same as “the Sun going out”, which is ~100% fatal. Bostrom’s shadow paper might actually work for nuclear war, from the perspective of a revived civilization, but I’d have to reread it to see.
The selection filter does not have to be total or near total for my point to stand, namely, Rule-of-Succession-like calculations can be useful even when one has enough information to think about the selection effects involved (provided that Rule-of-Succession-like calculations are ever useful).
And parenthetically selection effects on observations about whether nuclear exchanges happened in the past can be very strong. Consider for example a family who has lived in Washington, D.C., for the last 5 decades: Washington, D.C., is such an important target that it is unlikely the family would have survived the launch of most or all of the Soviet/Russian arsenal at the U.S. So, although I agree with you that the human race as a whole would probably have survived almost any plausible nuclear exchange, that does not do the family in D.C. much good. More precisely, it does not do much good for the family’s ability to use historical data on whether or not nukes were launched at the U.S. in the past to refine their probability of launches in the future.
An interesting bracket style. How am I supposed to know where the parenthetical ends?
Me too. This value is several orders of magnitude above my own estimate.
That said, it depends on your definition of “finished”. For example, it is much more plausible (relatively speaking) that FinalState will fail to produce an AGI, but will nevertheless produce an algorithm that performs some specific task—such as character recognition, unit pathing, natural language processing, etc. -- better than the leading solutions. In this case, I suppose one could still make the argument that FinalState’s project was finished somewhat successfully.
Oh, I see that I misinterpreted FinalState’s statement
as an indication of only being a few minutes away from having a working implementation.