I went through the top 20 list of most cited AI researchers on google scholar (thanks to Amanda for linking), and estimated that roughly 9 of them may qualify under Rohin’s criterion. Of those 9, my guess was that 7⁄9 would answer ‘Yes’ on Rohin’s question 3.
My sampling process was certainly biased. For one, AI researchers are likely to be more safety conscious than industry experts. My estimate also involved considerable guesswork, so I down-weighted the estimated 7⁄9 to a 65% chance that the >=0.5 threshold will be met within the first couple years. Given the extreme difference between my distribution and the others posted, I guess there’s a 1⁄3 chance that my estimate based on the top 20 sampling will carry significant weight in Rohin’s posterior.
The justification for the rest of my distribution is similar to what others have said here and elsewhere about AI safety. My AGI timeline is roughly in line with the metaculus estimate here. Before the advent of AGI, a number of eventualities are possible: a warning shot occurs, perhaps theoretical consensus will emerge, perhaps industry researchers will be oblivious to safety concerns because of a principal-agent nature to the problem, perhaps AGI will be invented before safety is worked out, etc.
Edit: One could certainly do a better job of estimating where the sample population of researchers currently stands by finding a less biased population. Maybe people interviewed by Lex Fridman, that might be a decent proxy for AGI-research-fame?
The above estimate was mislead since I had mistakenly read ′ I then compute the fraction #answers(Yes, Yes, Yes) / #answers(Yes, *, *) ′ as ′ I then compute the fraction #answers(Yes, Yes, Yes) / #answers(Yes, Yes, *)’.
I agree with Ethan’s recent comment that experience with RL matter a lot, so a lot comes down to how the ′ Is X’s work related to AGI? ′ criterion is cashed out. On some reading of this, many NLP researchers do not count, on another reading they do count. I’d say my previous prediction was a decent, if slightly over-estimate of the scenario in which ‘related to AGI’ is interpreted narrowly, and many NLP researchers are ruled out.
A second major confounder is whether prominent AI researchers are far more likely to have been asked about their opinion on AI safety in which case they have some impetus to go read up on the issue.
To cash some of these concerns out into probabilities:
75% that Rohin takes a broad interpretation of AGI which includes e.g. GPT-team, NAS research etc.
33% estimated (Yes,Yes,Yes) by assuming prominent researchers 2x as likely to have read up on AI safety.
25% downweighted from 33% taking into account industry being less concerned.
Assuming that we’re at ~33% now, 50% doesn’t seem too far out of reach, so my estimates for following decades are based on the same concerns I listed in my above comment framed with the 33% in mind.
(Post competition footnote: seems to me over short time horizons we should have a more-or-less geometric distribution. Think of the more-or-less independent per year chance that a NeurIPS keynote features AI safety, or youtube recommender algorithm goes bonkers for a bit. Seems strange to me that some other people’s distribution over the next 10-15 years—if not longer—do not look geometric.)
I do take the broad interpretation of AGI-related work.
I hadn’t considered the point that people may ask prominent AI researchers their opinion about AI safety, and that leading them to have better beliefs about safety. I think overall I don’t actually expect this to be a major factor, but it’s a good point and updated me slightly towards sooner.
I wouldn’t expect a geometric distribution—consensus building requires time, as a result you might expect a buildup from 0 for <time taken to build consensus> and then have it follow a geometric distribution. In addition, getting to 50% seems likely to require a warning shot of some significance; current AI systems don’t seem capable enough to produce a compelling enough warning shot.
My old prediction for when the fraction be >= 0.5: elicited
My old prediction for Rohin’s posterior: elicited
I went through the top 20 list of most cited AI researchers on google scholar (thanks to Amanda for linking), and estimated that roughly 9 of them may qualify under Rohin’s criterion. Of those 9, my guess was that 7⁄9 would answer ‘Yes’ on Rohin’s question 3.
My sampling process was certainly biased. For one, AI researchers are likely to be more safety conscious than industry experts. My estimate also involved considerable guesswork, so I down-weighted the estimated 7⁄9 to a 65% chance that the >=0.5 threshold will be met within the first couple years. Given the extreme difference between my distribution and the others posted, I guess there’s a 1⁄3 chance that my estimate based on the top 20 sampling will carry significant weight in Rohin’s posterior.
The justification for the rest of my distribution is similar to what others have said here and elsewhere about AI safety. My AGI timeline is roughly in line with the metaculus estimate here. Before the advent of AGI, a number of eventualities are possible: a warning shot occurs, perhaps theoretical consensus will emerge, perhaps industry researchers will be oblivious to safety concerns because of a principal-agent nature to the problem, perhaps AGI will be invented before safety is worked out, etc.
Edit: One could certainly do a better job of estimating where the sample population of researchers currently stands by finding a less biased population. Maybe people interviewed by Lex Fridman, that might be a decent proxy for AGI-research-fame?
The above estimate was mislead since I had mistakenly read ′ I then compute the fraction #answers(Yes, Yes, Yes) / #answers(Yes, *, *) ′ as ′ I then compute the fraction #answers(Yes, Yes, Yes) / #answers(Yes, Yes, *)’.
I agree with Ethan’s recent comment that experience with RL matter a lot, so a lot comes down to how the ′ Is X’s work related to AGI? ′ criterion is cashed out. On some reading of this, many NLP researchers do not count, on another reading they do count. I’d say my previous prediction was a decent, if slightly over-estimate of the scenario in which ‘related to AGI’ is interpreted narrowly, and many NLP researchers are ruled out.
A second major confounder is whether prominent AI researchers are far more likely to have been asked about their opinion on AI safety in which case they have some impetus to go read up on the issue.
To cash some of these concerns out into probabilities:
75% that Rohin takes a broad interpretation of AGI which includes e.g. GPT-team, NAS research etc.
33% estimated (Yes,Yes,Yes) by assuming prominent researchers 2x as likely to have read up on AI safety.
25% downweighted from 33% taking into account industry being less concerned.
Assuming that we’re at ~33% now, 50% doesn’t seem too far out of reach, so my estimates for following decades are based on the same concerns I listed in my above comment framed with the 33% in mind.
Updated personal distribution: elicited
Updated Rohin’s posterior: elicited
(Post competition footnote: seems to me over short time horizons we should have a more-or-less geometric distribution. Think of the more-or-less independent per year chance that a NeurIPS keynote features AI safety, or youtube recommender algorithm goes bonkers for a bit. Seems strange to me that some other people’s distribution over the next 10-15 years—if not longer—do not look geometric.)
I do take the broad interpretation of AGI-related work.
I hadn’t considered the point that people may ask prominent AI researchers their opinion about AI safety, and that leading them to have better beliefs about safety. I think overall I don’t actually expect this to be a major factor, but it’s a good point and updated me slightly towards sooner.
I wouldn’t expect a geometric distribution—consensus building requires time, as a result you might expect a buildup from 0 for <time taken to build consensus> and then have it follow a geometric distribution. In addition, getting to 50% seems likely to require a warning shot of some significance; current AI systems don’t seem capable enough to produce a compelling enough warning shot.