People in the alignment community are in a bubble, and talk about “alignment research” and “capability research” as if they are two distinct fields of approximately equal import. To everyone else, the field of “AI capability research” is just known as “AI research”. And so, by trying to remove people worried about AI risk from AI research, you are trying to bring about a scenario where the field of AI research has a consensus that AI risk is not a real problem.
It seems fairly obvious and probably uncontentious that reading AI safety related literature (MIRI stuff, and even more broadly the sequences, HPMOR) is reasonably well correlated with holding an AI risk or doomer worldview, which leads to a strong selection effect as those people end up competing for a limited number (although less so now) of AI safety jobs rather than working in DL advancing timelines. Indeed that was EY et al’s goal!
On the other hand, deep reading of DL/neurosci is less correlated or perhaps anticorrelated with the high p(doom) worldview. Part of that naturally could be selection effect (on both sides), but I argue that instead a large chunk of the standard doom arguments appeal only to those lacking grounding in DL/neuroscience—knowledge which tends to lead to a more optimistic wordlview.
All that being said, from what I see the net contribution of AI safety researchers to DL capability / AGI timelines effectively rounds to zero. The most relevant border exception is perhaps some of the interpretability work having engineering utility, but even there doesn’t seem to have developed much earlier.
On the other hand, deep reading of DL/neurosci is less correlated or perhaps anticorrelated with the high p(doom) worldview.
Which population are you evaluating this with respect to? If with the general population, it seems obviously false. If with those on LessWrong, I’m uncertain. If with other academics, probably false. If with other people who know anything about DL, probably weakly false, but I’d guess it looks more like people who know a moderate amount are less worried than those who know a bit, but also less worried than those who know a lot. But I’m also pretty uncertain about that.
For those in DL or neurosci outside of LW, high p(doom) seems more rare, from what I can gather. For some specific examples of the notable DL+neurosci people: Jeff Hawkins doesn’t think AGI/ASI poses much existential risk, Hassabis takes the risk seriously but his words/actions strongly imply[1] low p(doom). Carmack doesn’t seem very worried. Hinton doesn’t give a p estimate but from recent interviews I’d guess he’s anywhere between p(5) and p(50). Randall O’ Reilly’s paper on neurmorophic AGI safety is an example[2]: there is risk yes but those who generally believe we are getting neurmorphic AGI mostly aren’t nearly as worried as EY/doomers.
For the LW neurosci contingent the shard theory people like Q Pope aren’t nearly as worried and largely optimistic. I’m also in that contingent and put p(doom) at ~5% or so. I’m not sure what Bryne’s p(doom) is but I’d wager it’s less than 50%.
50% seems pretty high to me, I thought you were trying to make a population level case that the more knowledge you have about deep learning, the lower your probability of doom is. Outside LessWrong, I think most surveys are against your position. Most outside the field don’t see it as a world-ending issue, and surveys often turn up an average of over 10% among experts that it ends up being a world-ending issue. Though the ones I know of mostly look at DL researchers, not neuroscientists. I don’t think any survey has been done about the topic within the LessWrong-o-sphere. I do not know people’s pdooms off the top of my head. Seems plausible.
I thought you were trying to make a population level case that the more knowledge you have about deep learning, the lower your probability of doom is.
Yes sort of but not exactly—deep knowledge of DL and neurosci in particular is somewhat insulating against many of the doom arguments. People outside the field are not relevant here, i’m only concerned with a fairly elite group who have somewhat rare knowledge. For example there are only a handful of people on LW who I would consider demonstrably well read in DL&neurosci and they mostly have lower p(doom) then EY/MIRI.
Most outside the field don’t see it as a world-ending issue, and surveys often turn up an average of over 10% among experts that it ends up being a world-ending issue.
The actual results are near the complete opposite of what you claim.
The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%.
5% is near my p(doom) and that of Q Pope’s (who is a self proclaimed optimist). So the median DL respondent from their survey is an optimist, which proves my point.
Also only a small portion of those sent the survey actually responded, and only a small portion of those who responded − 162 individuals—actually answered the doom question. It seems extremely unlikely that responding to that question was correlated with optimism, so there is probably a large sample bias effect here.
I will note that I was correct in the number I gave. The mean is 14%, the median is 5%. Though I didn’t know the median was so low, so good piece of data to include. And your original claim was about the fraction of people with high pdoom, so the median seems more relevant.
Otherwise, good points. I guess I have more disagreements with the DL networks than I thought.
It seems fairly obvious and probably uncontentious that reading AI safety related literature (MIRI stuff, and even more broadly the sequences, HPMOR) is reasonably well correlated with holding an AI risk or doomer worldview, which leads to a strong selection effect as those people end up competing for a limited number (although less so now) of AI safety jobs rather than working in DL advancing timelines. Indeed that was EY et al’s goal!
On the other hand, deep reading of DL/neurosci is less correlated or perhaps anticorrelated with the high p(doom) worldview. Part of that naturally could be selection effect (on both sides), but I argue that instead a large chunk of the standard doom arguments appeal only to those lacking grounding in DL/neuroscience—knowledge which tends to lead to a more optimistic wordlview.
All that being said, from what I see the net contribution of AI safety researchers to DL capability / AGI timelines effectively rounds to zero. The most relevant border exception is perhaps some of the interpretability work having engineering utility, but even there doesn’t seem to have developed much earlier.
Which population are you evaluating this with respect to? If with the general population, it seems obviously false. If with those on LessWrong, I’m uncertain. If with other academics, probably false. If with other people who know anything about DL, probably weakly false, but I’d guess it looks more like people who know a moderate amount are less worried than those who know a bit, but also less worried than those who know a lot. But I’m also pretty uncertain about that.
For those in DL or neurosci outside of LW, high p(doom) seems more rare, from what I can gather. For some specific examples of the notable DL+neurosci people: Jeff Hawkins doesn’t think AGI/ASI poses much existential risk, Hassabis takes the risk seriously but his words/actions strongly imply[1] low p(doom). Carmack doesn’t seem very worried. Hinton doesn’t give a p estimate but from recent interviews I’d guess he’s anywhere between p(5) and p(50). Randall O’ Reilly’s paper on neurmorophic AGI safety is an example[2]: there is risk yes but those who generally believe we are getting neurmorphic AGI mostly aren’t nearly as worried as EY/doomers.
For the LW neurosci contingent the shard theory people like Q Pope aren’t nearly as worried and largely optimistic. I’m also in that contingent and put p(doom) at ~5% or so. I’m not sure what Bryne’s p(doom) is but I’d wager it’s less than 50%.
Like this interview for example.
link
50% seems pretty high to me, I thought you were trying to make a population level case that the more knowledge you have about deep learning, the lower your probability of doom is. Outside LessWrong, I think most surveys are against your position. Most outside the field don’t see it as a world-ending issue, and surveys often turn up an average of over 10% among experts that it ends up being a world-ending issue. Though the ones I know of mostly look at DL researchers, not neuroscientists. I don’t think any survey has been done about the topic within the LessWrong-o-sphere. I do not know people’s pdooms off the top of my head. Seems plausible.
Yes sort of but not exactly—deep knowledge of DL and neurosci in particular is somewhat insulating against many of the doom arguments. People outside the field are not relevant here, i’m only concerned with a fairly elite group who have somewhat rare knowledge. For example there are only a handful of people on LW who I would consider demonstrably well read in DL&neurosci and they mostly have lower p(doom) then EY/MIRI.
If you are referring to this survey: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/?ref=warpnews.org
The actual results are near the complete opposite of what you claim.
5% is near my p(doom) and that of Q Pope’s (who is a self proclaimed optimist). So the median DL respondent from their survey is an optimist, which proves my point.
Also only a small portion of those sent the survey actually responded, and only a small portion of those who responded − 162 individuals—actually answered the doom question. It seems extremely unlikely that responding to that question was correlated with optimism, so there is probably a large sample bias effect here.
I will note that I was correct in the number I gave. The mean is 14%, the median is 5%. Though I didn’t know the median was so low, so good piece of data to include. And your original claim was about the fraction of people with high pdoom, so the median seems more relevant.
Otherwise, good points. I guess I have more disagreements with the DL networks than I thought.