Hey everyone. I’m new here. I’ve recently been kinda freaking out about AGI and its timelines… Specially after reading Eliezer’s post about there being no fire-alarm.
However, he also mentions that “one could never be sure of a new thing coming, except for the last one or two years” (something along those lines).
So, in your opinion, are we already at a stage where AGI could arrive anytime? Because due to things like GPT-3, Wu Dao 2.0 and AlphaCode, I’ve been getting really scared… Plus if there is something more advanced being developed in secret...
Or will there at least be a 1-2 year “last epistemic stage” which we can be sure we haven’t reached yet? (as Soares also mentions)
Cause everyday I’ve been looking out the window expecting the nano-swarms to come lol… But I’m just a lay person, so I’d like to hear some more expert opinions.
(If you’re scared, in general it’s good to do things that give you courage. Perhaps think through your strengths, the ways you can change the world, and make sure you have good relationships with friends/family when you need support.)
IMO we are already at a stage where AGI could arrive at any time in some sense, but the probability of it arriving in the next year or so is pretty small—some AI lab would need to have some major breakthrough between now and then, something that enables them to do with merely hundreds of millions of dollars of compute what seems like it should take trillions (with current algorithms). I think we probably have like eight years left or something like that.
Sober view as well, and much closer to mine. I definitely agree that compute will be the big bottleneck—GPT-3 and the scaling hypothesis scare the heck out of me.
8 years makes a lot of sense, after all many predictions point to 2030.
A more paranoid me would like to ask, what number would you give to the probabilities of it arriving: a) next week, b) next year?
And also: are you also paranoid like me looking out the window fom the nano-swarms, or think that at least in the very, very near-term it’s still close to impossible?
I am not looking out my window for the nano-swarms; I think there’s a less than 1% chance of that happening this year. We would need a completely new AI paradigm I think, which is not impossible (It’s happened a bunch of times in the past, and there are a few ideas floating around that could be it) but unlikely and especially unlikely to happen all of a sudden without me hearing signs first. And then even with said new paradigm it would be surprising if takeoff was so fast that I saw nanobots before hearing any disturbing news through the grapevine.
So, <1% chance of nano-swarms surprising me this year, <<1% this week.
Maybe something like 2% chance of agentic AGI (or, APS-AI to use Carlsmith’s term) happening this year?
Another (very weird) counterpoint: you might not see the “swarm coming” because the annexing of our cosmic endowment might look way stranger than the best strategy human minds can come up with.
I remember a safety researcher once mentioned to me that they didn’t necessarily expect us to be killed, just contained, while superintelligence takes over the universe. The argument being that it might want to preserve its history (ie. us) to study it, instead of permanently destroying it. This is basically as bad as also killing everyone, because we’d still be imprisoned away from our largest possible impact. Similar to the serious component in “global poverty is just a rounding error”.
Now I think if you add that our “imprisonment” might be made barely comfortable (which is quite unlikely, but maybe plausible in some almost-aligned-but-ultimately-terrible uncanny value scenarios), then it’s possible that there’s never a discontinuous horror that we would see striking us; instead we will suddenly be blocked from our cosmic endowment without our knowledge. Things will seem to be going essentially on track. But we never quite seem to get to the future we’ve been waiting for.
It would be a steganographic takeoff.
Here’s a (only slightly) more fleshed out argument:
If
deception is something that “blocks induction on it” (eg. you can’t run a million tests on deceptive optimizers and hope for the pattern on the tests to continue), and if
all our “deductions” are really just an assertion of induction at higher levels of abstraction (eg. asserting that Logic will continue to hold)
then deception could look “steganographic” when it’s done at really meta levels, exploiting our more basic metaphysical mistakes.
Interesting stuff. And I agree. Once you have a nanosystem or something of equivalent power, humans are no longer any threat. But we’re yet to be sure if such thing is physically possible. I know many here think so, but I still have my doubts.
Maybe it’s even more likely that some random narrow AI failure will start big wars before anything more fancy. Although with the scaling hypothesis on sight, AGI could come suddenly indeed.
“This is basically as bad as also killing everyone, because we’d still be imprisoned away from our largest possible impact.”
Although I quite disagree with this. I’m not a huge supporter of our largest possible impact. I guess it’s naive to attribute any net positive expectation to that when you look at history or at the present. In fact, such outcome (things staying exactly the same forever) would probably be among the most positive ones in the advent of non aligned AI. As long as we could still take care of Earth, like ending factory farming and dictatorships, it really wouldn’t be that bad...
I am not an expert by any means, but here are my thoughts: While I find GPT-3 quite impressive, it’s not even close to AGI. All the models you mentioned are still focused on performing specific tasks. This alone will (probably) not be enough to create AGI, even if you try to increase the size of the models even further. I believe AGI is at least decades away, perhaps even a hundred years away. Now, there is a possibility of stuff being developed in secret, which is impossible to account for, but I’d say the probability of these developments being significantly more advanced that the publicly available technologies is pretty low.
A sober opinion (even if quite different from mine). My biggest fear is scaling a transformer + completing it with other “parts”, as in an agent (even if a dumb one), etc. Thanks
Has GPT-3 / large transformers actually led to anything with economic value? Not from what I can tell although anecdotal reports on Twitter are that many SWEs are finding Github Copilot extremely useful (it’s still in private beta though). I think transformers are going to start providing actual value soon, but the fact they haven’t so far despite almost two years of breathless hype is interesting to contemplate. I’ve learned to ignore hype, demos, cool cherry-picked sample outputs, and benchmark chasing and actually look at what is being deployed “in the real world” and bringing value to people. So many systems that looked amazing in academic papers have flopped when deployed—even from top firms—for instance Microsoft’s Tay and Google Health’s system for detecting diabetic retinopathy. Another example is Google’s Duplex. And for how long have we heard about burger flipping robots taking people’s jobs?
There are reasons to be skeptical about about a scaled up GPT leading to AGI. I touched on some of those points here. There’s also an argument that the hardware costs are going to balloon so quickly to make the entire project economically unfeasible, but I’m pretty skeptical about that.
I’m more worried about someone reverse engineering the wiring of cortical columns in the neocortex in the next few years and then replicating it in silicon.
Long story short, is existentially dangerous AI eminent? Not as far as we can see right now knowing what we know right now (we can’t see that far in the future, since it depends on discoveries and scientific knowledge we don’t have). Could that change quickly anytime? Yes. There is Knightian uncertainty here, I think (to use a concept that LessWrongers generally hate lol).
Economic value might not be a perfect measure. Nuclear fission didn’t generate any economic value either until 200.000 in Japan were incinerated. My fear is that a mixture of experts approach can lead to extremely fast progress towards AGI. Perhaps even less—maybe all it takes is an agent AI that can code as well as humans, to start a cascade of recursive self-improvement.
But indeed, a Knightian uncertainty here would already put me at some ease. As long as you can be sure that it won’t happen “just anytime” before some more barriers are crossed, at least you can still sleep at night and have the sanity to try to do something.
I don’t know, I’m not a technical person, that’s why I’m asking questions and hoping to learn more.
“I’m more worried about someone reverse engineering the wiring of cortical columns in the neocortex in the next few years and then replicating it in silicon.”
Personally that’s what worries me the least. We can’t even crack c.elegans! I don’t doubt that in 100-200 years we’d get there but I see many other way faster routes.
In general, whenever Reason makes you feel paralyzed, remember that Reason has many things to say. Thousands of people in history have been convinced by trains of thought of the form ‘X is unavoidable, everything is about X, you are screwed’. Many pairs of those trains of thought contradict each other. This pattern is all over the history of philosophy, religion, & politics.
Future hazards deserve more research funding, yes, but remember that the future is not certain.
“Thousands of people in history have been convinced by trains of thought of the form ‘X is unavoidable, everything is about X, you are screwed’.”
Care to give a few examples? Because I’d venture saying that, except for religious and other superstitious beliefs, and except for crazy lunatics too like fascists and communists, they were mostly right.
“the future is not certain”
Depends on what you mean by that. If you mean that it’s not extremely likely, like 90% plus, that we will develop some truly dangerous form of AI this century that will pose immense control challenges, then I’d say you’re deeply misguided given the smoke signals that have been coming up since 2017.
I mean, it’s like worrying about nuclear war. Is it certain that we’ll ever get a big nuclear war? No. Is it extremely likely if things stay the same and if enough time passes (10, 50, 100, 200, 300 years)? Hell yes. I mean, just look at the current situation...
Though I don’t care about nuclear war much because it is also extremely likely that it will come with a warning, so you can also run to the countryside, and even then if things go bad like you’re starving to death or dying of radiation poisoning, you can always put an end to your own suffering. With AI you might not be so lucky. You might end in an unbreakable dictatorship a la With Folded Hands.
How can you not feel paralyzed when you see chaos pointed at your head and at the heads of other humans, coming in as little as 5 or 10 years, and you see absolutely no solution, or much less anything you can do yourself?
We can’t even build a provably safe plane, how are we gonna build a provably safe TAI with the work of a few hundred people over 5-30 years, and with complete ignorance by most?
The world would have to wake up, and I don’t think it will.
Really, the only ways we will not build dangerous and uncontrollable AI is if either we destroy ourselves by some other way first (or even just with narrow AI), or the miracle happens that someone cracks advanced nanotechnology/magic through narrow AI and becomes a benevolent and omnipotent world dictator. There’s really no other way we won’t end up doing it.
Hey everyone. I’m new here. I’ve recently been kinda freaking out about AGI and its timelines… Specially after reading Eliezer’s post about there being no fire-alarm.
However, he also mentions that “one could never be sure of a new thing coming, except for the last one or two years” (something along those lines).
So, in your opinion, are we already at a stage where AGI could arrive anytime? Because due to things like GPT-3, Wu Dao 2.0 and AlphaCode, I’ve been getting really scared… Plus if there is something more advanced being developed in secret...
Or will there at least be a 1-2 year “last epistemic stage” which we can be sure we haven’t reached yet? (as Soares also mentions)
Cause everyday I’ve been looking out the window expecting the nano-swarms to come lol… But I’m just a lay person, so I’d like to hear some more expert opinions.
(If you’re scared, in general it’s good to do things that give you courage. Perhaps think through your strengths, the ways you can change the world, and make sure you have good relationships with friends/family when you need support.)
IMO we are already at a stage where AGI could arrive at any time in some sense, but the probability of it arriving in the next year or so is pretty small—some AI lab would need to have some major breakthrough between now and then, something that enables them to do with merely hundreds of millions of dollars of compute what seems like it should take trillions (with current algorithms). I think we probably have like eight years left or something like that.
Sober view as well, and much closer to mine. I definitely agree that compute will be the big bottleneck—GPT-3 and the scaling hypothesis scare the heck out of me.
8 years makes a lot of sense, after all many predictions point to 2030.
A more paranoid me would like to ask, what number would you give to the probabilities of it arriving: a) next week, b) next year?
And also: are you also paranoid like me looking out the window fom the nano-swarms, or think that at least in the very, very near-term it’s still close to impossible?
I am not looking out my window for the nano-swarms; I think there’s a less than 1% chance of that happening this year. We would need a completely new AI paradigm I think, which is not impossible (It’s happened a bunch of times in the past, and there are a few ideas floating around that could be it) but unlikely and especially unlikely to happen all of a sudden without me hearing signs first. And then even with said new paradigm it would be surprising if takeoff was so fast that I saw nanobots before hearing any disturbing news through the grapevine.
So, <1% chance of nano-swarms surprising me this year, <<1% this week.
Maybe something like 2% chance of agentic AGI (or, APS-AI to use Carlsmith’s term) happening this year?
Fair argument, thanks.
Another (very weird) counterpoint: you might not see the “swarm coming” because the annexing of our cosmic endowment might look way stranger than the best strategy human minds can come up with.
I remember a safety researcher once mentioned to me that they didn’t necessarily expect us to be killed, just contained, while superintelligence takes over the universe. The argument being that it might want to preserve its history (ie. us) to study it, instead of permanently destroying it. This is basically as bad as also killing everyone, because we’d still be imprisoned away from our largest possible impact. Similar to the serious component in “global poverty is just a rounding error”.
Now I think if you add that our “imprisonment” might be made barely comfortable (which is quite unlikely, but maybe plausible in some almost-aligned-but-ultimately-terrible uncanny value scenarios), then it’s possible that there’s never a discontinuous horror that we would see striking us; instead we will suddenly be blocked from our cosmic endowment without our knowledge. Things will seem to be going essentially on track. But we never quite seem to get to the future we’ve been waiting for.
It would be a steganographic takeoff.
Here’s a (only slightly) more fleshed out argument:
If
deception is something that “blocks induction on it” (eg. you can’t run a million tests on deceptive optimizers and hope for the pattern on the tests to continue), and if
all our “deductions” are really just an assertion of induction at higher levels of abstraction (eg. asserting that Logic will continue to hold)
then deception could look “steganographic” when it’s done at really meta levels, exploiting our more basic metaphysical mistakes.
Interesting stuff. And I agree. Once you have a nanosystem or something of equivalent power, humans are no longer any threat. But we’re yet to be sure if such thing is physically possible. I know many here think so, but I still have my doubts.
Maybe it’s even more likely that some random narrow AI failure will start big wars before anything more fancy. Although with the scaling hypothesis on sight, AGI could come suddenly indeed.
“This is basically as bad as also killing everyone, because we’d still be imprisoned away from our largest possible impact.”
Although I quite disagree with this. I’m not a huge supporter of our largest possible impact. I guess it’s naive to attribute any net positive expectation to that when you look at history or at the present. In fact, such outcome (things staying exactly the same forever) would probably be among the most positive ones in the advent of non aligned AI. As long as we could still take care of Earth, like ending factory farming and dictatorships, it really wouldn’t be that bad...
I am not an expert by any means, but here are my thoughts: While I find GPT-3 quite impressive, it’s not even close to AGI. All the models you mentioned are still focused on performing specific tasks. This alone will (probably) not be enough to create AGI, even if you try to increase the size of the models even further. I believe AGI is at least decades away, perhaps even a hundred years away. Now, there is a possibility of stuff being developed in secret, which is impossible to account for, but I’d say the probability of these developments being significantly more advanced that the publicly available technologies is pretty low.
A sober opinion (even if quite different from mine). My biggest fear is scaling a transformer + completing it with other “parts”, as in an agent (even if a dumb one), etc. Thanks
Has GPT-3 / large transformers actually led to anything with economic value? Not from what I can tell although anecdotal reports on Twitter are that many SWEs are finding Github Copilot extremely useful (it’s still in private beta though). I think transformers are going to start providing actual value soon, but the fact they haven’t so far despite almost two years of breathless hype is interesting to contemplate. I’ve learned to ignore hype, demos, cool cherry-picked sample outputs, and benchmark chasing and actually look at what is being deployed “in the real world” and bringing value to people. So many systems that looked amazing in academic papers have flopped when deployed—even from top firms—for instance Microsoft’s Tay and Google Health’s system for detecting diabetic retinopathy. Another example is Google’s Duplex. And for how long have we heard about burger flipping robots taking people’s jobs?
There are reasons to be skeptical about about a scaled up GPT leading to AGI. I touched on some of those points here. There’s also an argument that the hardware costs are going to balloon so quickly to make the entire project economically unfeasible, but I’m pretty skeptical about that.
I’m more worried about someone reverse engineering the wiring of cortical columns in the neocortex in the next few years and then replicating it in silicon.
Long story short, is existentially dangerous AI eminent? Not as far as we can see right now knowing what we know right now (we can’t see that far in the future, since it depends on discoveries and scientific knowledge we don’t have). Could that change quickly anytime? Yes. There is Knightian uncertainty here, I think (to use a concept that LessWrongers generally hate lol).
Economic value might not be a perfect measure. Nuclear fission didn’t generate any economic value either until 200.000 in Japan were incinerated. My fear is that a mixture of experts approach can lead to extremely fast progress towards AGI. Perhaps even less—maybe all it takes is an agent AI that can code as well as humans, to start a cascade of recursive self-improvement.
But indeed, a Knightian uncertainty here would already put me at some ease. As long as you can be sure that it won’t happen “just anytime” before some more barriers are crossed, at least you can still sleep at night and have the sanity to try to do something.
I don’t know, I’m not a technical person, that’s why I’m asking questions and hoping to learn more.
“I’m more worried about someone reverse engineering the wiring of cortical columns in the neocortex in the next few years and then replicating it in silicon.”
Personally that’s what worries me the least. We can’t even crack c.elegans! I don’t doubt that in 100-200 years we’d get there but I see many other way faster routes.
In general, whenever Reason makes you feel paralyzed, remember that Reason has many things to say. Thousands of people in history have been convinced by trains of thought of the form ‘X is unavoidable, everything is about X, you are screwed’. Many pairs of those trains of thought contradict each other. This pattern is all over the history of philosophy, religion, & politics.
Future hazards deserve more research funding, yes, but remember that the future is not certain.
“Thousands of people in history have been convinced by trains of thought of the form ‘X is unavoidable, everything is about X, you are screwed’.”
Care to give a few examples? Because I’d venture saying that, except for religious and other superstitious beliefs, and except for crazy lunatics too like fascists and communists, they were mostly right.
“the future is not certain”
Depends on what you mean by that. If you mean that it’s not extremely likely, like 90% plus, that we will develop some truly dangerous form of AI this century that will pose immense control challenges, then I’d say you’re deeply misguided given the smoke signals that have been coming up since 2017.
I mean, it’s like worrying about nuclear war. Is it certain that we’ll ever get a big nuclear war? No. Is it extremely likely if things stay the same and if enough time passes (10, 50, 100, 200, 300 years)? Hell yes. I mean, just look at the current situation...
Though I don’t care about nuclear war much because it is also extremely likely that it will come with a warning, so you can also run to the countryside, and even then if things go bad like you’re starving to death or dying of radiation poisoning, you can always put an end to your own suffering. With AI you might not be so lucky. You might end in an unbreakable dictatorship a la With Folded Hands.
How can you not feel paralyzed when you see chaos pointed at your head and at the heads of other humans, coming in as little as 5 or 10 years, and you see absolutely no solution, or much less anything you can do yourself?
We can’t even build a provably safe plane, how are we gonna build a provably safe TAI with the work of a few hundred people over 5-30 years, and with complete ignorance by most?
The world would have to wake up, and I don’t think it will.
Really, the only ways we will not build dangerous and uncontrollable AI is if either we destroy ourselves by some other way first (or even just with narrow AI), or the miracle happens that someone cracks advanced nanotechnology/magic through narrow AI and becomes a benevolent and omnipotent world dictator. There’s really no other way we won’t end up doing it.