I initially wrote a long comment discussing the post, but I rewrote it as a list-based version that tries to more efficiently parcel up the different objections/agreements/cruxes.
This list ended up basically just as long, but I feel it is better structured than my original intended comment.
(Section 1): How fast can humans develop novel technologies
I believe you assume too much about the necessary time based on specific human discoveries.
Some of your backing evidence just didn’t have the right pressure at the time to go further (ex: submarines) which means that I think a more accurate estimate of the time interval would be finding the time that people started paying attention to the problem again (though for many things that’s probably hard to find) and began deliberately working on/towards that issue.
Though, while I think focusing on when they began deliberately working is more accurate, I think there’s still a notable amount of noise and basic differences due to the difference in ability to focus of humans relative to AGI, the unity (relative to a company), and the large amount of existing data in the future
Other technologies I would expect were ‘put off’ because they’re also closely linked to the available technology at the time. It can be hard to do specific things if your Materials-science understanding simply isn’t good enough.
Then there’s the obvious throttling at the number of people in the industry focusing on that issue, or even capable of focusing on that issue.
As well, to assume thirty years means that you also assume that the AGI does not have the ability to provide more incentive to ‘speed up’. If it needs to build a factory, then yes there are practical limitations on how fast the factory can be built, but obstructions like regulation and cost are likely easier to remove for an AGI than a normal company.
Crux #1: How long it takes for human inventions to spread after being thought up / initially tested / etc.
This is honestly the one that seems to be the primary generator for your ‘decades’ estimate, however I did not find it that compelling even if I accept the premise that an AGI would not be able to build nanotechnology (without building new factories to build the new tools it needs to actually perform it)
Note: The other cruxes later on are probably more about how much the AI can speed up research (or already has access to), but this could probably include a a specific crux related to that before this crux.
(Section 2): Unstoppable intellect meets the complexity of the universe
While I agree that there are likely eventual physical limits (though likely you hit practical expected ROI before that) on intelligence and research results.
There would be many low-hanging fruits which are significantly easier to grab with a combination of high compute + intelligence that we simply didn’t/couldn’t grab beforehand. (This would be affected by the lead time, if we had good math prover/explainer AIs for two decades before AGI then we’d have started to pick a lot of the significant ideas, but as the next part points out, having more of the research already available just helps you)
I also think that the fact that we’ve gotten rid of many of the notable easier-to-reach pieces (ex: classical mechanics → GR → QM → QFT) is actually a sign that things are easier now in terms of doing something. The AGI has a significantly larger amount of information about physics, human behavior, logic, etcetera, that it can use without having to build it completely from the ground up.
If you (somehow) had an AGI appear in 1760 without much knowledge, then I’d expect that it would take many experiments and a lot of time to detail the nature of its reality. Far less than we took, but still a notable amount. This is the scenario where I can see it taking 80 years for the AGI to get set up, but even then I think that’s more due to restrictions on readily available compute to expand into after self-modification than other constraints.
However, we’ve picked out a lot of the high and low level models that work. Rather than building an understanding of atoms through careful experimentation procedures, it can assume that they exist and pretty much follow the rules its been given.
(maybe) Crux #2: Do we already have most of the knowledge needed to understand and/or build nanotechnology?
I’m listing this as ‘maybe’ as I’m more notably uncertain about this than others.
Does it just require the concentrated effort of a monolithic agent staring down at the problem and being willing to crunch a lot of calculations and physics simulators?
Or does it require some very new understanding of how our physics works?
(Section 3): What does AGI want?
Minor objection on the split of categories. I’d find it.. odd if we manage to make an AI that terminally values only ‘kill all humans’.
I’d expect more varying terminal values, with ‘make humans not a threat at all’ (through whatever means) as an instrumental goal
I do think it is somewhat useful for your thought experiments later on try making the point that even a ‘YOLO AGI’ would have a hard time having an effect
(Section 4): What does it take to make a pencil?
I think this analogy ignores various issues
Of course, we’re talking about pencils, but the analogy is more about ‘molecular-level 3d-printer’ or ‘factory technology needed to make molecular level printer’ (or ‘advanced protein synthesis machine’)
Making a handful of pencils if you really need them is a lot more efficient than setting up that entire system.
Though, of course, if you’re needing mass production levels of that object then yes you will need this sort of thing.
Crux #3: How feasible is it to make small numbers of specialized technology?
There’s some scientific setups that are absolutely massive and require enormous amounts of funding, however then there are those that with the appropriate tools you can setup in a home workshop. I highly doubt either of those is the latter, but I’d also be skeptical that they need to be the size of the LHC.
Note: Crux #4 (about feasibility of being able to make nanotechnology with a sufficient understanding of it and with current day or near-future protein synthesis) is closely related, but it felt more natural to put that with AlphaFold.
(Section 5): YOLO AGI?
I think your objection that they’re all perfectly doable by humans in the present is lacking.
By metaphor:
While it is possible for someone to calculate a million digits of pi by hand, the difference between speed and overall capability is shocking.
While it is possible for a monkey to kill all of its enemies, humans have a far easier time with modern weaponry, especially in terms of scale
Your assumption that it would take decades for even just the scenarios you list (except perhaps the last two) seems wrong
Unless you’re predicating on the goal being literally wiping out every human, but then that’s a problem with the model simplification of YOLO AGI. Where we model an extreme version of an AGI to talk about the more common, relatively less extreme versions that aren’t hell-bent on killing us, just neutralizing us. (Which is what I’m assuming the intent from the section #3 split and this is)
Then there’s, of course, other scenarios that you can think up. For various levels of speed and sure lethality
Ex: Relatively more mild memetic hazards (perhaps the level of ‘kill your neighbor’ memetic hazard is too hard to find) but still destructive can cause significant problems and gives room to be more obvious.
Synthesize a food/drink/recreational-drug that is quite nice (and probably cheap) that also sterilizes you after a decade, to use in combination with other plans to make it even harder to bounce back if you don’t manage to kill them in a decade
To say that an AGI focused on killing will only “somewhat” increase the chances seems to underplay it severely.
If I believed a nation state solidly wanted to do any of those on the list in order to kill humanity right now, then that would increase my worry significantly more than ‘somewhat’
For an AGI that:
Isn’t made up of humans who may value being alive, or are willing to put it off for a bit for more immediate rewards than their philosophy
Can essentially be a one-being research organization
Likely hides itself better
then I would be even more worried.
(Section 6): But what about AlphaFold?
This ignores how recent AlphaFold is.
I would expect that it would improve notably over the next decade, given the evidence that it works being supplied to the market.
(It would be like assuming GPT-1 would never improve, while there’s certainly limits on how much it can improve, do we have evidence now that AlphaFold is even halfway to the practical limit?)
This ignores possibility of more ‘normal’ simulation:
While simulating physics accurately is highly computationally expensive, I don’t find it infeasible that
AI before, or the AGI itself, will find some neat ways of specializing the problem to their specific class of problems that they’re interested (aka abstractions over the behavior of specific molecules, rather than accurately simulating them) that are just intractable for an unassisted human to find
This also has benefits in that it is relatively more well understood, which makes it likely easier to model for errors than AlphaFold (though the difference depends on how far we/the-AGI get with AI interpretability)
The AI can get access to relatively large amounts of compute when it needs it.
I expect that it can make a good amount of progress in theory before it needs to do detailed physics implementations to test its ideas.
I also expect this to only grow over time, unless it takes actions to harshly restrict compute to prevent rivals
I’m very skeptical of the claim that it would need decades of lab experiments to fill in the gaps in our understanding of proteins.
If the methods for predicting proteins get only to twice as good as AlphaFold, then the AGI would specifically design to avoid hard-to-predict proteins
My argument here is primarily that you can do a tradeoff of making your design more complex-in-terms-of-lots-of-basic-pieces-rather-than-a-mostly-single-whole/large in order to get better predictive accuracy.
Crux #4: How good can technology to simulate physics (and/or isolated to a specific part of physics, like protein interactions) practically get?
(Specifically practical in terms of ROI, maybe we can only completely crack protein folding with planet sized computers, but that isn’t feasible for us or the AGI on the timescales we’re talking about)
Are we near the limit already? Even before we gain a deeper understanding of how networks work and how to improve their efficiency? Even before powerful AI/AGI are applied to the issue?
(Section 7): What if AGI settles for a robot army?
‘The robots are running on pre-programmed runs in a human-designed course and are not capable of navigating through unknown terrain’
Are they actually pre-programmed in the sense that they flashed the rom (or probably uploaded onto the host OS) the specific steps, or is it “Go from point A to point B along this path” where it then dodges obstacles?
As well, this doesn’t stop it from being a body to just directly control.
We’ll also have further notable advancements in robots that can navigate appropriately by the time AGI comes about
As well as increased number, though this depends on how popular/useful they are. I don’t expect a ‘Mr. Handy’ Fallout style butler in every home, but I’d expect robots from places like Boston Mechanics to start filtering out more and more to organizations that want them over the next decade or two.
Existing factories already exist (likely now and almost certainly in the future), which dodges the issue of having to design + build them. AGI buys Boston Mechanics / manipulates / just buys robots and then can have robots that it doesn’t even have to hack remotely but can aggressively tear down if it wants. Though, of course the equivalent(s) at the time.
I think you ovestimate how hard it would be to control robots remotely.
As for, hosting a clone of the AGI, I do think this is unlikely in part due to feasibility but also that there’s better methods.
Though, I would note that I believe it makes sense to expect that we can reduce model sizes significantly (either during training or afterwards) with help of better models of how networks work and that with AI help we could reduce it further.
Though, while this may mean that in the future it might be feasible to run GPT-3 on a normal laptop at that time, that doesn’t mean that you can fit the AGI on a robot. Perhaps you could fit a seed AGI, but then you lose a lot of data. Anyway.
I’d be surprised if the battery usage couldn’t be improved significantly, whether through better battery designs over the next two decades or more efficient designs or larger bodies (since that’s for Spot, which isn’t humanoid sized, so carrying around a heavy battery is more significant)
I also object that the AGI has little reason to bother with normal human warfare, unless it really makes itself obvious.
It has little reason to keep large swaths of land. (It could protect some factory, but unless you’re getting supplies then that’s a problem)
It has incentive to just disappear as best as possible, or just shrug and release a plague since humanities risk just went up
Again, a thirty years prediction.
I’ve already argued against it even needing to bother with thirty years, and I don’t think that it needs a typical conception of robot army in most cases
I think this claim of ‘thirty years’ for this thing depends (beyond the other bits) on how much we’ve automated various parts of the system before then. We have a trend towards it, and our AIs are getting better at tasks like that, so I don’t think its unlikely. Though I also think its reasonable to expect we’ll settle somewhere before almost full automation.
(Section 8): Mere mortals can’t comprehend AGI
While there is the mildly fun idea of the AGI discovering the one unique trick that immediately makes it effectively a god, I do agree its unlikely.
However, I don’t think that provides much evidence for your thirty years timeframe suggestion
I do think you should be more wary of black swan events, where the AI basically cracks an area of math/problem-solving/socialization-rules/etcetera, but this doesn’t play a notable role in my analysis above.
(Section 9): (Not commented upon)
General:
I think the ‘take a while to use human manufacturing’ is a possible scenario, but I think relative to shorter methods of neutralization (ex: nanotech) it ranks low.
(Minor note: It probably ranks higher in probability than nanotech, but that’s because nanotech is so specific relative to ‘uses human manufacturing for a while’, but I don’t think it ranks higher than a bunch of ways to neutralize humanity that take < 3 years)
Overall, I think the article makes some good points in a few places, but I also think it is not doing great epistemically in terms of considering what those you disagree with believe or might believe and in terms of your certainty.
Just to preface: Eliezer’s article has this issue, but it is a list/introducing-generator-of-thoughts, more for bringing in unsaid ideas explicitly into words as well as for for reference. Your article is an explainer of the reasons why you think he’s wrong about a specific issue.
(If there’s odd grammar/spelling, then that’s primarily because I wrote this while feeling sleepy and then continued for several more hours)
I initially wrote a long comment discussing the post, but I rewrote it as a list-based version that tries to more efficiently parcel up the different objections/agreements/cruxes.
This list ended up basically just as long, but I feel it is better structured than my original intended comment.
(Section 1): How fast can humans develop novel technologies
I believe you assume too much about the necessary time based on specific human discoveries.
Some of your backing evidence just didn’t have the right pressure at the time to go further (ex: submarines) which means that I think a more accurate estimate of the time interval would be finding the time that people started paying attention to the problem again (though for many things that’s probably hard to find) and began deliberately working on/towards that issue.
Though, while I think focusing on when they began deliberately working is more accurate, I think there’s still a notable amount of noise and basic differences due to the difference in ability to focus of humans relative to AGI, the unity (relative to a company), and the large amount of existing data in the future
Other technologies I would expect were ‘put off’ because they’re also closely linked to the available technology at the time. It can be hard to do specific things if your Materials-science understanding simply isn’t good enough.
Then there’s the obvious throttling at the number of people in the industry focusing on that issue, or even capable of focusing on that issue.
As well, to assume thirty years means that you also assume that the AGI does not have the ability to provide more incentive to ‘speed up’. If it needs to build a factory, then yes there are practical limitations on how fast the factory can be built, but obstructions like regulation and cost are likely easier to remove for an AGI than a normal company.
Crux #1: How long it takes for human inventions to spread after being thought up / initially tested / etc.
This is honestly the one that seems to be the primary generator for your ‘decades’ estimate, however I did not find it that compelling even if I accept the premise that an AGI would not be able to build nanotechnology (without building new factories to build the new tools it needs to actually perform it)
Note: The other cruxes later on are probably more about how much the AI can speed up research (or already has access to), but this could probably include a a specific crux related to that before this crux.
(Section 2): Unstoppable intellect meets the complexity of the universe
While I agree that there are likely eventual physical limits (though likely you hit practical expected ROI before that) on intelligence and research results.
There would be many low-hanging fruits which are significantly easier to grab with a combination of high compute + intelligence that we simply didn’t/couldn’t grab beforehand. (This would be affected by the lead time, if we had good math prover/explainer AIs for two decades before AGI then we’d have started to pick a lot of the significant ideas, but as the next part points out, having more of the research already available just helps you)
I also think that the fact that we’ve gotten rid of many of the notable easier-to-reach pieces (ex: classical mechanics → GR → QM → QFT) is actually a sign that things are easier now in terms of doing something. The AGI has a significantly larger amount of information about physics, human behavior, logic, etcetera, that it can use without having to build it completely from the ground up.
If you (somehow) had an AGI appear in 1760 without much knowledge, then I’d expect that it would take many experiments and a lot of time to detail the nature of its reality. Far less than we took, but still a notable amount. This is the scenario where I can see it taking 80 years for the AGI to get set up, but even then I think that’s more due to restrictions on readily available compute to expand into after self-modification than other constraints.
However, we’ve picked out a lot of the high and low level models that work. Rather than building an understanding of atoms through careful experimentation procedures, it can assume that they exist and pretty much follow the rules its been given.
(maybe) Crux #2: Do we already have most of the knowledge needed to understand and/or build nanotechnology?
I’m listing this as ‘maybe’ as I’m more notably uncertain about this than others.
Does it just require the concentrated effort of a monolithic agent staring down at the problem and being willing to crunch a lot of calculations and physics simulators?
Or does it require some very new understanding of how our physics works?
(Section 3): What does AGI want?
Minor objection on the split of categories. I’d find it.. odd if we manage to make an AI that terminally values only ‘kill all humans’.
I’d expect more varying terminal values, with ‘make humans not a threat at all’ (through whatever means) as an instrumental goal
I do think it is somewhat useful for your thought experiments later on try making the point that even a ‘YOLO AGI’ would have a hard time having an effect
(Section 4): What does it take to make a pencil?
I think this analogy ignores various issues
Of course, we’re talking about pencils, but the analogy is more about ‘molecular-level 3d-printer’ or ‘factory technology needed to make molecular level printer’ (or ‘advanced protein synthesis machine’)
Making a handful of pencils if you really need them is a lot more efficient than setting up that entire system.
Though, of course, if you’re needing mass production levels of that object then yes you will need this sort of thing.
Crux #3: How feasible is it to make small numbers of specialized technology?
There’s some scientific setups that are absolutely massive and require enormous amounts of funding, however then there are those that with the appropriate tools you can setup in a home workshop. I highly doubt either of those is the latter, but I’d also be skeptical that they need to be the size of the LHC.
Note: Crux #4 (about feasibility of being able to make nanotechnology with a sufficient understanding of it and with current day or near-future protein synthesis) is closely related, but it felt more natural to put that with AlphaFold.
(Section 5): YOLO AGI?
I think your objection that they’re all perfectly doable by humans in the present is lacking.
By metaphor:
While it is possible for someone to calculate a million digits of pi by hand, the difference between speed and overall capability is shocking.
While it is possible for a monkey to kill all of its enemies, humans have a far easier time with modern weaponry, especially in terms of scale
Your assumption that it would take decades for even just the scenarios you list (except perhaps the last two) seems wrong
Unless you’re predicating on the goal being literally wiping out every human, but then that’s a problem with the model simplification of YOLO AGI. Where we model an extreme version of an AGI to talk about the more common, relatively less extreme versions that aren’t hell-bent on killing us, just neutralizing us. (Which is what I’m assuming the intent from the section #3 split and this is)
Then there’s, of course, other scenarios that you can think up. For various levels of speed and sure lethality
Ex: Relatively more mild memetic hazards (perhaps the level of ‘kill your neighbor’ memetic hazard is too hard to find) but still destructive can cause significant problems and gives room to be more obvious.
Synthesize a food/drink/recreational-drug that is quite nice (and probably cheap) that also sterilizes you after a decade, to use in combination with other plans to make it even harder to bounce back if you don’t manage to kill them in a decade
To say that an AGI focused on killing will only “somewhat” increase the chances seems to underplay it severely.
If I believed a nation state solidly wanted to do any of those on the list in order to kill humanity right now, then that would increase my worry significantly more than ‘somewhat’
For an AGI that:
Isn’t made up of humans who may value being alive, or are willing to put it off for a bit for more immediate rewards than their philosophy
Can essentially be a one-being research organization
Likely hides itself better
then I would be even more worried.
(Section 6): But what about AlphaFold?
This ignores how recent AlphaFold is.
I would expect that it would improve notably over the next decade, given the evidence that it works being supplied to the market.
(It would be like assuming GPT-1 would never improve, while there’s certainly limits on how much it can improve, do we have evidence now that AlphaFold is even halfway to the practical limit?)
This ignores possibility of more ‘normal’ simulation:
While simulating physics accurately is highly computationally expensive, I don’t find it infeasible that
AI before, or the AGI itself, will find some neat ways of specializing the problem to their specific class of problems that they’re interested (aka abstractions over the behavior of specific molecules, rather than accurately simulating them) that are just intractable for an unassisted human to find
This also has benefits in that it is relatively more well understood, which makes it likely easier to model for errors than AlphaFold (though the difference depends on how far we/the-AGI get with AI interpretability)
The AI can get access to relatively large amounts of compute when it needs it.
I expect that it can make a good amount of progress in theory before it needs to do detailed physics implementations to test its ideas.
I also expect this to only grow over time, unless it takes actions to harshly restrict compute to prevent rivals
I’m very skeptical of the claim that it would need decades of lab experiments to fill in the gaps in our understanding of proteins.
If the methods for predicting proteins get only to twice as good as AlphaFold, then the AGI would specifically design to avoid hard-to-predict proteins
My argument here is primarily that you can do a tradeoff of making your design more complex-in-terms-of-lots-of-basic-pieces-rather-than-a-mostly-single-whole/large in order to get better predictive accuracy.
Crux #4: How good can technology to simulate physics (and/or isolated to a specific part of physics, like protein interactions) practically get?
(Specifically practical in terms of ROI, maybe we can only completely crack protein folding with planet sized computers, but that isn’t feasible for us or the AGI on the timescales we’re talking about)
Are we near the limit already? Even before we gain a deeper understanding of how networks work and how to improve their efficiency? Even before powerful AI/AGI are applied to the issue?
(Section 7): What if AGI settles for a robot army?
‘The robots are running on pre-programmed runs in a human-designed course and are not capable of navigating through unknown terrain’
Are they actually pre-programmed in the sense that they flashed the rom (or probably uploaded onto the host OS) the specific steps, or is it “Go from point A to point B along this path” where it then dodges obstacles?
As well, this doesn’t stop it from being a body to just directly control.
We’ll also have further notable advancements in robots that can navigate appropriately by the time AGI comes about
As well as increased number, though this depends on how popular/useful they are. I don’t expect a ‘Mr. Handy’ Fallout style butler in every home, but I’d expect robots from places like Boston Mechanics to start filtering out more and more to organizations that want them over the next decade or two.
Existing factories already exist (likely now and almost certainly in the future), which dodges the issue of having to design + build them. AGI buys Boston Mechanics / manipulates / just buys robots and then can have robots that it doesn’t even have to hack remotely but can aggressively tear down if it wants. Though, of course the equivalent(s) at the time.
I think you ovestimate how hard it would be to control robots remotely.
As for, hosting a clone of the AGI, I do think this is unlikely in part due to feasibility but also that there’s better methods.
Though, I would note that I believe it makes sense to expect that we can reduce model sizes significantly (either during training or afterwards) with help of better models of how networks work and that with AI help we could reduce it further.
Though, while this may mean that in the future it might be feasible to run GPT-3 on a normal laptop at that time, that doesn’t mean that you can fit the AGI on a robot. Perhaps you could fit a seed AGI, but then you lose a lot of data. Anyway.
I’d be surprised if the battery usage couldn’t be improved significantly, whether through better battery designs over the next two decades or more efficient designs or larger bodies (since that’s for Spot, which isn’t humanoid sized, so carrying around a heavy battery is more significant)
I also object that the AGI has little reason to bother with normal human warfare, unless it really makes itself obvious.
It has little reason to keep large swaths of land. (It could protect some factory, but unless you’re getting supplies then that’s a problem)
It has incentive to just disappear as best as possible, or just shrug and release a plague since humanities risk just went up
Again, a thirty years prediction.
I’ve already argued against it even needing to bother with thirty years, and I don’t think that it needs a typical conception of robot army in most cases
I think this claim of ‘thirty years’ for this thing depends (beyond the other bits) on how much we’ve automated various parts of the system before then. We have a trend towards it, and our AIs are getting better at tasks like that, so I don’t think its unlikely. Though I also think its reasonable to expect we’ll settle somewhere before almost full automation.
(Section 8): Mere mortals can’t comprehend AGI
While there is the mildly fun idea of the AGI discovering the one unique trick that immediately makes it effectively a god, I do agree its unlikely.
However, I don’t think that provides much evidence for your thirty years timeframe suggestion
I do think you should be more wary of black swan events, where the AI basically cracks an area of math/problem-solving/socialization-rules/etcetera, but this doesn’t play a notable role in my analysis above.
(Section 9): (Not commented upon)
General:
I think the ‘take a while to use human manufacturing’ is a possible scenario, but I think relative to shorter methods of neutralization (ex: nanotech) it ranks low.
(Minor note: It probably ranks higher in probability than nanotech, but that’s because nanotech is so specific relative to ‘uses human manufacturing for a while’, but I don’t think it ranks higher than a bunch of ways to neutralize humanity that take < 3 years)
Overall, I think the article makes some good points in a few places, but I also think it is not doing great epistemically in terms of considering what those you disagree with believe or might believe and in terms of your certainty.
Just to preface: Eliezer’s article has this issue, but it is a list/introducing-generator-of-thoughts, more for bringing in unsaid ideas explicitly into words as well as for for reference. Your article is an explainer of the reasons why you think he’s wrong about a specific issue.
(If there’s odd grammar/spelling, then that’s primarily because I wrote this while feeling sleepy and then continued for several more hours)