I think this is an excellent, well-researched contribution and am confused about why it’s not being upvoted more (on LW that is; it seems to be doing much better on EAF, interestingly).
At a guess (not having voted on it myself): because most of the model doesn’t engage with the parts of the question that those voting consider interesting/relevant, such as the many requirements laid out for “transformative AI” which don’t see at all necessary for x-risk. While this does seem to be targeting OpenPhil’s given definition of AGI, they do say in a footnote:
What we’re actually interested in is the potential existential threat posed by advanced AI systems.
While some people do have AI x-risk models that route through ~full automation (or substantial automation, with a clearly visible path to full automation), I think most people here don’t have models that require that, or even have substantial probability mass on it.
Interesting. When I participated in the AI Adversarial Collaboration Project, a study funded by Open Philanthropy and executed by the Forecasting Research Institute, I got the sense that most folks concerned about AI x-risk mostly believed that AGIs would kill us on their own accord (rather than by accident or as a result of human direction), that AGIs would have self-preservation goals, and therefore AGIs would likely only kill us after solving robotic supply chains (or enslaving/manipulating humans, as I argued as an alternative).
Sounds like your perception is that LessWrong folks don’t think robotic supply chain automation will be a likely prerequisite to AI x-risk?
There’s an interesting question: if a power-seeking AI had a button that instantly murdered every human, how much human-requiring preparatory work would it want to do before pressing the button? People seem to have strongly clashing intuitions here, and there aren’t any great writeups IMO. Some takes on the side of “AI wouldn’t press the button until basically the whole world economy was run by robots” are 1, 2, 3, 4, 5. I tend to be on the other side, for example I wrote here:
it seems pretty plausible to me that if there’s an AGI server and a solar cell and one teleoperated robot body in an otherwise-empty post-apocalyptic Earth, well then that one teleoperated robot body could build a janky second teleoperated robot body from salvaged car parts or whatever, and then the two of them could find more car parts to build a third and fourth, and those four could build up to eight, etc. That was basically the story I was telling here.…
Some cruxes:
One crux on that is how much compute is needed to run a robot—if it’s “1 consumer-grade GPU” then my story above seems to work, if it’s “10⁶ SOTA GPUs” then probably not.
Another crux is how much R&D needs to be done before we can build a computational substrate using self-assembling nanotechnology (whose supply chain / infrastructure needs are presumably much much lower than chip fabs). This is clearly possible, since human brains are in that category, but it’s unclear just how much R&D needs to be done before an AI could start doing that.
For example, Eliezer is optimistic (umm, I guess that’s the wrong word) that this is doable without very much real-world experimenting (as opposed to “thinking” and doing simulations / calculations via computer), and this path is part of why he expects AI might kill every human seemingly out of nowhere.
Another crux is just how minimal is a “minimal supply chain that can make good-enough chips” if the self-assembling route of the previous bullet point is not feasible. Such a supply chain would presumably be very very different from the supply chain that humans use to make chips, because obviously we’re not optimizing for that. As a possible example, e-beam lithography (EBL) is extraordinarily slow and expensive but works even better than EUV photolithography, and it’s enormously easier to build a janky EBL than to get EUV working. A commercial fab in the human world would never dream of mass-manufacturing chips by filling giant warehouses with zillions of janky and slow EBL machines, because the cost would be astronomical, the market can’t support that. But for an AI rebuilding in an empty postapocalyptic world, why not? And that’s just one (possible) example.
Robotic supply chain automation only seems necessary in worlds where it’s either surprisingly difficult to get AGI to a sufficiently superhuman level of cognitive ability (such that it can find a much faster route to takeover), worlds where faster/more reliable routes to takeover either don’t exist or are inaccessible even to moderately superhuman AGI, or some combination of the two.
I think this is an excellent, well-researched contribution and am confused about why it’s not being upvoted more (on LW that is; it seems to be doing much better on EAF, interestingly).
At a guess (not having voted on it myself): because most of the model doesn’t engage with the parts of the question that those voting consider interesting/relevant, such as the many requirements laid out for “transformative AI” which don’t see at all necessary for x-risk. While this does seem to be targeting OpenPhil’s given definition of AGI, they do say in a footnote:
While some people do have AI x-risk models that route through ~full automation (or substantial automation, with a clearly visible path to full automation), I think most people here don’t have models that require that, or even have substantial probability mass on it.
Interesting. When I participated in the AI Adversarial Collaboration Project, a study funded by Open Philanthropy and executed by the Forecasting Research Institute, I got the sense that most folks concerned about AI x-risk mostly believed that AGIs would kill us on their own accord (rather than by accident or as a result of human direction), that AGIs would have self-preservation goals, and therefore AGIs would likely only kill us after solving robotic supply chains (or enslaving/manipulating humans, as I argued as an alternative).
Sounds like your perception is that LessWrong folks don’t think robotic supply chain automation will be a likely prerequisite to AI x-risk?
There’s an interesting question: if a power-seeking AI had a button that instantly murdered every human, how much human-requiring preparatory work would it want to do before pressing the button? People seem to have strongly clashing intuitions here, and there aren’t any great writeups IMO. Some takes on the side of “AI wouldn’t press the button until basically the whole world economy was run by robots” are 1, 2, 3, 4, 5. I tend to be on the other side, for example I wrote here:
Some cruxes:
One crux on that is how much compute is needed to run a robot—if it’s “1 consumer-grade GPU” then my story above seems to work, if it’s “10⁶ SOTA GPUs” then probably not.
Another crux is how much R&D needs to be done before we can build a computational substrate using self-assembling nanotechnology (whose supply chain / infrastructure needs are presumably much much lower than chip fabs). This is clearly possible, since human brains are in that category, but it’s unclear just how much R&D needs to be done before an AI could start doing that.
For example, Eliezer is optimistic (umm, I guess that’s the wrong word) that this is doable without very much real-world experimenting (as opposed to “thinking” and doing simulations / calculations via computer), and this path is part of why he expects AI might kill every human seemingly out of nowhere.
Another crux is just how minimal is a “minimal supply chain that can make good-enough chips” if the self-assembling route of the previous bullet point is not feasible. Such a supply chain would presumably be very very different from the supply chain that humans use to make chips, because obviously we’re not optimizing for that. As a possible example, e-beam lithography (EBL) is extraordinarily slow and expensive but works even better than EUV photolithography, and it’s enormously easier to build a janky EBL than to get EUV working. A commercial fab in the human world would never dream of mass-manufacturing chips by filling giant warehouses with zillions of janky and slow EBL machines, because the cost would be astronomical, the market can’t support that. But for an AI rebuilding in an empty postapocalyptic world, why not? And that’s just one (possible) example.
Robotic supply chain automation only seems necessary in worlds where it’s either surprisingly difficult to get AGI to a sufficiently superhuman level of cognitive ability (such that it can find a much faster route to takeover), worlds where faster/more reliable routes to takeover either don’t exist or are inaccessible even to moderately superhuman AGI, or some combination of the two.