My stock counterargument to this: insofar as humans’ advantage over other animals stems primarily from our ability to transmit knowledge/memes/etc across individuals and through generations, we should expect AI to have a much larger advantage, because they can do the same thing far, far better. This doesn’t even require the AI to be all that “smart”—even just the ability to copy minds directly would allow transmission from “parent” to “child” with far less knowledge-loss than humans can achieve. (Consider, for instance, the notorious difficulty of training a student to the same level as an unusually accomplished human—and then imagine how much more effective it would be to just copy the accomplished human.) And that’s just copying. The ability to merge two minds does not necessarily come for free with any AI (as copying does), but will likely be far easier for AI than for humans, and would provide a much more extreme version of what humans achieve by passing information via language. (Consider, for instance, the notorious difficulty of communicating well and avoiding misunderstanding—and then imagine how much easier that would be if two minds or components-of-minds could merge.)
What these arguments suggest is that humans’ language capabilities are themselves relatively primitive ways to transmit knowledge/memes/etc between individuals, and it is not in-principle hard to do better once one starts to optimize directly for that goal without the constraints of biological wetware. More generally, we should expect that whatever did give humans such a large advantage (be it language or something else), humans have a relatively primitive version of it, because otherwise humans would have taken off sooner in our evolutionary history. As the saying goes, “humans are as stupid as they can be while still taking over the world, because otherwise they would have done so earlier”.
Yes, I agree with this. I mean, even if we assume that the AIs are basically equivalent to human simulations, they still get obvious advantages from the ability to be copy-pasted, the ability to be restored to a checkpoint, the ability to be run at higher clock speeds, and the ability to make credible pre-commitments, etc etc. I therefore certainly don’t think there is any plausible scenario in which unchecked AI systems wouldn’t end up with most of the power on earth. However, there is a meaningful difference between the scenario where their advantages mainly come from overwhelmingly great intelligence, and the scenario where their advantages mainly (or at least in large part) come from other sources. For example, scaleable oversight is a more realistic possibility in the latter scenario than it is in the former scenario. Boxing methods are also more realistic in the latter scenario than the former scenario, etc.
I think this scenario is still strategically isomorphic to “advantages mainly come from overwhelmingly great intelligence”. It’s intelligence at the level of a collective, rather than the individual level, but the conclusion is the same. For instance, scalable oversight of a group of AIs which is collectively far smarter than any group of humans is hard in basically the same ways as oversight of one highly-intelligent AI. Boxing the group of AIs is hard for the same reasons as boxing one. Etc.
I think the broad strokes are mostly similar, but that a bunch of relevant details are different.
Yes, a large collective of near-human AI that is allowed to interact freely over a (subjectively) long period of time is presumably roughly as hard to understand and control as a Bostrom/Yudkowsky-esque God in a box. However, in this scenario, we have the option to not allow free interaction between multiple instances, while still being able to extract useful work from them. It is also probably much easier to align a system that is not of overwhelming intelligence, and this could be done before the AIs are allowed to interact. We might also be able to significantly influence their collective behaviour by controlling the initial conditions of their interactions (similarly to how institutions and cultural norms have a substantial long-term impact on the trajectory of a country, for example). It is also more plausible that humans (or human simulations or emulations) could be kept in the loop for a long time period in this scenario. Moreover, if intelligence is bottle-necked by external resources (such as memory, data, CPU cycles, etc) rather than internal algorithmic efficiency, then you can exert more control over the resulting intelligence explosion by controlling those resources. Etc etc.
My stock counterargument to this: insofar as humans’ advantage over other animals stems primarily from our ability to transmit knowledge/memes/etc across individuals and through generations, we should expect AI to have a much larger advantage, because they can do the same thing far, far better. This doesn’t even require the AI to be all that “smart”—even just the ability to copy minds directly would allow transmission from “parent” to “child” with far less knowledge-loss than humans can achieve. (Consider, for instance, the notorious difficulty of training a student to the same level as an unusually accomplished human—and then imagine how much more effective it would be to just copy the accomplished human.) And that’s just copying. The ability to merge two minds does not necessarily come for free with any AI (as copying does), but will likely be far easier for AI than for humans, and would provide a much more extreme version of what humans achieve by passing information via language. (Consider, for instance, the notorious difficulty of communicating well and avoiding misunderstanding—and then imagine how much easier that would be if two minds or components-of-minds could merge.)
What these arguments suggest is that humans’ language capabilities are themselves relatively primitive ways to transmit knowledge/memes/etc between individuals, and it is not in-principle hard to do better once one starts to optimize directly for that goal without the constraints of biological wetware. More generally, we should expect that whatever did give humans such a large advantage (be it language or something else), humans have a relatively primitive version of it, because otherwise humans would have taken off sooner in our evolutionary history. As the saying goes, “humans are as stupid as they can be while still taking over the world, because otherwise they would have done so earlier”.
Yes, I agree with this. I mean, even if we assume that the AIs are basically equivalent to human simulations, they still get obvious advantages from the ability to be copy-pasted, the ability to be restored to a checkpoint, the ability to be run at higher clock speeds, and the ability to make credible pre-commitments, etc etc. I therefore certainly don’t think there is any plausible scenario in which unchecked AI systems wouldn’t end up with most of the power on earth. However, there is a meaningful difference between the scenario where their advantages mainly come from overwhelmingly great intelligence, and the scenario where their advantages mainly (or at least in large part) come from other sources. For example, scaleable oversight is a more realistic possibility in the latter scenario than it is in the former scenario. Boxing methods are also more realistic in the latter scenario than the former scenario, etc.
I think this scenario is still strategically isomorphic to “advantages mainly come from overwhelmingly great intelligence”. It’s intelligence at the level of a collective, rather than the individual level, but the conclusion is the same. For instance, scalable oversight of a group of AIs which is collectively far smarter than any group of humans is hard in basically the same ways as oversight of one highly-intelligent AI. Boxing the group of AIs is hard for the same reasons as boxing one. Etc.
I think the broad strokes are mostly similar, but that a bunch of relevant details are different.
Yes, a large collective of near-human AI that is allowed to interact freely over a (subjectively) long period of time is presumably roughly as hard to understand and control as a Bostrom/Yudkowsky-esque God in a box. However, in this scenario, we have the option to not allow free interaction between multiple instances, while still being able to extract useful work from them. It is also probably much easier to align a system that is not of overwhelming intelligence, and this could be done before the AIs are allowed to interact. We might also be able to significantly influence their collective behaviour by controlling the initial conditions of their interactions (similarly to how institutions and cultural norms have a substantial long-term impact on the trajectory of a country, for example). It is also more plausible that humans (or human simulations or emulations) could be kept in the loop for a long time period in this scenario. Moreover, if intelligence is bottle-necked by external resources (such as memory, data, CPU cycles, etc) rather than internal algorithmic efficiency, then you can exert more control over the resulting intelligence explosion by controlling those resources. Etc etc.