There are a bunch of baked assumptions here from EY. Remember he came up with many of these ideas years ago, before deep learning existed.
(1) the AGIs have to be agentic with a global score. This is not true, but very early AI agents often did work this way. Take one of the simplest possible RL agents, the q-learner. All it does is pick the action that has the maximum discounted reward. Thus the q-learner learns it’s environment, filling out an array in memory called the q-table, and then just does whatever it’s source code told it has the max reward. (some of the first successful deep learning papers just replaced that array with a neural network)
You could imagine building an “embodied” robot that from the moment you switch it on, it always tries to make that “reward” number go ever higher, the same way.
This kind of AGI is likely lethally dangerous.
(2) Intelligence scales very high. In a simple game, intelligence has diminishing returns that collapse to zero. (once you have enough intelligence to solve a task, you have 0 error gradient or reason to develop any more)
In more complex games (including reality), intelligence goes further, but there always is a limit. For example, if you think about a task like “primate locates and picks apples”, ever more intelligence can make the primate more efficient at searching for the apple, or to take a more efficient path towards reaching and grasping the apple. But it’s logarithmically diminishing returns, and no amount of intelligence will let the primate find an apple if it’s paralyzed or unable to explore at least some of the forest. Nor can it instruct another primate to find the apple for it if the paralyzed one has never seen the forest at all.
Note also that in reality, an agent’s reward equals (resource gain—resource cost). One term in ‘resource cost’ is the cost of compute. Hence, for example you would not want to make a robot that mines copper too smart as adding more and more cognitive capacity adds less and less incremental efficiency gain in how much copper it collects, but costs more and more compute to realize. Similarly there is no reason to train the agent in simulation past a certain point, for the same cost reason. Intelligence stops adding marginal net utility.
EY posits that technologies that we think will probably take methodically improvements and careful experiments on a very large scale to develop could be “leapfrogged” by just skipping direct to advanced capabilities. For example, diamondoid nanotechnology not from carefully studying small assemblies of diamond on a large scale, and methodically working up the tool chain, at a large scale using many billions of dollars of equipment, but instead just hacking it direct from hijacking biology.
From an agent that has no direct experimental data with biology—EY gives examples where the AGI has done everything in sim. Note EY has never been to high school or college per wikipedia. He may be an extreme edge of the bell curve genius, but there may be small flaws in his knowledge base that are leading to these faulty assumptions. Which is exactly the problem an AGI with infinite compute but no empirical data not regurgitated from humans would have. It would model biology and the nanoscale using all human papers, but small errors would cause the simulation to diverge from reality, causing the AGI to make plans based on nonsense. (see how RL agents exploit environments by exploiting flaws in the physics sim for an example of this)
There are a bunch of baked assumptions here from EY. Remember he came up with many of these ideas years ago, before deep learning existed.
(1) the AGIs have to be agentic with a global score. This is not true, but very early AI agents often did work this way. Take one of the simplest possible RL agents, the q-learner. All it does is pick the action that has the maximum discounted reward. Thus the q-learner learns it’s environment, filling out an array in memory called the q-table, and then just does whatever it’s source code told it has the max reward. (some of the first successful deep learning papers just replaced that array with a neural network)
You could imagine building an “embodied” robot that from the moment you switch it on, it always tries to make that “reward” number go ever higher, the same way.
This kind of AGI is likely lethally dangerous.
(2) Intelligence scales very high. In a simple game, intelligence has diminishing returns that collapse to zero. (once you have enough intelligence to solve a task, you have 0 error gradient or reason to develop any more)
In more complex games (including reality), intelligence goes further, but there always is a limit. For example, if you think about a task like “primate locates and picks apples”, ever more intelligence can make the primate more efficient at searching for the apple, or to take a more efficient path towards reaching and grasping the apple. But it’s logarithmically diminishing returns, and no amount of intelligence will let the primate find an apple if it’s paralyzed or unable to explore at least some of the forest. Nor can it instruct another primate to find the apple for it if the paralyzed one has never seen the forest at all.
Note also that in reality, an agent’s reward equals (resource gain—resource cost). One term in ‘resource cost’ is the cost of compute. Hence, for example you would not want to make a robot that mines copper too smart as adding more and more cognitive capacity adds less and less incremental efficiency gain in how much copper it collects, but costs more and more compute to realize. Similarly there is no reason to train the agent in simulation past a certain point, for the same cost reason. Intelligence stops adding marginal net utility.
EY posits that technologies that we think will probably take methodically improvements and careful experiments on a very large scale to develop could be “leapfrogged” by just skipping direct to advanced capabilities. For example, diamondoid nanotechnology not from carefully studying small assemblies of diamond on a large scale, and methodically working up the tool chain, at a large scale using many billions of dollars of equipment, but instead just hacking it direct from hijacking biology.
From an agent that has no direct experimental data with biology—EY gives examples where the AGI has done everything in sim. Note EY has never been to high school or college per wikipedia. He may be an extreme edge of the bell curve genius, but there may be small flaws in his knowledge base that are leading to these faulty assumptions. Which is exactly the problem an AGI with infinite compute but no empirical data not regurgitated from humans would have. It would model biology and the nanoscale using all human papers, but small errors would cause the simulation to diverge from reality, causing the AGI to make plans based on nonsense. (see how RL agents exploit environments by exploiting flaws in the physics sim for an example of this)