I assume this is a general law for all intelligence. It is self evidently correct—on any task you can name, your gains scale with the log of effort.
This applies to limit cases. If you imagine a task performed by a human scale robot, say collecting apples, and you compare it to the average human, each increase in intelligence has a diminishing return on how many real apples/hour.
This is true for all tasks and all activities of humans.
A second reason is that there is a hard limit for future advances without collecting new scientific data. It has to do with noise in the data putting a limit on any processing algorithm extracting useful symbols from that data. (expressed mathematically with Shannon and others)
This is why I am completely confident that species killing bioweapons, or diamond MNT nanotechnology cannot be developed without a large amount of new scientific data and a large amount of new manipulation experiments. No “in a garage” solutions to the problems. The floor (minimum resources required) to get to a species killing bioweapon is higher, and the floor for a nanoforge is very high.
So viewed in this frame—you give the AI a coding optimization task, and it’s at the limit allowed by the provided computer + search time for a better self optimization. It might produce code that is 10% faster than the best humans.
You give it infinite compute (theoretically) and no new information. It is now 11% faster than the best humans.
This is an infinite superintelligence, a literal deity, but it cannot do better than 11% because the task won’t allow it. (or whatever, it’s a made up example, it doesn’t change my point if the number were 1000% and 1010%).
Another way to rephrase it is to compare a TSP solution made by a modern algorithm vs the NP complete solution you usually can’t find. The difference is usually very small.
So you’re not “threatened” by a machine that can do the latter.
Note also that an infinite superintelligence cannot solve MNT, even though it has the compute to play forward the universe by known laws of physics until it gets the present.
This is because with infinite compute there are many universes with differences in the laws of physics that match up perfectly to the observable present, and the machine doesn’t know which one it’s in, so it cannot design nanotechnology still—it doesn’t know the rules of physics well enough.
This also applies to “xanatos gambits” as well.
I usually don’t think of the limit like this but the above is generally correct.
Oh, because loss improvements logarithmically diminishes with the increase compute and data. [...]
This is true for all tasks and all activities of humans.
So, to make one of the simplest arguments at my disposal (ie, keeping to the OP we are discussing), why didn’t this argument apply to Go?
Relevant quote from OP:
And then another year, they threw out all the complexities and the training from human databases of Go games and built a new system, AlphaGo Zero, that trained itself from scratch. No looking at the human playbooks, no special purpose code, just a general purpose game player being specialized to Go, more or less. Three days, there’s a quote from Guern about this, which I forget exactly, but it was something like, we know how long AlphaGo Zero, or AlphaZero, two different systems, was equivalent to a human Go player. And it was like 30 minutes on the following floor of this such and such DeepMind building. Maybe the first system doesn’t improve that quickly, and they build another system that does. And all of that with AlphaGo over the course of years, going from it takes a long time to train to it trains very quickly and without looking at the human playbook. That’s not with an artificial intelligence system that improves itself,
(Whereas you propose a system that improves itself recursively in a much stronger sense.)
Not that I’m not arguing that Go engines lack the logarithmic return property you mention, but rather, Go engines stayed within the human-level window for a relatively short time DESPITE having diminishing returns similar to what you predict.
(Also note that I’m not claiming that Go playing is tantamount to AGI; rather, I’m asking why your argument doesn’t work for Go if it does work for AGI.)
So the question becomes, granting log returns or something similar, why do you anticipate that the mildly superhuman capability range is a broad one rather than narrow, when we average across lots and lots of tasks, when it lacks this property on (most) individual task-areas?
A second reason is that there is a hard limit for future advances without collecting new scientific data. It has to do with noise in the data putting a limit on any processing algorithm extracting useful symbols from that data. (expressed mathematically with Shannon and others)
This also has a super-standard Eliezer response, namely: yes, and that limit is extremely, extremely high. If we’re talking about the limit of what you can extrapolate from data using unbounded computation, it doesn’t keep you in the mildly-superhuman range.
And if we’re talking about what you can extract with bounded computation, then that takes us back to the previous point.
So viewed in this frame—you give the AI a coding optimization task, and it’s at the limit allowed by the provided computer + search time for a better self optimization. It might produce code that is 10% faster than the best humans.
You give it infinite compute (theoretically) and no new information. It is now 11% faster than the best humans.
This is an infinite superintelligence, a literal deity, but it cannot do better than 11% because the task won’t allow it. (or whatever, it’s a made up example, it doesn’t change my point if the number were 1000% and 1010%).
For the specific example of code optimization, more processing power totally eliminates the empirical bottleneck, since the system can go and actually simulate examples in order to check speed and correctness. So this is an especially good example of how the empirical bottleneck evaporates with enough processing power.
I agree that the actual speed improvement for the optimized code can’t go to infinity, since you can only optimize code so much. This is an example of diminishing returns due to the task itself having a bound. I think this general argument (that the task itself has a bound in how well you can do) is a central part of your confidence that diminishing returns will be ubiquitous.
But that final bottleneck should not give any confidence that ‘mildly superhuman’ is a broad rather than narrow band, if we think stuff that’s more than mildly superhuman can exist at all. Like, yes, something that compares to us as we compare to insects might only be able to make a sorting algorithm 90% faster or whatever. But that’s similar to observing that a God can’t make 2+2=3. The God could still split the world like a pea.
Note also that an infinite superintelligence cannot solve MNT, even though it has the compute to play forward the universe by known laws of physics until it gets the present.
This is because with infinite compute there are many universes with differences in the laws of physics that match up perfectly to the observable present, and the machine doesn’t know which one it’s in, so it cannot design nanotechnology still—it doesn’t know the rules of physics well enough.
It’s not clear to me whether this is correct, but I don’t think I need to argue that AI can solve nanotech to argue that it’s dangerous. I think an AI only needs to be a mildly superhuman politician plus engineer, to be deadly dangerous. (To eliminate nanotech from Eliezer’s example scenario, we can simply replace the nano-virus with a normal virus.)
This is why I am completely confident that species killing bioweapons, or diamond MNT nanotechnology cannot be developed without a large amount of new scientific data and a large amount of new manipulation experiments. No “in a garage” solutions to the problems. The floor (minimum resources required) to get to a species killing bioweapon is higher, and the floor for a nanoforge is very high.
I don’t get why you think the floor for species killing bioweapon is so high. Going back to the argument from the beginning of this comment, I think your argument here proves far too much. It seems like you are arguing that the generality of diminishing returns proves that nothing very much beyond current technology is possible without vastly more resources. Like, someone in the 1920s could have used your argument to prove the impossibility of atomic weapons, because clearly explosive power has diminishing returns to a broad variety of inputs, so even if governments put in hundreds of times the research, the result is only going to be bombs with a few times the explosive power.
Sometimes the returns just don’t diminish that fast.
Sometimes the returns just don’t diminish that fast.
I have a biology degree not mentioned on linkedin. I will say that I think for biology, the returns diminish faster. That is because bioscience knowledge from humans is mostly guesswork and low resolution information. Biology is very complex and the current laboratory science model I think fails to systematize gaining information in a useful way for most purposes. What this means is, you can get “results”, but not gain the information you would need to stop filling morgues with dead humans and animals, at least not without needing thousands of years at the current rate of progress.
I do not think an AGI can do a lot better for the reason that the data was never collected for most of it (the gene sequencing data is good, because it was collected via automation). I think that an AGI could control biology, for both good and bad, but it would need very large robotic facilities to systematize manipulating biology. Essentially it would have had to throw away almost all human knowledge, as there are hidden errors in it, and recreate all the information from scratch, keeping far more data from each experiment than is published in papers.
Using robots to perform the experiments and keeping data, especially for “negative” experiments, would give the information needed to actually get reliable results from manipulating biology, either for good or bad.
It means garage bioweapons aren’t possible. Yes, the last step of ordering synthetic DNA strands and preparing it could be done in a garage, but the information on human immunity at scale, or virion stability in air, or strategies to control mutations so that the lethal payload isn’t lost, requires information humans didn’t collect.
This poster calls this “Diminishing Marginal Returns”. Note that Diminishing marginal returns is empirical reality, it’s not merely an opinion, across most AI papers. (for humans, due to the inaccuracies in trying to assess IQ/talent, it’s difficult to falsify)
I agree that the actual speed improvement for the optimized code can’t go to infinity, since you can only optimize code so much. This is an example of diminishing returns due to the task itself having a bound. I think this general argument (that the task itself has a bound in how well you can do) is a central part of your confidence that diminishing returns will be ubiquitous.
This is where I think we break. How many dan is AlphaZero over the average human? How many dan is KataGo? I read it’s about 9 stones above humans.
What is the best possible agent at? 11?
Thinking of it as ‘stones’ illustrates what I am saying. In the physical world, intelligence gives a diminishing advantage. It could mean so long as humans are even still “in the running” with the aid of synthetic tools like open agency AI, we can defeat AI superintelligence in conflicts, even if that superintelligence is infinitely smart. We have to have a resource advantage—such as being allowed extra stones in the Go match—but we can win.
Eliezer assumes that the advantage of intelligence scales forever, when it obviously doesn’t. (note that this uses baked in assumptions. If say physics has a major useful exploit humans haven’t found, this breaks, the infinitely intelligent AI finds the exploit and tiles the universe)
Oh, because loss improvements logarithmically diminishes with the increase compute and data. https://arxiv.org/pdf/2001.08361.pdf
I assume this is a general law for all intelligence. It is self evidently correct—on any task you can name, your gains scale with the log of effort.
This applies to limit cases. If you imagine a task performed by a human scale robot, say collecting apples, and you compare it to the average human, each increase in intelligence has a diminishing return on how many real apples/hour.
This is true for all tasks and all activities of humans.
A second reason is that there is a hard limit for future advances without collecting new scientific data. It has to do with noise in the data putting a limit on any processing algorithm extracting useful symbols from that data. (expressed mathematically with Shannon and others)
This is why I am completely confident that species killing bioweapons, or diamond MNT nanotechnology cannot be developed without a large amount of new scientific data and a large amount of new manipulation experiments. No “in a garage” solutions to the problems. The floor (minimum resources required) to get to a species killing bioweapon is higher, and the floor for a nanoforge is very high.
So viewed in this frame—you give the AI a coding optimization task, and it’s at the limit allowed by the provided computer + search time for a better self optimization. It might produce code that is 10% faster than the best humans.
You give it infinite compute (theoretically) and no new information. It is now 11% faster than the best humans.
This is an infinite superintelligence, a literal deity, but it cannot do better than 11% because the task won’t allow it. (or whatever, it’s a made up example, it doesn’t change my point if the number were 1000% and 1010%).
Another way to rephrase it is to compare a TSP solution made by a modern algorithm vs the NP complete solution you usually can’t find. The difference is usually very small.
So you’re not “threatened” by a machine that can do the latter.
Note also that an infinite superintelligence cannot solve MNT, even though it has the compute to play forward the universe by known laws of physics until it gets the present.
This is because with infinite compute there are many universes with differences in the laws of physics that match up perfectly to the observable present, and the machine doesn’t know which one it’s in, so it cannot design nanotechnology still—it doesn’t know the rules of physics well enough.
This also applies to “xanatos gambits” as well.
I usually don’t think of the limit like this but the above is generally correct.
So, to make one of the simplest arguments at my disposal (ie, keeping to the OP we are discussing), why didn’t this argument apply to Go?
Relevant quote from OP:
(Whereas you propose a system that improves itself recursively in a much stronger sense.)
Not that I’m not arguing that Go engines lack the logarithmic return property you mention, but rather, Go engines stayed within the human-level window for a relatively short time DESPITE having diminishing returns similar to what you predict.
(Also note that I’m not claiming that Go playing is tantamount to AGI; rather, I’m asking why your argument doesn’t work for Go if it does work for AGI.)
So the question becomes, granting log returns or something similar, why do you anticipate that the mildly superhuman capability range is a broad one rather than narrow, when we average across lots and lots of tasks, when it lacks this property on (most) individual task-areas?
This also has a super-standard Eliezer response, namely: yes, and that limit is extremely, extremely high. If we’re talking about the limit of what you can extrapolate from data using unbounded computation, it doesn’t keep you in the mildly-superhuman range.
And if we’re talking about what you can extract with bounded computation, then that takes us back to the previous point.
For the specific example of code optimization, more processing power totally eliminates the empirical bottleneck, since the system can go and actually simulate examples in order to check speed and correctness. So this is an especially good example of how the empirical bottleneck evaporates with enough processing power.
I agree that the actual speed improvement for the optimized code can’t go to infinity, since you can only optimize code so much. This is an example of diminishing returns due to the task itself having a bound. I think this general argument (that the task itself has a bound in how well you can do) is a central part of your confidence that diminishing returns will be ubiquitous.
But that final bottleneck should not give any confidence that ‘mildly superhuman’ is a broad rather than narrow band, if we think stuff that’s more than mildly superhuman can exist at all. Like, yes, something that compares to us as we compare to insects might only be able to make a sorting algorithm 90% faster or whatever. But that’s similar to observing that a God can’t make 2+2=3. The God could still split the world like a pea.
It’s not clear to me whether this is correct, but I don’t think I need to argue that AI can solve nanotech to argue that it’s dangerous. I think an AI only needs to be a mildly superhuman politician plus engineer, to be deadly dangerous. (To eliminate nanotech from Eliezer’s example scenario, we can simply replace the nano-virus with a normal virus.)
I don’t get why you think the floor for species killing bioweapon is so high. Going back to the argument from the beginning of this comment, I think your argument here proves far too much. It seems like you are arguing that the generality of diminishing returns proves that nothing very much beyond current technology is possible without vastly more resources. Like, someone in the 1920s could have used your argument to prove the impossibility of atomic weapons, because clearly explosive power has diminishing returns to a broad variety of inputs, so even if governments put in hundreds of times the research, the result is only going to be bombs with a few times the explosive power.
Sometimes the returns just don’t diminish that fast.
Sometimes the returns just don’t diminish that fast.
I have a biology degree not mentioned on linkedin. I will say that I think for biology, the returns diminish faster. That is because bioscience knowledge from humans is mostly guesswork and low resolution information. Biology is very complex and the current laboratory science model I think fails to systematize gaining information in a useful way for most purposes. What this means is, you can get “results”, but not gain the information you would need to stop filling morgues with dead humans and animals, at least not without needing thousands of years at the current rate of progress.
I do not think an AGI can do a lot better for the reason that the data was never collected for most of it (the gene sequencing data is good, because it was collected via automation). I think that an AGI could control biology, for both good and bad, but it would need very large robotic facilities to systematize manipulating biology. Essentially it would have had to throw away almost all human knowledge, as there are hidden errors in it, and recreate all the information from scratch, keeping far more data from each experiment than is published in papers.
Using robots to perform the experiments and keeping data, especially for “negative” experiments, would give the information needed to actually get reliable results from manipulating biology, either for good or bad.
It means garage bioweapons aren’t possible. Yes, the last step of ordering synthetic DNA strands and preparing it could be done in a garage, but the information on human immunity at scale, or virion stability in air, or strategies to control mutations so that the lethal payload isn’t lost, requires information humans didn’t collect.
Same issue with nanotechnology.
Update : https://www.lesswrong.com/posts/jdLmC46ZuXS54LKzL/why-i-m-sceptical-of-foom
This poster calls this “Diminishing Marginal Returns”. Note that Diminishing marginal returns is empirical reality, it’s not merely an opinion, across most AI papers. (for humans, due to the inaccuracies in trying to assess IQ/talent, it’s difficult to falsify)
I agree that the actual speed improvement for the optimized code can’t go to infinity, since you can only optimize code so much. This is an example of diminishing returns due to the task itself having a bound. I think this general argument (that the task itself has a bound in how well you can do) is a central part of your confidence that diminishing returns will be ubiquitous.
This is where I think we break. How many dan is AlphaZero over the average human? How many dan is KataGo? I read it’s about 9 stones above humans.
What is the best possible agent at? 11?
Thinking of it as ‘stones’ illustrates what I am saying. In the physical world, intelligence gives a diminishing advantage. It could mean so long as humans are even still “in the running” with the aid of synthetic tools like open agency AI, we can defeat AI superintelligence in conflicts, even if that superintelligence is infinitely smart. We have to have a resource advantage—such as being allowed extra stones in the Go match—but we can win.
Eliezer assumes that the advantage of intelligence scales forever, when it obviously doesn’t. (note that this uses baked in assumptions. If say physics has a major useful exploit humans haven’t found, this breaks, the infinitely intelligent AI finds the exploit and tiles the universe)