It’s not really novel. It is really just coupling together 3 ideas:
Well, I wasn’t trying to claim that it was ‘really novel’; the overall point there was more the question of why you’re pretty confident that the RSI procedure tops out at mildly superhuman.
I’m guessing, but my guess is that you have a mental image where ‘mildly superhuman’ is a pretty big space above ‘human-level’, rather than a narrow target to hit.
So to go back to arguments made in the interview we’ve been discussing, why isn’t this analogous to Go, like Eliezer argued:
Three days, there’s a quote from Guern about this, which I forget exactly, but it was something like, we know how long AlphaGo Zero, or AlphaZero, two different systems, was equivalent to a human Go player. And it was like 30 minutes on the following floor of this such and such DeepMind building. Maybe the first system doesn’t improve that quickly, and they build another system that does. And all of that with AlphaGo over the course of years, going from it takes a long time to train to it trains very quickly and without looking at the human playbook. That’s not with an artificial intelligence system that improves itself, or even that sort of like, get smarter as you run it, the way that human beings, not just as you evolve them, but as you run them over the course of their own lifetimes, improve. So if the first system doesn’t improve fast enough to kill everyone very quickly, they will build one that’s meant to spit out more gold than that.
To forestall the obvious objection, I’m not saying that Go is general intelligence; as you mentioned already, superhuman ability at special tasks like Go doesn’t automatically generalize to superhuman ability at anything else.
But you propose a framework to specifically bootstrap up to superhuman levels of general intelligence itself, including lots of task variety to get as much gain from cross-task generalization as possible, and also including the task of doing the bootstrapping itself.
So why is this going to stall out at, specifically, mildly superhuman rather than greatly superhuman intelligence? Why isn’t this more like Go, where the window during bootstrapping when it’s roughly human-level is about 30 minutes?
And, to reiterate some more of Eliezer’s points, supposing the first such system does turn out to top out at mildly superhuman, why wouldn’t we see another system in a small number of months/years which didn’t top out in that way?
I assume this is a general law for all intelligence. It is self evidently correct—on any task you can name, your gains scale with the log of effort.
This applies to limit cases. If you imagine a task performed by a human scale robot, say collecting apples, and you compare it to the average human, each increase in intelligence has a diminishing return on how many real apples/hour.
This is true for all tasks and all activities of humans.
A second reason is that there is a hard limit for future advances without collecting new scientific data. It has to do with noise in the data putting a limit on any processing algorithm extracting useful symbols from that data. (expressed mathematically with Shannon and others)
This is why I am completely confident that species killing bioweapons, or diamond MNT nanotechnology cannot be developed without a large amount of new scientific data and a large amount of new manipulation experiments. No “in a garage” solutions to the problems. The floor (minimum resources required) to get to a species killing bioweapon is higher, and the floor for a nanoforge is very high.
So viewed in this frame—you give the AI a coding optimization task, and it’s at the limit allowed by the provided computer + search time for a better self optimization. It might produce code that is 10% faster than the best humans.
You give it infinite compute (theoretically) and no new information. It is now 11% faster than the best humans.
This is an infinite superintelligence, a literal deity, but it cannot do better than 11% because the task won’t allow it. (or whatever, it’s a made up example, it doesn’t change my point if the number were 1000% and 1010%).
Another way to rephrase it is to compare a TSP solution made by a modern algorithm vs the NP complete solution you usually can’t find. The difference is usually very small.
So you’re not “threatened” by a machine that can do the latter.
Note also that an infinite superintelligence cannot solve MNT, even though it has the compute to play forward the universe by known laws of physics until it gets the present.
This is because with infinite compute there are many universes with differences in the laws of physics that match up perfectly to the observable present, and the machine doesn’t know which one it’s in, so it cannot design nanotechnology still—it doesn’t know the rules of physics well enough.
This also applies to “xanatos gambits” as well.
I usually don’t think of the limit like this but the above is generally correct.
Oh, because loss improvements logarithmically diminishes with the increase compute and data. [...]
This is true for all tasks and all activities of humans.
So, to make one of the simplest arguments at my disposal (ie, keeping to the OP we are discussing), why didn’t this argument apply to Go?
Relevant quote from OP:
And then another year, they threw out all the complexities and the training from human databases of Go games and built a new system, AlphaGo Zero, that trained itself from scratch. No looking at the human playbooks, no special purpose code, just a general purpose game player being specialized to Go, more or less. Three days, there’s a quote from Guern about this, which I forget exactly, but it was something like, we know how long AlphaGo Zero, or AlphaZero, two different systems, was equivalent to a human Go player. And it was like 30 minutes on the following floor of this such and such DeepMind building. Maybe the first system doesn’t improve that quickly, and they build another system that does. And all of that with AlphaGo over the course of years, going from it takes a long time to train to it trains very quickly and without looking at the human playbook. That’s not with an artificial intelligence system that improves itself,
(Whereas you propose a system that improves itself recursively in a much stronger sense.)
Not that I’m not arguing that Go engines lack the logarithmic return property you mention, but rather, Go engines stayed within the human-level window for a relatively short time DESPITE having diminishing returns similar to what you predict.
(Also note that I’m not claiming that Go playing is tantamount to AGI; rather, I’m asking why your argument doesn’t work for Go if it does work for AGI.)
So the question becomes, granting log returns or something similar, why do you anticipate that the mildly superhuman capability range is a broad one rather than narrow, when we average across lots and lots of tasks, when it lacks this property on (most) individual task-areas?
A second reason is that there is a hard limit for future advances without collecting new scientific data. It has to do with noise in the data putting a limit on any processing algorithm extracting useful symbols from that data. (expressed mathematically with Shannon and others)
This also has a super-standard Eliezer response, namely: yes, and that limit is extremely, extremely high. If we’re talking about the limit of what you can extrapolate from data using unbounded computation, it doesn’t keep you in the mildly-superhuman range.
And if we’re talking about what you can extract with bounded computation, then that takes us back to the previous point.
So viewed in this frame—you give the AI a coding optimization task, and it’s at the limit allowed by the provided computer + search time for a better self optimization. It might produce code that is 10% faster than the best humans.
You give it infinite compute (theoretically) and no new information. It is now 11% faster than the best humans.
This is an infinite superintelligence, a literal deity, but it cannot do better than 11% because the task won’t allow it. (or whatever, it’s a made up example, it doesn’t change my point if the number were 1000% and 1010%).
For the specific example of code optimization, more processing power totally eliminates the empirical bottleneck, since the system can go and actually simulate examples in order to check speed and correctness. So this is an especially good example of how the empirical bottleneck evaporates with enough processing power.
I agree that the actual speed improvement for the optimized code can’t go to infinity, since you can only optimize code so much. This is an example of diminishing returns due to the task itself having a bound. I think this general argument (that the task itself has a bound in how well you can do) is a central part of your confidence that diminishing returns will be ubiquitous.
But that final bottleneck should not give any confidence that ‘mildly superhuman’ is a broad rather than narrow band, if we think stuff that’s more than mildly superhuman can exist at all. Like, yes, something that compares to us as we compare to insects might only be able to make a sorting algorithm 90% faster or whatever. But that’s similar to observing that a God can’t make 2+2=3. The God could still split the world like a pea.
Note also that an infinite superintelligence cannot solve MNT, even though it has the compute to play forward the universe by known laws of physics until it gets the present.
This is because with infinite compute there are many universes with differences in the laws of physics that match up perfectly to the observable present, and the machine doesn’t know which one it’s in, so it cannot design nanotechnology still—it doesn’t know the rules of physics well enough.
It’s not clear to me whether this is correct, but I don’t think I need to argue that AI can solve nanotech to argue that it’s dangerous. I think an AI only needs to be a mildly superhuman politician plus engineer, to be deadly dangerous. (To eliminate nanotech from Eliezer’s example scenario, we can simply replace the nano-virus with a normal virus.)
This is why I am completely confident that species killing bioweapons, or diamond MNT nanotechnology cannot be developed without a large amount of new scientific data and a large amount of new manipulation experiments. No “in a garage” solutions to the problems. The floor (minimum resources required) to get to a species killing bioweapon is higher, and the floor for a nanoforge is very high.
I don’t get why you think the floor for species killing bioweapon is so high. Going back to the argument from the beginning of this comment, I think your argument here proves far too much. It seems like you are arguing that the generality of diminishing returns proves that nothing very much beyond current technology is possible without vastly more resources. Like, someone in the 1920s could have used your argument to prove the impossibility of atomic weapons, because clearly explosive power has diminishing returns to a broad variety of inputs, so even if governments put in hundreds of times the research, the result is only going to be bombs with a few times the explosive power.
Sometimes the returns just don’t diminish that fast.
Sometimes the returns just don’t diminish that fast.
I have a biology degree not mentioned on linkedin. I will say that I think for biology, the returns diminish faster. That is because bioscience knowledge from humans is mostly guesswork and low resolution information. Biology is very complex and the current laboratory science model I think fails to systematize gaining information in a useful way for most purposes. What this means is, you can get “results”, but not gain the information you would need to stop filling morgues with dead humans and animals, at least not without needing thousands of years at the current rate of progress.
I do not think an AGI can do a lot better for the reason that the data was never collected for most of it (the gene sequencing data is good, because it was collected via automation). I think that an AGI could control biology, for both good and bad, but it would need very large robotic facilities to systematize manipulating biology. Essentially it would have had to throw away almost all human knowledge, as there are hidden errors in it, and recreate all the information from scratch, keeping far more data from each experiment than is published in papers.
Using robots to perform the experiments and keeping data, especially for “negative” experiments, would give the information needed to actually get reliable results from manipulating biology, either for good or bad.
It means garage bioweapons aren’t possible. Yes, the last step of ordering synthetic DNA strands and preparing it could be done in a garage, but the information on human immunity at scale, or virion stability in air, or strategies to control mutations so that the lethal payload isn’t lost, requires information humans didn’t collect.
This poster calls this “Diminishing Marginal Returns”. Note that Diminishing marginal returns is empirical reality, it’s not merely an opinion, across most AI papers. (for humans, due to the inaccuracies in trying to assess IQ/talent, it’s difficult to falsify)
I agree that the actual speed improvement for the optimized code can’t go to infinity, since you can only optimize code so much. This is an example of diminishing returns due to the task itself having a bound. I think this general argument (that the task itself has a bound in how well you can do) is a central part of your confidence that diminishing returns will be ubiquitous.
This is where I think we break. How many dan is AlphaZero over the average human? How many dan is KataGo? I read it’s about 9 stones above humans.
What is the best possible agent at? 11?
Thinking of it as ‘stones’ illustrates what I am saying. In the physical world, intelligence gives a diminishing advantage. It could mean so long as humans are even still “in the running” with the aid of synthetic tools like open agency AI, we can defeat AI superintelligence in conflicts, even if that superintelligence is infinitely smart. We have to have a resource advantage—such as being allowed extra stones in the Go match—but we can win.
Eliezer assumes that the advantage of intelligence scales forever, when it obviously doesn’t. (note that this uses baked in assumptions. If say physics has a major useful exploit humans haven’t found, this breaks, the infinitely intelligent AI finds the exploit and tiles the universe)
And, to reiterate some more of Eliezer’s points, supposing the first such system does turn out to top out at mildly superhuman, why wouldn’t we see another system in a small number of months/years which didn’t top out in that way?
So the model is it becomes limited not by the algorithm directly, but by (compute, robotics, or data). Over the months/years, as more of each term is supplied, capabilities scale with the amount of supplied resources to whichever term is rate limiting.
A superintelligence requires logarithmically large amounts of resources to become a “high” superintelligence in all 3 terms. So literal mountain sized research labs (cubic kilometers of support equipment), buildings full of compute nodes (and gigawatts of power needed), and cubic kilometers of factory equipment.
This is very well pattern matched to every other technological advance humans have made, and the corresponding support equipment needed to fully exploit it. Notice how as tech became more advanced, the support footprint grew corespondingly.
In nature there are many examples of this. Nothing really fooms more than briefly. Every apparatus with exponential growth rapidly terminates for some reason. For example a nuke blasts itself apart, a supernova blasts itself apart, a bacteria colony runs out of food, water, ecological space, or oxygen.
Ultimately, yes. This whole debate is arguing that the critical threshold where it comes to this is farther away, and we humans should empower ourselves with helpful low superintelligences immediately.
It’s always better to be more powerful than helpless, which is the current situation. We are helpless to aging, death, pollution, resource shortages, enemy nations with nuclear weapons, disease, asteroid strikes, and so on. Hell just bad software—something the current llms are likely months from empowering us to fix.
And eliezer is saying not to take one more step towards fixing this because it MIGHT be hostile, when the entire universe is against us as it is. It already plans to kill us as it is, either from aging, or the inevitability of nuclear war over a long enough timespan, or the sun engulfing us.
eliezer is saying not to take one more step towards fixing this because it MIGHT be hostile
His position is to avoid taking one more step because it DEFINITELY kills everyone. I think it’s very clear that his position is not that it MIGHT be hostile.
Sure, and if there was some way to quantify the risks accurately I would agree with pausing AGI research if the expected value of the risks were less than the potential benefit.
Oh and pausing was even possible.
All it takes is a rival power, which there are several, or just a rival company and you have no choice. You must take the risk because it might be a poisoned banana or it might be giving the other primate a rocket launcher in a sticks and stones society.
This does explain why EY is so despondent. If he’s right it doesn’t matter, the AI wars have begun and only if it doesn’t work from a technical level will things slow down ever again.
Correctness of EY’s position (being infeasible to assess) is unrelated to the question of what EY’s position is, which is what I was commenting on.
When you argue against the position that AGI research should be stopped because it might be dangerous, there is no need to additionally claim that someone in particular holds that position, especially when it seems clear that they don’t.
Well, I wasn’t trying to claim that it was ‘really novel’; the overall point there was more the question of why you’re pretty confident that the RSI procedure tops out at mildly superhuman.
I’m guessing, but my guess is that you have a mental image where ‘mildly superhuman’ is a pretty big space above ‘human-level’, rather than a narrow target to hit.
So to go back to arguments made in the interview we’ve been discussing, why isn’t this analogous to Go, like Eliezer argued:
To forestall the obvious objection, I’m not saying that Go is general intelligence; as you mentioned already, superhuman ability at special tasks like Go doesn’t automatically generalize to superhuman ability at anything else.
But you propose a framework to specifically bootstrap up to superhuman levels of general intelligence itself, including lots of task variety to get as much gain from cross-task generalization as possible, and also including the task of doing the bootstrapping itself.
So why is this going to stall out at, specifically, mildly superhuman rather than greatly superhuman intelligence? Why isn’t this more like Go, where the window during bootstrapping when it’s roughly human-level is about 30 minutes?
And, to reiterate some more of Eliezer’s points, supposing the first such system does turn out to top out at mildly superhuman, why wouldn’t we see another system in a small number of months/years which didn’t top out in that way?
Oh, because loss improvements logarithmically diminishes with the increase compute and data. https://arxiv.org/pdf/2001.08361.pdf
I assume this is a general law for all intelligence. It is self evidently correct—on any task you can name, your gains scale with the log of effort.
This applies to limit cases. If you imagine a task performed by a human scale robot, say collecting apples, and you compare it to the average human, each increase in intelligence has a diminishing return on how many real apples/hour.
This is true for all tasks and all activities of humans.
A second reason is that there is a hard limit for future advances without collecting new scientific data. It has to do with noise in the data putting a limit on any processing algorithm extracting useful symbols from that data. (expressed mathematically with Shannon and others)
This is why I am completely confident that species killing bioweapons, or diamond MNT nanotechnology cannot be developed without a large amount of new scientific data and a large amount of new manipulation experiments. No “in a garage” solutions to the problems. The floor (minimum resources required) to get to a species killing bioweapon is higher, and the floor for a nanoforge is very high.
So viewed in this frame—you give the AI a coding optimization task, and it’s at the limit allowed by the provided computer + search time for a better self optimization. It might produce code that is 10% faster than the best humans.
You give it infinite compute (theoretically) and no new information. It is now 11% faster than the best humans.
This is an infinite superintelligence, a literal deity, but it cannot do better than 11% because the task won’t allow it. (or whatever, it’s a made up example, it doesn’t change my point if the number were 1000% and 1010%).
Another way to rephrase it is to compare a TSP solution made by a modern algorithm vs the NP complete solution you usually can’t find. The difference is usually very small.
So you’re not “threatened” by a machine that can do the latter.
Note also that an infinite superintelligence cannot solve MNT, even though it has the compute to play forward the universe by known laws of physics until it gets the present.
This is because with infinite compute there are many universes with differences in the laws of physics that match up perfectly to the observable present, and the machine doesn’t know which one it’s in, so it cannot design nanotechnology still—it doesn’t know the rules of physics well enough.
This also applies to “xanatos gambits” as well.
I usually don’t think of the limit like this but the above is generally correct.
So, to make one of the simplest arguments at my disposal (ie, keeping to the OP we are discussing), why didn’t this argument apply to Go?
Relevant quote from OP:
(Whereas you propose a system that improves itself recursively in a much stronger sense.)
Not that I’m not arguing that Go engines lack the logarithmic return property you mention, but rather, Go engines stayed within the human-level window for a relatively short time DESPITE having diminishing returns similar to what you predict.
(Also note that I’m not claiming that Go playing is tantamount to AGI; rather, I’m asking why your argument doesn’t work for Go if it does work for AGI.)
So the question becomes, granting log returns or something similar, why do you anticipate that the mildly superhuman capability range is a broad one rather than narrow, when we average across lots and lots of tasks, when it lacks this property on (most) individual task-areas?
This also has a super-standard Eliezer response, namely: yes, and that limit is extremely, extremely high. If we’re talking about the limit of what you can extrapolate from data using unbounded computation, it doesn’t keep you in the mildly-superhuman range.
And if we’re talking about what you can extract with bounded computation, then that takes us back to the previous point.
For the specific example of code optimization, more processing power totally eliminates the empirical bottleneck, since the system can go and actually simulate examples in order to check speed and correctness. So this is an especially good example of how the empirical bottleneck evaporates with enough processing power.
I agree that the actual speed improvement for the optimized code can’t go to infinity, since you can only optimize code so much. This is an example of diminishing returns due to the task itself having a bound. I think this general argument (that the task itself has a bound in how well you can do) is a central part of your confidence that diminishing returns will be ubiquitous.
But that final bottleneck should not give any confidence that ‘mildly superhuman’ is a broad rather than narrow band, if we think stuff that’s more than mildly superhuman can exist at all. Like, yes, something that compares to us as we compare to insects might only be able to make a sorting algorithm 90% faster or whatever. But that’s similar to observing that a God can’t make 2+2=3. The God could still split the world like a pea.
It’s not clear to me whether this is correct, but I don’t think I need to argue that AI can solve nanotech to argue that it’s dangerous. I think an AI only needs to be a mildly superhuman politician plus engineer, to be deadly dangerous. (To eliminate nanotech from Eliezer’s example scenario, we can simply replace the nano-virus with a normal virus.)
I don’t get why you think the floor for species killing bioweapon is so high. Going back to the argument from the beginning of this comment, I think your argument here proves far too much. It seems like you are arguing that the generality of diminishing returns proves that nothing very much beyond current technology is possible without vastly more resources. Like, someone in the 1920s could have used your argument to prove the impossibility of atomic weapons, because clearly explosive power has diminishing returns to a broad variety of inputs, so even if governments put in hundreds of times the research, the result is only going to be bombs with a few times the explosive power.
Sometimes the returns just don’t diminish that fast.
Sometimes the returns just don’t diminish that fast.
I have a biology degree not mentioned on linkedin. I will say that I think for biology, the returns diminish faster. That is because bioscience knowledge from humans is mostly guesswork and low resolution information. Biology is very complex and the current laboratory science model I think fails to systematize gaining information in a useful way for most purposes. What this means is, you can get “results”, but not gain the information you would need to stop filling morgues with dead humans and animals, at least not without needing thousands of years at the current rate of progress.
I do not think an AGI can do a lot better for the reason that the data was never collected for most of it (the gene sequencing data is good, because it was collected via automation). I think that an AGI could control biology, for both good and bad, but it would need very large robotic facilities to systematize manipulating biology. Essentially it would have had to throw away almost all human knowledge, as there are hidden errors in it, and recreate all the information from scratch, keeping far more data from each experiment than is published in papers.
Using robots to perform the experiments and keeping data, especially for “negative” experiments, would give the information needed to actually get reliable results from manipulating biology, either for good or bad.
It means garage bioweapons aren’t possible. Yes, the last step of ordering synthetic DNA strands and preparing it could be done in a garage, but the information on human immunity at scale, or virion stability in air, or strategies to control mutations so that the lethal payload isn’t lost, requires information humans didn’t collect.
Same issue with nanotechnology.
Update : https://www.lesswrong.com/posts/jdLmC46ZuXS54LKzL/why-i-m-sceptical-of-foom
This poster calls this “Diminishing Marginal Returns”. Note that Diminishing marginal returns is empirical reality, it’s not merely an opinion, across most AI papers. (for humans, due to the inaccuracies in trying to assess IQ/talent, it’s difficult to falsify)
I agree that the actual speed improvement for the optimized code can’t go to infinity, since you can only optimize code so much. This is an example of diminishing returns due to the task itself having a bound. I think this general argument (that the task itself has a bound in how well you can do) is a central part of your confidence that diminishing returns will be ubiquitous.
This is where I think we break. How many dan is AlphaZero over the average human? How many dan is KataGo? I read it’s about 9 stones above humans.
What is the best possible agent at? 11?
Thinking of it as ‘stones’ illustrates what I am saying. In the physical world, intelligence gives a diminishing advantage. It could mean so long as humans are even still “in the running” with the aid of synthetic tools like open agency AI, we can defeat AI superintelligence in conflicts, even if that superintelligence is infinitely smart. We have to have a resource advantage—such as being allowed extra stones in the Go match—but we can win.
Eliezer assumes that the advantage of intelligence scales forever, when it obviously doesn’t. (note that this uses baked in assumptions. If say physics has a major useful exploit humans haven’t found, this breaks, the infinitely intelligent AI finds the exploit and tiles the universe)
And, to reiterate some more of Eliezer’s points, supposing the first such system does turn out to top out at mildly superhuman, why wouldn’t we see another system in a small number of months/years which didn’t top out in that way?
So the model is it becomes limited not by the algorithm directly, but by (compute, robotics, or data). Over the months/years, as more of each term is supplied, capabilities scale with the amount of supplied resources to whichever term is rate limiting.
A superintelligence requires logarithmically large amounts of resources to become a “high” superintelligence in all 3 terms. So literal mountain sized research labs (cubic kilometers of support equipment), buildings full of compute nodes (and gigawatts of power needed), and cubic kilometers of factory equipment.
This is very well pattern matched to every other technological advance humans have made, and the corresponding support equipment needed to fully exploit it. Notice how as tech became more advanced, the support footprint grew corespondingly.
In nature there are many examples of this. Nothing really fooms more than briefly. Every apparatus with exponential growth rapidly terminates for some reason. For example a nuke blasts itself apart, a supernova blasts itself apart, a bacteria colony runs out of food, water, ecological space, or oxygen.
For AGI, the speed of light.
Ultimately, yes. This whole debate is arguing that the critical threshold where it comes to this is farther away, and we humans should empower ourselves with helpful low superintelligences immediately.
It’s always better to be more powerful than helpless, which is the current situation. We are helpless to aging, death, pollution, resource shortages, enemy nations with nuclear weapons, disease, asteroid strikes, and so on. Hell just bad software—something the current llms are likely months from empowering us to fix.
And eliezer is saying not to take one more step towards fixing this because it MIGHT be hostile, when the entire universe is against us as it is. It already plans to kill us as it is, either from aging, or the inevitability of nuclear war over a long enough timespan, or the sun engulfing us.
His position is to avoid taking one more step because it DEFINITELY kills everyone. I think it’s very clear that his position is not that it MIGHT be hostile.
(My position is that there might be some steps that don’t kill everyone immediately, but probably still do immediately thereafter, while giving a bit more of a chance than doing all the other things that do kill us directly. Doing none of these things would be preferable, because at least aging doesn’t kill the civilization, but Moloch is the one in charge.)
Sure, and if there was some way to quantify the risks accurately I would agree with pausing AGI research if the expected value of the risks were less than the potential benefit.
Oh and pausing was even possible.
All it takes is a rival power, which there are several, or just a rival company and you have no choice. You must take the risk because it might be a poisoned banana or it might be giving the other primate a rocket launcher in a sticks and stones society.
This does explain why EY is so despondent. If he’s right it doesn’t matter, the AI wars have begun and only if it doesn’t work from a technical level will things slow down ever again.
Correctness of EY’s position (being infeasible to assess) is unrelated to the question of what EY’s position is, which is what I was commenting on.
When you argue against the position that AGI research should be stopped because it might be dangerous, there is no need to additionally claim that someone in particular holds that position, especially when it seems clear that they don’t.