I feel like this and many other arguments for AI-skepticism are implicitly assuming AGI that is amazingly dumb and then proving that there is no need to worry about this dumb superintelligence.
Remember the old “AI will never beat humans at every task because there isn’t one architecture that is optimal at every task. An AI optimised to play chess won’t be great at trading stocks (or whatever) and vice versa”? Well, I’m capable of running a different program on my computer depending on the task at hand. If your AGI can’t do the same as a random idiot with a PC, it’s not really AGI.
I am emphatically not saying that Robin Hanson has ever made this particular blunder but I think he’s making a more subtle one in the same vein.
Sure, if you think of AGI as a collection of image recognisers and go engines etc. then there is no ironclad argument for FOOM. But the moment (and probably sooner) that it becomes capable of actual general problem solving on par with it’s creators (i.e. actual AGI) and turns its powers to recursive self-improvement—how can that result in anything but FOOM? Doesn’t matter if further improvements require more complexity or less complexity or a different kind of complexity or whatever. If human researchers can do it then AGI can do it faster and better because it scales better, doesn’t sleep, doesn’t eat and doesn’t waste time arguing with people on facebook.
This must have been said a million times already. Is this not obvious? What am I missing?
Sure, if you think of AGI as a collection of image recognisers and go engines etc. then there is no ironclad argument for FOOM. But the moment (and probably sooner) that it becomes capable of actual general problem solving on par with it’s creators (i.e. actual AGI) and turns its powers to recursive self-improvement—how can that result in anything but FOOM?
The main thing that would predict slower takeoff is if early AGI systems turn out to be extremely computationally expensive. The MIRI people I’ve talked to about this are actually skeptical because they think we’re already in hardware overhang mode.
This isn’t obvious from a simple comparison of the world’s available computing hardware to the apparent requirements of human brains, though; it requires going into questions like, “Given that the human brain wasn’t built to e.g. convert compute into solving problems in biochemistry, how many of the categories of things the brain is doing do you need if you are just trying to convert compute into solving problems in biochemistry?”
If the first AGI systems are slow and expensive to run, then it could take time to use them to make major hardware or software improvements, particularly if the first AGI systems are complicated kludges in the fashion of human brains. (Though this is a really likely world-gets-destroyed scenario, because you probably can’t align something if you don’t understand its core algorithms, how it decomposes problems into subproblems, what kinds of cognitive work different parts of the system are doing and how this work contribute to the desired outcome, etc.)
If human researchers can do it then AGI can do it faster and better because it scales better, doesn’t sleep, doesn’t eat and doesn’t waste time arguing with people on facebook.
Agreed. This is a really good way of stating “humans brains aren’t efficiently converting compute into research”; those little distractions, confirmation biases, losses of motivation, coffee breaks, rationalizations, etc. add up fast.
The main thing that would predict slower takeoff is if early AGI systems turn out to be extremely computationally expensive.
Surely that’s only under the assumption that Eliezer’s conception of AGI (simple general optimisation algorithm) is right, and Robin’s (very many separate modules comprising a big intricate system) is wrong? Is it just that you think that assumption is pretty certain to be right? Or, are you saying that even under the Hansonian model of AI, we’d still get a FOOM anyway?
I wouldn’t say that the first AGI systems are likely to be “simple.” I’d say they’re likely to be much more complex than typical narrow systems today (though shooting for relative simplicity is a good idea for safety/robustness reasons).
Humans didn’t evolve separate specialized modules for doing theoretical physics, chemistry, computer science, etc.; indeed, we didn’t undergo selection for any of those capacities at all, they just naturally fell out of a different set of capacities we were being selected for. So if the separate-modules proposal is that we’re likely to figure out how to achieve par-human chemistry without being able to achieve par-human mechanical engineering at more or less the same time, then yeah, I feel confident that’s not how things will shake out.
I think that “general” reasoning in real-world environments (glossed, e.g., as “human-comparable modeling of the features of too-complex-to-fully-simulate systems that are relevant for finding plans for changing the too-complex-to-simulate system in predictable ways”) is likely to be complicated and to require combining many different insights and techniques. (Though maybe not to the extent Robin’s thinking?) But I also think it’s likely to be a discrete research target that doesn’t look like “a par-human surgeon, combined with a par-human chemist, combined with a par-human programmer, …” You just get all the capabilities at once, and on the path to hitting that threshold you might not get many useful precursor or spin-off technologies.
Humans didn’t evolve separate specialized modules for doing theoretical physics, chemistry, computer science, etc.; indeed, we didn’t undergo selection for any of those capacities at all, they just naturally fell out of a different set of capacities we were being selected for.
Yes, a model of brain modularity in which the modules are fully independent end-to-end mechanisms for doing tasks we never faced in the evolutionary environment is pretty clearly wrong. I don’t think anyone would argue otherwise. The plausible version of the modularity model claims the modules or subsystems are specialised for performing relatively narrow subtasks, with a real-world task making use of many modules in concert—like how complex software systems today work.
As an analogy, consider a toolbox. It contains many different tools, and you could reasonably describe it as ‘modular’. But this doesn’t at all imply that it contains a separate tool for each DIY task: a wardrobe-builder, a chest-of-drawers-builder, and so on. Rather, each tool performs a certain narrow subtask; whole high-level DIY tasks are completed by applying a variety of different tools to different parts of the problem; and of course each tool can be used in solving many different high-level tasks. Generality is achieved by your toolset offering broad enough coverage to enable you to tackle most problems, not by having a single universal thing-doer.
… I also think [HLAI is] likely to be a discrete research target … You just get all the capabilities at once, and on the path to hitting that threshold you might not get many useful precursor or spin-off technologies.
What’s your basis for this view? For example, do you have some strong reason to believe the human brain similarly achieves generality via a single universal mechanism, rather than via the combination of many somewhat-specialised subsystems?
I’m updating because I think you outline a very useful concept here. Narrow algorithms can be made much more general given a good ‘algorithm switcher’. A canny switcher/coordinator program can be given a task and decide which of several narrow programs to apply to it. This is analogous to the IBM Watson system that competed in Jeopardy and to the human you describe using a PC to switch between applications. I often forget about this technique during discussions about narrow machine learning software.
I feel like this and many other arguments for AI-skepticism are implicitly assuming AGI that is amazingly dumb and then proving that there is no need to worry about this dumb superintelligence.
Remember the old “AI will never beat humans at every task because there isn’t one architecture that is optimal at every task. An AI optimised to play chess won’t be great at trading stocks (or whatever) and vice versa”? Well, I’m capable of running a different program on my computer depending on the task at hand. If your AGI can’t do the same as a random idiot with a PC, it’s not really AGI.
I am emphatically not saying that Robin Hanson has ever made this particular blunder but I think he’s making a more subtle one in the same vein.
Sure, if you think of AGI as a collection of image recognisers and go engines etc. then there is no ironclad argument for FOOM. But the moment (and probably sooner) that it becomes capable of actual general problem solving on par with it’s creators (i.e. actual AGI) and turns its powers to recursive self-improvement—how can that result in anything but FOOM? Doesn’t matter if further improvements require more complexity or less complexity or a different kind of complexity or whatever. If human researchers can do it then AGI can do it faster and better because it scales better, doesn’t sleep, doesn’t eat and doesn’t waste time arguing with people on facebook.
This must have been said a million times already. Is this not obvious? What am I missing?
The main thing that would predict slower takeoff is if early AGI systems turn out to be extremely computationally expensive. The MIRI people I’ve talked to about this are actually skeptical because they think we’re already in hardware overhang mode.
This isn’t obvious from a simple comparison of the world’s available computing hardware to the apparent requirements of human brains, though; it requires going into questions like, “Given that the human brain wasn’t built to e.g. convert compute into solving problems in biochemistry, how many of the categories of things the brain is doing do you need if you are just trying to convert compute into solving problems in biochemistry?”
If the first AGI systems are slow and expensive to run, then it could take time to use them to make major hardware or software improvements, particularly if the first AGI systems are complicated kludges in the fashion of human brains. (Though this is a really likely world-gets-destroyed scenario, because you probably can’t align something if you don’t understand its core algorithms, how it decomposes problems into subproblems, what kinds of cognitive work different parts of the system are doing and how this work contribute to the desired outcome, etc.)
Agreed. This is a really good way of stating “humans brains aren’t efficiently converting compute into research”; those little distractions, confirmation biases, losses of motivation, coffee breaks, rationalizations, etc. add up fast.
Surely that’s only under the assumption that Eliezer’s conception of AGI (simple general optimisation algorithm) is right, and Robin’s (very many separate modules comprising a big intricate system) is wrong? Is it just that you think that assumption is pretty certain to be right? Or, are you saying that even under the Hansonian model of AI, we’d still get a FOOM anyway?
I wouldn’t say that the first AGI systems are likely to be “simple.” I’d say they’re likely to be much more complex than typical narrow systems today (though shooting for relative simplicity is a good idea for safety/robustness reasons).
Humans didn’t evolve separate specialized modules for doing theoretical physics, chemistry, computer science, etc.; indeed, we didn’t undergo selection for any of those capacities at all, they just naturally fell out of a different set of capacities we were being selected for. So if the separate-modules proposal is that we’re likely to figure out how to achieve par-human chemistry without being able to achieve par-human mechanical engineering at more or less the same time, then yeah, I feel confident that’s not how things will shake out.
I think that “general” reasoning in real-world environments (glossed, e.g., as “human-comparable modeling of the features of too-complex-to-fully-simulate systems that are relevant for finding plans for changing the too-complex-to-simulate system in predictable ways”) is likely to be complicated and to require combining many different insights and techniques. (Though maybe not to the extent Robin’s thinking?) But I also think it’s likely to be a discrete research target that doesn’t look like “a par-human surgeon, combined with a par-human chemist, combined with a par-human programmer, …” You just get all the capabilities at once, and on the path to hitting that threshold you might not get many useful precursor or spin-off technologies.
Yes, a model of brain modularity in which the modules are fully independent end-to-end mechanisms for doing tasks we never faced in the evolutionary environment is pretty clearly wrong. I don’t think anyone would argue otherwise. The plausible version of the modularity model claims the modules or subsystems are specialised for performing relatively narrow subtasks, with a real-world task making use of many modules in concert—like how complex software systems today work.
As an analogy, consider a toolbox. It contains many different tools, and you could reasonably describe it as ‘modular’. But this doesn’t at all imply that it contains a separate tool for each DIY task: a wardrobe-builder, a chest-of-drawers-builder, and so on. Rather, each tool performs a certain narrow subtask; whole high-level DIY tasks are completed by applying a variety of different tools to different parts of the problem; and of course each tool can be used in solving many different high-level tasks. Generality is achieved by your toolset offering broad enough coverage to enable you to tackle most problems, not by having a single universal thing-doer.
What’s your basis for this view? For example, do you have some strong reason to believe the human brain similarly achieves generality via a single universal mechanism, rather than via the combination of many somewhat-specialised subsystems?
I’m updating because I think you outline a very useful concept here. Narrow algorithms can be made much more general given a good ‘algorithm switcher’. A canny switcher/coordinator program can be given a task and decide which of several narrow programs to apply to it. This is analogous to the IBM Watson system that competed in Jeopardy and to the human you describe using a PC to switch between applications. I often forget about this technique during discussions about narrow machine learning software.