Sure, if you think of AGI as a collection of image recognisers and go engines etc. then there is no ironclad argument for FOOM. But the moment (and probably sooner) that it becomes capable of actual general problem solving on par with it’s creators (i.e. actual AGI) and turns its powers to recursive self-improvement—how can that result in anything but FOOM?
The main thing that would predict slower takeoff is if early AGI systems turn out to be extremely computationally expensive. The MIRI people I’ve talked to about this are actually skeptical because they think we’re already in hardware overhang mode.
This isn’t obvious from a simple comparison of the world’s available computing hardware to the apparent requirements of human brains, though; it requires going into questions like, “Given that the human brain wasn’t built to e.g. convert compute into solving problems in biochemistry, how many of the categories of things the brain is doing do you need if you are just trying to convert compute into solving problems in biochemistry?”
If the first AGI systems are slow and expensive to run, then it could take time to use them to make major hardware or software improvements, particularly if the first AGI systems are complicated kludges in the fashion of human brains. (Though this is a really likely world-gets-destroyed scenario, because you probably can’t align something if you don’t understand its core algorithms, how it decomposes problems into subproblems, what kinds of cognitive work different parts of the system are doing and how this work contribute to the desired outcome, etc.)
If human researchers can do it then AGI can do it faster and better because it scales better, doesn’t sleep, doesn’t eat and doesn’t waste time arguing with people on facebook.
Agreed. This is a really good way of stating “humans brains aren’t efficiently converting compute into research”; those little distractions, confirmation biases, losses of motivation, coffee breaks, rationalizations, etc. add up fast.
The main thing that would predict slower takeoff is if early AGI systems turn out to be extremely computationally expensive.
Surely that’s only under the assumption that Eliezer’s conception of AGI (simple general optimisation algorithm) is right, and Robin’s (very many separate modules comprising a big intricate system) is wrong? Is it just that you think that assumption is pretty certain to be right? Or, are you saying that even under the Hansonian model of AI, we’d still get a FOOM anyway?
I wouldn’t say that the first AGI systems are likely to be “simple.” I’d say they’re likely to be much more complex than typical narrow systems today (though shooting for relative simplicity is a good idea for safety/robustness reasons).
Humans didn’t evolve separate specialized modules for doing theoretical physics, chemistry, computer science, etc.; indeed, we didn’t undergo selection for any of those capacities at all, they just naturally fell out of a different set of capacities we were being selected for. So if the separate-modules proposal is that we’re likely to figure out how to achieve par-human chemistry without being able to achieve par-human mechanical engineering at more or less the same time, then yeah, I feel confident that’s not how things will shake out.
I think that “general” reasoning in real-world environments (glossed, e.g., as “human-comparable modeling of the features of too-complex-to-fully-simulate systems that are relevant for finding plans for changing the too-complex-to-simulate system in predictable ways”) is likely to be complicated and to require combining many different insights and techniques. (Though maybe not to the extent Robin’s thinking?) But I also think it’s likely to be a discrete research target that doesn’t look like “a par-human surgeon, combined with a par-human chemist, combined with a par-human programmer, …” You just get all the capabilities at once, and on the path to hitting that threshold you might not get many useful precursor or spin-off technologies.
Humans didn’t evolve separate specialized modules for doing theoretical physics, chemistry, computer science, etc.; indeed, we didn’t undergo selection for any of those capacities at all, they just naturally fell out of a different set of capacities we were being selected for.
Yes, a model of brain modularity in which the modules are fully independent end-to-end mechanisms for doing tasks we never faced in the evolutionary environment is pretty clearly wrong. I don’t think anyone would argue otherwise. The plausible version of the modularity model claims the modules or subsystems are specialised for performing relatively narrow subtasks, with a real-world task making use of many modules in concert—like how complex software systems today work.
As an analogy, consider a toolbox. It contains many different tools, and you could reasonably describe it as ‘modular’. But this doesn’t at all imply that it contains a separate tool for each DIY task: a wardrobe-builder, a chest-of-drawers-builder, and so on. Rather, each tool performs a certain narrow subtask; whole high-level DIY tasks are completed by applying a variety of different tools to different parts of the problem; and of course each tool can be used in solving many different high-level tasks. Generality is achieved by your toolset offering broad enough coverage to enable you to tackle most problems, not by having a single universal thing-doer.
… I also think [HLAI is] likely to be a discrete research target … You just get all the capabilities at once, and on the path to hitting that threshold you might not get many useful precursor or spin-off technologies.
What’s your basis for this view? For example, do you have some strong reason to believe the human brain similarly achieves generality via a single universal mechanism, rather than via the combination of many somewhat-specialised subsystems?
The main thing that would predict slower takeoff is if early AGI systems turn out to be extremely computationally expensive. The MIRI people I’ve talked to about this are actually skeptical because they think we’re already in hardware overhang mode.
This isn’t obvious from a simple comparison of the world’s available computing hardware to the apparent requirements of human brains, though; it requires going into questions like, “Given that the human brain wasn’t built to e.g. convert compute into solving problems in biochemistry, how many of the categories of things the brain is doing do you need if you are just trying to convert compute into solving problems in biochemistry?”
If the first AGI systems are slow and expensive to run, then it could take time to use them to make major hardware or software improvements, particularly if the first AGI systems are complicated kludges in the fashion of human brains. (Though this is a really likely world-gets-destroyed scenario, because you probably can’t align something if you don’t understand its core algorithms, how it decomposes problems into subproblems, what kinds of cognitive work different parts of the system are doing and how this work contribute to the desired outcome, etc.)
Agreed. This is a really good way of stating “humans brains aren’t efficiently converting compute into research”; those little distractions, confirmation biases, losses of motivation, coffee breaks, rationalizations, etc. add up fast.
Surely that’s only under the assumption that Eliezer’s conception of AGI (simple general optimisation algorithm) is right, and Robin’s (very many separate modules comprising a big intricate system) is wrong? Is it just that you think that assumption is pretty certain to be right? Or, are you saying that even under the Hansonian model of AI, we’d still get a FOOM anyway?
I wouldn’t say that the first AGI systems are likely to be “simple.” I’d say they’re likely to be much more complex than typical narrow systems today (though shooting for relative simplicity is a good idea for safety/robustness reasons).
Humans didn’t evolve separate specialized modules for doing theoretical physics, chemistry, computer science, etc.; indeed, we didn’t undergo selection for any of those capacities at all, they just naturally fell out of a different set of capacities we were being selected for. So if the separate-modules proposal is that we’re likely to figure out how to achieve par-human chemistry without being able to achieve par-human mechanical engineering at more or less the same time, then yeah, I feel confident that’s not how things will shake out.
I think that “general” reasoning in real-world environments (glossed, e.g., as “human-comparable modeling of the features of too-complex-to-fully-simulate systems that are relevant for finding plans for changing the too-complex-to-simulate system in predictable ways”) is likely to be complicated and to require combining many different insights and techniques. (Though maybe not to the extent Robin’s thinking?) But I also think it’s likely to be a discrete research target that doesn’t look like “a par-human surgeon, combined with a par-human chemist, combined with a par-human programmer, …” You just get all the capabilities at once, and on the path to hitting that threshold you might not get many useful precursor or spin-off technologies.
Yes, a model of brain modularity in which the modules are fully independent end-to-end mechanisms for doing tasks we never faced in the evolutionary environment is pretty clearly wrong. I don’t think anyone would argue otherwise. The plausible version of the modularity model claims the modules or subsystems are specialised for performing relatively narrow subtasks, with a real-world task making use of many modules in concert—like how complex software systems today work.
As an analogy, consider a toolbox. It contains many different tools, and you could reasonably describe it as ‘modular’. But this doesn’t at all imply that it contains a separate tool for each DIY task: a wardrobe-builder, a chest-of-drawers-builder, and so on. Rather, each tool performs a certain narrow subtask; whole high-level DIY tasks are completed by applying a variety of different tools to different parts of the problem; and of course each tool can be used in solving many different high-level tasks. Generality is achieved by your toolset offering broad enough coverage to enable you to tackle most problems, not by having a single universal thing-doer.
What’s your basis for this view? For example, do you have some strong reason to believe the human brain similarly achieves generality via a single universal mechanism, rather than via the combination of many somewhat-specialised subsystems?