I’m glad for this, LessWrong can always use more engaging critiques of substance. I partially agree with Holden’s conclusions, although I reach them from a substantially different route. I’m a little surprised then that few of the replies have directly engaged what I find to be the more obvious flaws in Holden’s argument: namely objection 2 and the inherent contradictions with it and objection 1.
Holden posits that many (most?) well-known current AI applications more or less operate as sophisticated knowledge
bases. His tool/agent distinction draws a boundary around AI tools: systems whose only external actions consist of communicating results to humans, and the rest being agents which actually plan and execute actions with external side
effects. Holden distinguishes ‘tool’ AI from Oracle AI, the latter really being agent AI (designed for autonomy) which is trapped in some sort of box. Accepting Holden’s terminology and tool/agent distinction, he then asserts:
That ‘tool’ AGI already is and will continue to be the dominant type of AI system.
That AGI running in tool mode will: ” be extraordinarily useful but far more safe than an AGI running in agent mode,”
I can accept that any AGI running in ‘tool’ mode will be far safer than an AGI running in agent mode (although perhaps still not completely safe), but I believe Holden critically overestimates the domain and potential of ‘tool’ AGI, given his distinction.
It is true that many well known current AI systems operate as sophisticated knowledge tools, rather than agents. Search engines such as google are the first example Holden lists, but I haven’t heard many people refer to search engines as AGIs.
In fact, having the capability to act in the world and learn from the history of such interactions is a crucial component of many AGI architectures, and perhaps all with the potential for human-level general intelligence. One could certainly remove the AGI’s capacity for action at a later date: in Holden’s terminology this would be switching the AGI from tool mode to agent mode. If we were using more everyday terminology we might as well call this paralyzing the AGI.
Yes switching an existing agent AGI into ‘tool’ mode (paralyzing it) certainly solves most safety issues regarding that particular agent, but this is far from a global panacea. Being less charitable, I would say it adds little of substance to the discussions of AI existential risk. It’s much like one saying “but we can simply disable the nukes!”. (and it’s even potentially less effective than the analogy implies, because superpowerful unsafe agent AIs may not be so easy to ‘switch’ into ‘tool’ mode, to put it mildly).
After Google, Holden’s next examples of primarily ‘tool’ mode AI are Siri and Watson. Siri is actually an agent in Holden’s terminology, it can execute some web tasks in its limited set of domains. This may be a small percent of its current usage, but I don’t expect that to hold true for its future descendants.
What Holden fails to mention are any relevant examples of all the current agent AI systems we already have today, and what tomorrow may bring.
The world of financial trading is already dominated by AI agents, and this particular history is most telling. Decades ago, when computer were very weak, they were used as simple tools to evaluate financial models which in turn were just a component of a human agent’s overall trading strategy. As computers grew in power and became integrally connected in financial networks, computers began to take on more and more book-keeping actions, and eventually people started using computers to execute entire simple trading strategies on their own (albeit with much hands on supervision). Fast forward to 2012 and we now have the majority of trades executed by automated and increasingly autonomous trading systems. They are still under the supervision of human caretakers, but as they grow in complexity this increasingly becomes a nominal role.
There is a vast profitable micro-realm where these agents trade on lightning fast millisecond timescales; an economic niche that humans literally can not enter: it’s an alien environment, and we have been engineering and evolving alien agents to inhabit and exploit it for us.
To one who has only basic familiarity with software development, one may imagine that software is something that humans design, write and build according to their specifications. That is only partially true, moreso for smaller projects.
The other truth, perhaps a deeper wisdom, is that large software systems evolve. Huge software systems are too massive to be designed or even fully understood by individual human minds, so their development follows a trajectory that is perhaps better understood in evolutionary terms. This is already the cause of much concern in the context of operating systems and security, which itself is only a small taste of the safety issues in a future world dominated by large, complex evolved agent systems.
It is true that there are many application domains where agents have had little impact as of yet, but this just shows us the niches that agents will eventually occupy.
In the end of the day, we need just compare the ultimate economic value of a tool vs an agent. What fraction of current human occupations can be considered ‘tool’ jobs vs ‘agent’ jobs? Agents which drive cars or execute finacial trades are just the beginning, the big opportunites are in agents which autonomously build software systems, design aircraft, perform biology research, and so on. Systems such as Siri and Watson today mainly function as knowledge tools, but we can expect that their descendants will eventually occupy a broad full range of human jobs, most of which involve varying degrees of autonomous agent behavior.
Consider the current domain of software development. What does a tool-mode AGI really add here in a world that already has google and the web? A tool-mode AGI could take a vague design and set of business constraints and output a detailed design or perhaps even an entire program, but that program would still need to be tested, and you might as well automate that. And most large software systems consist of ecologies of interacting programs: web crawlers, databases, planning/logistic systems, and so on where most of the ‘actions’ are already automated rather than assigned to humans.
As another example consider the ‘lights-out’ automated factory. The foundries that produce microchips are becoming increasingly autonomous systems, as is the front side design industry. If we extrapolate that to the future …
The IBM of tommorrow may well consist of a small lucky pool of human stockowners reaping the benefits of a vast army of watson’s future descendants who have gone on to replace all the expensive underperforming human employees of our time. International Business Machines, indeed: a world where everything from the low level hardware and foundry machinery up to the software and even business strategy is designed and built by complex ecologies of autonomous software agents. That seems to be not only where we are heading, but to some limited degree, where we already are.
Thus I find it highly unlikely that tool mode AI is and will be the dominate paradigm, as Holden asserts. Moreover, his argument really depends on tool mode being dominate by a significant fraction. If agent AI consists of even only 5% of the market at some future date, it still could contribute an unacceptable majority of risk.
I’m glad for this, LessWrong can always use more engaging critiques of substance. I partially agree with Holden’s conclusions, although I reach them from a substantially different route. I’m a little surprised then that few of the replies have directly engaged what I find to be the more obvious flaws in Holden’s argument: namely objection 2 and the inherent contradictions with it and objection 1.
Holden posits that many (most?) well-known current AI applications more or less operate as sophisticated knowledge bases. His tool/agent distinction draws a boundary around AI tools: systems whose only external actions consist of communicating results to humans, and the rest being agents which actually plan and execute actions with external side effects. Holden distinguishes ‘tool’ AI from Oracle AI, the latter really being agent AI (designed for autonomy) which is trapped in some sort of box. Accepting Holden’s terminology and tool/agent distinction, he then asserts:
That ‘tool’ AGI already is and will continue to be the dominant type of AI system.
That AGI running in tool mode will: ” be extraordinarily useful but far more safe than an AGI running in agent mode,”
I can accept that any AGI running in ‘tool’ mode will be far safer than an AGI running in agent mode (although perhaps still not completely safe), but I believe Holden critically overestimates the domain and potential of ‘tool’ AGI, given his distinction.
It is true that many well known current AI systems operate as sophisticated knowledge tools, rather than agents. Search engines such as google are the first example Holden lists, but I haven’t heard many people refer to search engines as AGIs.
In fact, having the capability to act in the world and learn from the history of such interactions is a crucial component of many AGI architectures, and perhaps all with the potential for human-level general intelligence. One could certainly remove the AGI’s capacity for action at a later date: in Holden’s terminology this would be switching the AGI from tool mode to agent mode. If we were using more everyday terminology we might as well call this paralyzing the AGI.
Yes switching an existing agent AGI into ‘tool’ mode (paralyzing it) certainly solves most safety issues regarding that particular agent, but this is far from a global panacea. Being less charitable, I would say it adds little of substance to the discussions of AI existential risk. It’s much like one saying “but we can simply disable the nukes!”. (and it’s even potentially less effective than the analogy implies, because superpowerful unsafe agent AIs may not be so easy to ‘switch’ into ‘tool’ mode, to put it mildly).
After Google, Holden’s next examples of primarily ‘tool’ mode AI are Siri and Watson. Siri is actually an agent in Holden’s terminology, it can execute some web tasks in its limited set of domains. This may be a small percent of its current usage, but I don’t expect that to hold true for its future descendants.
What Holden fails to mention are any relevant examples of all the current agent AI systems we already have today, and what tomorrow may bring.
The world of financial trading is already dominated by AI agents, and this particular history is most telling. Decades ago, when computer were very weak, they were used as simple tools to evaluate financial models which in turn were just a component of a human agent’s overall trading strategy. As computers grew in power and became integrally connected in financial networks, computers began to take on more and more book-keeping actions, and eventually people started using computers to execute entire simple trading strategies on their own (albeit with much hands on supervision). Fast forward to 2012 and we now have the majority of trades executed by automated and increasingly autonomous trading systems. They are still under the supervision of human caretakers, but as they grow in complexity this increasingly becomes a nominal role.
There is a vast profitable micro-realm where these agents trade on lightning fast millisecond timescales; an economic niche that humans literally can not enter: it’s an alien environment, and we have been engineering and evolving alien agents to inhabit and exploit it for us.
To one who has only basic familiarity with software development, one may imagine that software is something that humans design, write and build according to their specifications. That is only partially true, moreso for smaller projects.
The other truth, perhaps a deeper wisdom, is that large software systems evolve. Huge software systems are too massive to be designed or even fully understood by individual human minds, so their development follows a trajectory that is perhaps better understood in evolutionary terms. This is already the cause of much concern in the context of operating systems and security, which itself is only a small taste of the safety issues in a future world dominated by large, complex evolved agent systems.
It is true that there are many application domains where agents have had little impact as of yet, but this just shows us the niches that agents will eventually occupy.
In the end of the day, we need just compare the ultimate economic value of a tool vs an agent. What fraction of current human occupations can be considered ‘tool’ jobs vs ‘agent’ jobs? Agents which drive cars or execute finacial trades are just the beginning, the big opportunites are in agents which autonomously build software systems, design aircraft, perform biology research, and so on. Systems such as Siri and Watson today mainly function as knowledge tools, but we can expect that their descendants will eventually occupy a broad full range of human jobs, most of which involve varying degrees of autonomous agent behavior.
Consider the current domain of software development. What does a tool-mode AGI really add here in a world that already has google and the web? A tool-mode AGI could take a vague design and set of business constraints and output a detailed design or perhaps even an entire program, but that program would still need to be tested, and you might as well automate that. And most large software systems consist of ecologies of interacting programs: web crawlers, databases, planning/logistic systems, and so on where most of the ‘actions’ are already automated rather than assigned to humans.
As another example consider the ‘lights-out’ automated factory. The foundries that produce microchips are becoming increasingly autonomous systems, as is the front side design industry. If we extrapolate that to the future …
The IBM of tommorrow may well consist of a small lucky pool of human stockowners reaping the benefits of a vast army of watson’s future descendants who have gone on to replace all the expensive underperforming human employees of our time. International Business Machines, indeed: a world where everything from the low level hardware and foundry machinery up to the software and even business strategy is designed and built by complex ecologies of autonomous software agents. That seems to be not only where we are heading, but to some limited degree, where we already are.
Thus I find it highly unlikely that tool mode AI is and will be the dominate paradigm, as Holden asserts. Moreover, his argument really depends on tool mode being dominate by a significant fraction. If agent AI consists of even only 5% of the market at some future date, it still could contribute an unacceptable majority of risk.