I see “AGI” used for everything from existing LLMs to superintelligence, and massive resulting confusion and illusory disagreements. I finally thought of a term I like for what I mean by AGI. It’s an acronym that’s also somewhat intuitive without reading the definition:
Reasoning, Reflective Entities with Autonomy and Learning
might be called “(R)REAL AGI” or “real AGI”. See below for further definitions.
Hoping for AI to remain hobbled by being cognitively incomplete looks like wishful thinking to me. Nor can we be sure that these improvements and the resulting dangers won’t happen soon.
I think there are good reasons to expect that we get such “real AGI” very soon after we have useful AI. After 2+ decades of studying how the brain performs complex cognition, I’m pretty sure that our cognitive abilities are the result of multiple brain subsystems and cognitive capacities working synergistically. A similar approach is likely to advance AI.
Adding these other cognitive capacities creates language model agents/cognitive architectures (LMCAs). Adding each of these seems relatively easy (compared to developing language models) and almost guaranteed to add useful (but dangerous) capabilities.
More on this and expanded arguments and definitions in an upcoming post; this is primarily a reference for this definition of AGI.
Reasoning
Deliberative “System 2” reasoning allows humans to trade off “thinking time” for accuracy.
Aggregating cognition for better results can be added to nearly any system in fairly straightforward ways.
Reflective
Can think about their own cognition.
Useful for organizing and improving cognition.
Reflective stability of values/goals has important upsides and downsides for alignment
Entities
Evokes the intuition of a whole mind, rather than piece of a mind or a cognitive tool
with Autonomy
Acts independently without close direction.
Very useful for getting things done efficiently
Has agency in that they take actions, and have explicit goals they pursue with flexible strategies
Including deriving and pursuing explicit, novel subgoals
This is highly useful for factoring novel complex tasks
One useful subgoal is “make sure nobody prevents me from working on my main goal.” ;)
and Learning
Continuous learning from ongoing experience
Humans have at least four types; LMCAs currently have one and a fraction.[1]
These are all straightforwardly implementable for LMCA agents and probably for other potential network AGI designs.
AGI- Artificial (FULLY) General Intelligence
All of the above are (arguably) implied by fully general intelligence:
Humans can think about anything with some success.
That includes thinking about their own cognition (Reflection) which enables Reasoning by allowing strategic aggregation of cognitive steps.
It requires online Learning to think about topics not in the previous training set.
It almost requires goal-directed Autonomy to gather useful new information and arguably, to take “mental actions” that travel conceptual space strategically.
Those together imply an Entity that is functionally coherent and goal-directed.
You could drop one of the Rs or aggregate them if you wanted a nicer acronym.
The above capacities are often synergistic, in that having each makes others work better. For instance, a “real AGI” can Learn important results of its time-consuming Reasoning, and can Reason more efficiently using Learned strategies. The different types of Learning are synergistic with each other, etc. More on some potential synergies in Capabilities and alignment of LLM cognitive architectures; the logic applies to other multi-capacity AI systems as well.
I like two other possible terms for the same definition of AGI: “full AGI” for artificial fully general intelligence; or “parahuman AGI” to imply having all the same cognitive capacities as humans, and working with humans.
This definition is highly similar to Steve Byrnes’ in “Artificial General Intelligence”: an extremely brief FAQ, although his explanation is different enough to be complementary. It does not specify all of the same cognitive abilities, and provides different intuition pumps. Something like this conception of advanced AI appears to be common in most treatments of aligning superintelligence, but not in prosaic alignment work.
More on the definition and arguments for the inclusion of each of those cognitive capacities will be included in a future post, linked here when it’s done. I wanted to get this out and have a succinct definition. Questions and critiques of the definitions and claims here will make that a better post.
All feedback is welcome. If anyone’s got better terms, I’d love to adopt them.
Edit: title changed in a fit of indecision. Quotation marks used to emphasize that it’s a definition, not a claim about what AGI “really” is.
“Real AGI”
I see “AGI” used for everything from existing LLMs to superintelligence, and massive resulting confusion and illusory disagreements. I finally thought of a term I like for what I mean by AGI. It’s an acronym that’s also somewhat intuitive without reading the definition:
Reasoning, Reflective Entities with Autonomy and Learning
might be called “(R)REAL AGI” or “real AGI”. See below for further definitions.
Hoping for AI to remain hobbled by being cognitively incomplete looks like wishful thinking to me. Nor can we be sure that these improvements and the resulting dangers won’t happen soon.
I think there are good reasons to expect that we get such “real AGI” very soon after we have useful AI. After 2+ decades of studying how the brain performs complex cognition, I’m pretty sure that our cognitive abilities are the result of multiple brain subsystems and cognitive capacities working synergistically. A similar approach is likely to advance AI.
Adding these other cognitive capacities creates language model agents/cognitive architectures (LMCAs). Adding each of these seems relatively easy (compared to developing language models) and almost guaranteed to add useful (but dangerous) capabilities.
More on this and expanded arguments and definitions in an upcoming post; this is primarily a reference for this definition of AGI.
Reasoning
Deliberative “System 2” reasoning allows humans to trade off “thinking time” for accuracy.
Aggregating cognition for better results can be added to nearly any system in fairly straightforward ways.
Reflective
Can think about their own cognition.
Useful for organizing and improving cognition.
Reflective stability of values/goals has important upsides and downsides for alignment
Entities
Evokes the intuition of a whole mind, rather than piece of a mind or a cognitive tool
with Autonomy
Acts independently without close direction.
Very useful for getting things done efficiently
Has agency in that they take actions, and have explicit goals they pursue with flexible strategies
Including deriving and pursuing explicit, novel subgoals
This is highly useful for factoring novel complex tasks
One useful subgoal is “make sure nobody prevents me from working on my main goal.” ;)
and Learning
Continuous learning from ongoing experience
Humans have at least four types; LMCAs currently have one and a fraction.[1]
These are all straightforwardly implementable for LMCA agents and probably for other potential network AGI designs.
AGI- Artificial (FULLY) General Intelligence
All of the above are (arguably) implied by fully general intelligence:
Humans can think about anything with some success.
That includes thinking about their own cognition (Reflection) which enables Reasoning by allowing strategic aggregation of cognitive steps.
It requires online Learning to think about topics not in the previous training set.
It almost requires goal-directed Autonomy to gather useful new information and arguably, to take “mental actions” that travel conceptual space strategically.
Those together imply an Entity that is functionally coherent and goal-directed.
You could drop one of the Rs or aggregate them if you wanted a nicer acronym.
The above capacities are often synergistic, in that having each makes others work better. For instance, a “real AGI” can Learn important results of its time-consuming Reasoning, and can Reason more efficiently using Learned strategies. The different types of Learning are synergistic with each other, etc. More on some potential synergies in Capabilities and alignment of LLM cognitive architectures; the logic applies to other multi-capacity AI systems as well.
I like two other possible terms for the same definition of AGI: “full AGI” for artificial fully general intelligence; or “parahuman AGI” to imply having all the same cognitive capacities as humans, and working with humans.
This definition is highly similar to Steve Byrnes’ in “Artificial General Intelligence”: an extremely brief FAQ, although his explanation is different enough to be complementary. It does not specify all of the same cognitive abilities, and provides different intuition pumps. Something like this conception of advanced AI appears to be common in most treatments of aligning superintelligence, but not in prosaic alignment work.
More on the definition and arguments for the inclusion of each of those cognitive capacities will be included in a future post, linked here when it’s done. I wanted to get this out and have a succinct definition. Questions and critiques of the definitions and claims here will make that a better post.
All feedback is welcome. If anyone’s got better terms, I’d love to adopt them.
Edit: title changed in a fit of indecision. Quotation marks used to emphasize that it’s a definition, not a claim about what AGI “really” is.
Types of continuous learning in humans (and language model cognitive architecture (LMCA) equivalents):
Working memory
(LMCAs have the context window)
Semantic memory/habit learning
(Model weight updates from experience)
Episodic memory for important snapshots of cognition
(Vector-based text memory is this but poorly implemented)
Dopamine-based RL using a powerful critic
(self-supervised RLAIF and/or RLHF during deployment)