It depends on what you mean by “the world” and what you think matters in a practical sense.
I think it’s important to note that an “LLM” is a neural network that receives and emits tokens, but there is no reason it has to only be trained on languages. Images, sound, text, video, proprioception all work, and the outputs can be robotics commands like below, from https://robotics-transformer-x.github.io/ :
These days anything that isn’t SOTA doesn’t count, and this method outperforms previous methods of robotics control by 1.5x-3x.
When I see “the real world”, as an ML engineer who works on embedded systems, I think of the world as literally just the space accessible to a robot. Not “the earth” but “the objects in a robotics cell or on the table in front of it”.
Future AI systems will be able to handle more complex worlds, for example if mamba happens to work for robotics control as well, then the world complexity can be increased by a factor of 10-20 later this year or next year when this is integrated into robotics models.
Is this what you meant?
I think it’s important to remember that the “real world” in some ways is actually easier than you might think. You gave an example,
If, after seeing a million examples you become pretty good at predicting the next token in the sequence “5 + 4 =”, does this imply that you have learned something about sheep?
The world of language effectively has no rules, for instance in programming APIs a new unseen API may be missing methods on it’s objects that every similar API the model has ever seen has. So the model will attempt to call those methods, ‘hallucinating’ code that won’t run.
However the ‘real’ world has many consistent rules, such as gravity, friction, inertia, object fragility, permanence, and others. This means a robotic system can learn general strategies that will apply even to never seen objects, and a transformer neural network will naturally learn some form of compressed strategy in order to learn an effective robotics policy.
Example:
What happens is the “carrot” is blue? If it’s a real carrot? A metal object that happens to look like a carrot? Object is covered in oil?
Is it possible for the model to attempt a grab based on wrong assumptions and change it’s strategy once it learns the object is different?
Thanks for the detailed answer! I think that helped
Does the following make sense:
We use language to talk about events and objects (could be emotions, trees, etc). Since those are things that we have observed, our language will have some patterns that are related to the patterns of the world. However, the patterns in the language are not a perfect representation of the patterns in the world (we can talk about things falling away from our planet, we can talk about fire which consumes heat instead of producing it, etc). An LLM trained on text then learns the patterns of the language but not the patterns of the world. Its “world” is only language, and that’s the only thing it can learn about.
Does the above sound true? What are the problems with it?
I am ignoring your point that neural networks can be trained on a host of other things since there is little discussion around whether or not Midjourney “understands’ the images it is generating. However, the same point should apply to other modalities as well
I think what you are saying simply generalizes to:
“The model learns only what is has to learn to minimize loss in the training environment”.
Every weight is there to minimize loss. It won’t learn a deep understanding of anything when a shallow understanding works.
And this is somewhat true. It’s more complex than this and depends heavily on the specific method used—architecture, hyperparams, distillation or sparsity.
In practice, what does this mean? How do you get the model to have a deep understanding of a topic? In the real world you mostly get this by being forced to have it, aka actually doing a task in a domain.
Note that this is the second step to LLM training, rounds of RLHF that teach the model to do the task of question answering, or RL against another LLM to filter out unwanted responses.
This should also work for real world task domains.
Okay, all of that makes sense. Could this mean that the model didn’t learn anything about the real world, but it learned something about the patterns of words which give it thimbs up from the RLHFers?
This becomes a philosophical question. I think you are still kinda stuck on a somewhat obsolete view of intelligence. For decades people thought that AGI would require artificial self awareness and complex forms of metacognition.
This is not the case. We care about correct answers on tasks, where we care the most about “0 shot” performance on complex real world tasks. This is where the model gets no specific practice on the task, just a lot of practice to develop the skills needed for each step needed to complete the task, and access to reference materials and a task description.
For example if you could simulate an auto shop environment you might train the model to rebuild Chevy engines and have a 0 shot test on rebuilding Ford engines. Any model that scores well on a benchmark of Ford rebuilds has learned a general understanding about the real world.
Then imagine you don’t have just 1 benchmark, but 10,000+, over a wide range of real world 0 shot tasks. Any model that can do at human level, across as many of these tests as the average human (derived with sampling since a human taking the test suite doesn’t have time for 10,000 tests) is an AGI.
Once the model beats all living humans at all tasks in expected value on the suite the model is ASI.
It doesn’t matter how it works internally, the scores on a realistic unseen test is proof the model is AGI.
It depends on what you mean by “the world” and what you think matters in a practical sense.
I think it’s important to note that an “LLM” is a neural network that receives and emits tokens, but there is no reason it has to only be trained on languages. Images, sound, text, video, proprioception all work, and the outputs can be robotics commands like below, from https://robotics-transformer-x.github.io/ :
These days anything that isn’t SOTA doesn’t count, and this method outperforms previous methods of robotics control by 1.5x-3x.
When I see “the real world”, as an ML engineer who works on embedded systems, I think of the world as literally just the space accessible to a robot. Not “the earth” but “the objects in a robotics cell or on the table in front of it”.
Future AI systems will be able to handle more complex worlds, for example if mamba happens to work for robotics control as well, then the world complexity can be increased by a factor of 10-20 later this year or next year when this is integrated into robotics models.
Is this what you meant?
I think it’s important to remember that the “real world” in some ways is actually easier than you might think. You gave an example,
The world of language effectively has no rules, for instance in programming APIs a new unseen API may be missing methods on it’s objects that every similar API the model has ever seen has. So the model will attempt to call those methods, ‘hallucinating’ code that won’t run.
However the ‘real’ world has many consistent rules, such as gravity, friction, inertia, object fragility, permanence, and others. This means a robotic system can learn general strategies that will apply even to never seen objects, and a transformer neural network will naturally learn some form of compressed strategy in order to learn an effective robotics policy.
Example:
What happens is the “carrot” is blue? If it’s a real carrot? A metal object that happens to look like a carrot? Object is covered in oil?
Is it possible for the model to attempt a grab based on wrong assumptions and change it’s strategy once it learns the object is different?
Thanks for the detailed answer! I think that helped
Does the following make sense:
We use language to talk about events and objects (could be emotions, trees, etc). Since those are things that we have observed, our language will have some patterns that are related to the patterns of the world. However, the patterns in the language are not a perfect representation of the patterns in the world (we can talk about things falling away from our planet, we can talk about fire which consumes heat instead of producing it, etc). An LLM trained on text then learns the patterns of the language but not the patterns of the world. Its “world” is only language, and that’s the only thing it can learn about.
Does the above sound true? What are the problems with it?
I am ignoring your point that neural networks can be trained on a host of other things since there is little discussion around whether or not Midjourney “understands’ the images it is generating. However, the same point should apply to other modalities as well
I think what you are saying simply generalizes to:
“The model learns only what is has to learn to minimize loss in the training environment”.
Every weight is there to minimize loss. It won’t learn a deep understanding of anything when a shallow understanding works.
And this is somewhat true. It’s more complex than this and depends heavily on the specific method used—architecture, hyperparams, distillation or sparsity.
In practice, what does this mean? How do you get the model to have a deep understanding of a topic? In the real world you mostly get this by being forced to have it, aka actually doing a task in a domain.
Note that this is the second step to LLM training, rounds of RLHF that teach the model to do the task of question answering, or RL against another LLM to filter out unwanted responses.
This should also work for real world task domains.
Okay, all of that makes sense. Could this mean that the model didn’t learn anything about the real world, but it learned something about the patterns of words which give it thimbs up from the RLHFers?
This becomes a philosophical question. I think you are still kinda stuck on a somewhat obsolete view of intelligence. For decades people thought that AGI would require artificial self awareness and complex forms of metacognition.
This is not the case. We care about correct answers on tasks, where we care the most about “0 shot” performance on complex real world tasks. This is where the model gets no specific practice on the task, just a lot of practice to develop the skills needed for each step needed to complete the task, and access to reference materials and a task description.
For example if you could simulate an auto shop environment you might train the model to rebuild Chevy engines and have a 0 shot test on rebuilding Ford engines. Any model that scores well on a benchmark of Ford rebuilds has learned a general understanding about the real world.
Then imagine you don’t have just 1 benchmark, but 10,000+, over a wide range of real world 0 shot tasks. Any model that can do at human level, across as many of these tests as the average human (derived with sampling since a human taking the test suite doesn’t have time for 10,000 tests) is an AGI.
Once the model beats all living humans at all tasks in expected value on the suite the model is ASI.
It doesn’t matter how it works internally, the scores on a realistic unseen test is proof the model is AGI.
I think I understand, thank you. For reference, this is the tweet which sparked the question: https://twitter.com/RichardMCNgo/status/1735571667420598737
I was confused as to why you would necessarily need “understanding” and not just simple next token prediction to do what ChatGPT does