A dictionary defines all words circularly, but of course nobody learns all words from a dictionary—the assumption is you’re looking up a small number of words you don’t know.
Humans learn their first few words by seeing how they’re used in relation to objects, and the rest can be derived from there without needing circularity.
However the dictionary provides very tight constraints on what words can mean. Whatever the words “wood”, “is”, “made”, “from”, and “trees” mean, the sentence “wood is made from trees” must be true. The vast majority of all possible meanings fail this. Using only circular definitions, is it possible to constraint words meanings so tightly that there’s only one possible model which fits those constraints?
LLMs seem to provide a resounding yes to that question. Whilst 1st generation LLMs only ever saw text and had no hard coded knowledge, so could only possibly figure out what words meant based on how they’re used in relation to other words, they understood the meaning of words sufficiently well to reason about the physical properties of the objects they represented.
Using only circular definitions, is it possible to constraint words meanings so tightly that there’s only one possible model which fits those constraints?
Isn’t this sort-of what all formal mathematical systems do? You start with some axioms that define how your atoms must relate to each other, and (in a good system) those axioms pin the concepts down well enough that you can start proving a bunch of theorems about them.
The intended question, I think, is if you were to find a dictionary for some alien language (not a translators dictionary, but a dictionary for people who speak that language to look up definitions of words), can you translate most of the dictionary to English? What if you additionally had access to large amounts of conversations in that language, without any indication of what the aliens were looking at/doing at the time of the conversation?
A dictionary defines all words circularly, but of course nobody learns all words from a dictionary—the assumption is you’re looking up a small number of words you don’t know.
Humans learn their first few words by seeing how they’re used in relation to objects, and the rest can be derived from there without needing circularity.
However the dictionary provides very tight constraints on what words can mean. Whatever the words “wood”, “is”, “made”, “from”, and “trees” mean, the sentence “wood is made from trees” must be true. The vast majority of all possible meanings fail this. Using only circular definitions, is it possible to constraint words meanings so tightly that there’s only one possible model which fits those constraints?
LLMs seem to provide a resounding yes to that question. Whilst 1st generation LLMs only ever saw text and had no hard coded knowledge, so could only possibly figure out what words meant based on how they’re used in relation to other words, they understood the meaning of words sufficiently well to reason about the physical properties of the objects they represented.
Isn’t this sort-of what all formal mathematical systems do? You start with some axioms that define how your atoms must relate to each other, and (in a good system) those axioms pin the concepts down well enough that you can start proving a bunch of theorems about them.
The intended question, I think, is if you were to find a dictionary for some alien language (not a translators dictionary, but a dictionary for people who speak that language to look up definitions of words), can you translate most of the dictionary to English? What if you additionally had access to large amounts of conversations in that language, without any indication of what the aliens were looking at/doing at the time of the conversation?
“wood is made from trees”
“trees are made of wood”
A new circularity!