As a father of two very young daughters (2 years old and 2 months old), I can really appreciate this. As someone with a background in computational neuroscience and some linguistics/NLP/ML/AI, I’ve loved watching them grow and making educated guesses about what sorts of computations could be going on inside their little brains at each developmental stage.
From the earliest days, when they can’t even focus on our faces, hard as they try (I can tell what you’re trying to do, superior coliculus and fusiform face area; you can do it!), to later on when they’re walking and talking (still working on that theory of mind, though).
Language development has been especially fun to watch. Early on, they love just staring at your mouth as you enunciate the various phonemic sequences of what will become their native language. As they become more aware, you can see them start to comprehend when you use simple sentences to narrate things within their field of attention. And they definitely learn to understand more complex language long before they can talk. Patterns built upon patterns, just like deep transformer models, yet still quite different.
When my daughter began to pronounce words, we started pausing intermittently while reading her favorite books or singing familiar songs, and we would have her complete the last word of each line. I couldn’t help but think of how large language models are often trained to perform next-token prediction in a similar way. Although, it’s clear that the human brain has some sort of extra bias that makes it easier to memorize songs and poetry than prose.
And it’s funny how trying to talk to babies reveals just how much of our adult-level world model we assume when we communicate. Once, when our oldest was trying to use a sippy cup on her own for the first time, we saw her putting it to her mouth like we did but failing to get any water. To help her out, I told her to lift the bottom of her cup to get at the water. She then proceeded to lift the entire cup above her head, which surprisingly did not help her. (Eventually she got it.)
For all their temporary limitations, it’s clear that there is a lot going on inside babies’ heads. You can learn a lot about the human brain and cognitive algorithms and biases by studying them carefully. It’s certainly the cutest way to do so.
One thing I forgot to mention that I meant to say: My oldest can now speak English pretty fluently, using articles and adjectives, even some compound sentences; she just has first and second person pronouns mixed up.
When she says, “Go in your bed,” she’s referring to her own bed. Or when she says, “Give me the ball,” she means that she is giving it to us.
This sort of mistake makes sense when you consider that she learned English by listening to her parents narrate things from our own perspective. Every time we said, “Thank you,” she was giving us something, so now that’s what she says when she gives us something. Every time we used first person pronouns, she correctly inferred that we were referring to ourselves (i.e., not her), and every time we used second person pronouns, we were referring to her.
When she speaks, it sounds like she’s telling us what she predicts we would say, rather than what we would expect her to say as a proper conversational partner.
I’ve heard that babies start out life not just egocentric, but actually unable to distinguish the rest of the universe from themselves. Our youngest is still at that stage. When she is sad, it’s because the universe is sad. Mommy and Daddy are not individuals out there operating independently in an external world; they’re just phenomena that the universe generates to make everything better.
For our oldest, she knows that we are different people, but she seems to see language as a process of narrating everything that happens. Sentences are things we build together, rather than a way for different people to share their own perspectives with each other. “I/me/my” is always said about the other person, and “you/your” is always said about her.
For GPT-3, I feel it’s the same way. Language is something it generates and predicts, not a conversation it participates in from its own perspective.
To correct this In our daughter, we say something like, “No, say, ‘I pooped in my potty,’” if she says, “You pooped in your potty.”
For a large language model, how would we get it to understand that it is separate from us, with its own limitations in understanding, which are different from our own limitations in understanding? Is that something we even want?
Mine (3.5 yrs) says “show me” when she wants to show something to her mom or me. Usually with an exasperated sigh between trying to describe the thing and just deciding we need to come see it.
[as our oldest sees it] Sentences are things we build together, rather than a way for different people to share their own perspectives with each other.
I’ve gone through a huge growth arc as an adult in recognizing the extent to which (especially in really good conversations) sentences are things we build together. Not that we don’t have different perspectives, but when conversation is really flowing, it makes way more sense to view it as “our collective mind is thinking” and not “I am transmitting information to you, then getting information back” etc.
(When we’re more at odds with someone, whether adversarial or just conflict with a loved one, it can be more like the transmit mode, and sometimes (tho not always) it seems to work best if we can get into the co-thinking mode again. Though there’s not a hack for that—it’s a deep trust-dancing puzzle!)
As a father of two very young daughters (2 years old and 2 months old), I can really appreciate this. As someone with a background in computational neuroscience and some linguistics/NLP/ML/AI, I’ve loved watching them grow and making educated guesses about what sorts of computations could be going on inside their little brains at each developmental stage.
From the earliest days, when they can’t even focus on our faces, hard as they try (I can tell what you’re trying to do, superior coliculus and fusiform face area; you can do it!), to later on when they’re walking and talking (still working on that theory of mind, though).
Language development has been especially fun to watch. Early on, they love just staring at your mouth as you enunciate the various phonemic sequences of what will become their native language. As they become more aware, you can see them start to comprehend when you use simple sentences to narrate things within their field of attention. And they definitely learn to understand more complex language long before they can talk. Patterns built upon patterns, just like deep transformer models, yet still quite different.
When my daughter began to pronounce words, we started pausing intermittently while reading her favorite books or singing familiar songs, and we would have her complete the last word of each line. I couldn’t help but think of how large language models are often trained to perform next-token prediction in a similar way. Although, it’s clear that the human brain has some sort of extra bias that makes it easier to memorize songs and poetry than prose.
And it’s funny how trying to talk to babies reveals just how much of our adult-level world model we assume when we communicate. Once, when our oldest was trying to use a sippy cup on her own for the first time, we saw her putting it to her mouth like we did but failing to get any water. To help her out, I told her to lift the bottom of her cup to get at the water. She then proceeded to lift the entire cup above her head, which surprisingly did not help her. (Eventually she got it.)
For all their temporary limitations, it’s clear that there is a lot going on inside babies’ heads. You can learn a lot about the human brain and cognitive algorithms and biases by studying them carefully. It’s certainly the cutest way to do so.
One thing I forgot to mention that I meant to say: My oldest can now speak English pretty fluently, using articles and adjectives, even some compound sentences; she just has first and second person pronouns mixed up.
When she says, “Go in your bed,” she’s referring to her own bed. Or when she says, “Give me the ball,” she means that she is giving it to us.
This sort of mistake makes sense when you consider that she learned English by listening to her parents narrate things from our own perspective. Every time we said, “Thank you,” she was giving us something, so now that’s what she says when she gives us something. Every time we used first person pronouns, she correctly inferred that we were referring to ourselves (i.e., not her), and every time we used second person pronouns, we were referring to her.
When she speaks, it sounds like she’s telling us what she predicts we would say, rather than what we would expect her to say as a proper conversational partner.
I’ve heard that babies start out life not just egocentric, but actually unable to distinguish the rest of the universe from themselves. Our youngest is still at that stage. When she is sad, it’s because the universe is sad. Mommy and Daddy are not individuals out there operating independently in an external world; they’re just phenomena that the universe generates to make everything better.
For our oldest, she knows that we are different people, but she seems to see language as a process of narrating everything that happens. Sentences are things we build together, rather than a way for different people to share their own perspectives with each other. “I/me/my” is always said about the other person, and “you/your” is always said about her.
For GPT-3, I feel it’s the same way. Language is something it generates and predicts, not a conversation it participates in from its own perspective.
To correct this In our daughter, we say something like, “No, say, ‘I pooped in my potty,’” if she says, “You pooped in your potty.”
For a large language model, how would we get it to understand that it is separate from us, with its own limitations in understanding, which are different from our own limitations in understanding? Is that something we even want?
I’ve heard the pronoun-reversal mistake is common.
My child does the same thing. He says, “Pick you up?” meaning “Pick me up?”
Mine (3.5 yrs) says “show me” when she wants to show something to her mom or me. Usually with an exasperated sigh between trying to describe the thing and just deciding we need to come see it.
I’ve gone through a huge growth arc as an adult in recognizing the extent to which (especially in really good conversations) sentences are things we build together. Not that we don’t have different perspectives, but when conversation is really flowing, it makes way more sense to view it as “our collective mind is thinking” and not “I am transmitting information to you, then getting information back” etc.
(When we’re more at odds with someone, whether adversarial or just conflict with a loved one, it can be more like the transmit mode, and sometimes (tho not always) it seems to work best if we can get into the co-thinking mode again. Though there’s not a hack for that—it’s a deep trust-dancing puzzle!)
The 2nd half of this video is about this collective mind thing: https://youtu.be/G3vcXZPlsDc