Caveat: I know little to nothing about the architecture of such things, please take this as naive user feedback if you wish, or you could ignore it.
Just now I was asking the Meta AI chatbot how to do an ‘L-Cut’ using the Android Instagram app. It hallucinated for quite a few messages instructions how to ‘overlap’ two video tracks when editing a reel before it finally admitted that no such ability in fact exists in the Instagram App.
My grossly negligent mistake was assuming that a AI LLM with Meta Branding would have current or accurate knowledge of Meta properties and products.
However, imagine that there are two versions of the Instagram App, one that has this feature and one that doesn’t—why didn’t it ask “Just to check, what kind of phone are you using?” (which, also, would allay fears of ‘spying’ and invasion of privacy) and then, and only then give me advice or apologetically inform me that such a feature is not available. (In other branded/commercial chatbots, wouldn’t this be an opportunity for an up-sell?[1])
Maybe I’m not prompting LLMs in the right way, but they almost never ask me questions. And certainly never clarification questions. They make an assumption, and produce a slab of text with a boilerplate “let me know if this helps?” tacked on the end. I assume this is a conscious choice and not a limitation of the technology. To test my observation I entered the same prompt into three different LLMs:
I just asked Claude Haiku 3 “When was Rosberg’s first Grand Prix?” to see if it assumes I mean Keke Rosberg, or his son, Nico Rosberg.Claude apologies for not having information on Nico Rosberg’s first Grand Prix… and then goes on to state that “Formula One has had multiple drivers named Rosberg compete over the years. More details would be needed to determine the exact driver and their debut race.”
What I find interesting, not one sentence in the answer takes the form of an interrogative!
Next, I asked GPT-4o mini the exact same question, to which it confidently assumes which Rosberg I mean: “Nico Rosberg’s first Formula One Grand Prix was at the 2006 Bahrain Grand Prix, which took place on March 12, 2006. He raced for the Williams team.”
Microsoft CoPilot makes the same assumption, with the traditional tacked on boilerplate question: ”Nico Rosberg made his Formula One debut at the 2006 Bahrain Grand Prix. He went on to have a successful career, culminating in winning the World Drivers’ Championship in 2016. Is there anything else you’d like to know about his career or Formula One in general? [FI car Emoji, Smoke Emoji]”
Meta AI makes the same assumption, and gives a lot more unsolicited information, but it does mention his father: ”Nico Rosberg’s first Grand Prix was the 2006 Bahrain Grand Prix. This marked the beginning of his Formula One career, which would eventually see him drive for Williams from 2006 to 2009 and then Mercedes from 2010 to 2016. During his time with Mercedes, Rosberg won the Formula One Driver’s championship in 2016, a feat that made him and his father, Keke Rosberg, the second-father pairing to achieve this impressive milestone” Not that it is relevant, but Graham and Damon Hill were the first.
This is not an exhaustive sample by any means, but does confirm my naive observation that LLMs do not ask clarifying questions or use interrogatives in order to better shape their responses to my needs.
I imagine such a commercial hellscape would look a little like this:
”I was just wondering why I wasn’t earning any Ultrabonus points with my purchases” “Before we continue, could you tell me, do you have a Overcharge Co. Premium savings account, or a Overcharge Co. Platinum savings account?” ”Uhh I think it is a Premium.” ″I’m so sorry. if you have a Overcharge Co. Platinum savings account then you will not be able to enjoy our Overcharge co. ultrabonus points loyalty system. However you may be suprised that for only a small increase in account fee, you too can enjoy the range of rewards and discounts offered with the Overcharge co. ultrabonus points loyalty system. Would you like to learn more?”
These sorts of behavioral choices are determined by the feedback given by the people who train the AI. Nothing to do with the AI’s architecture or fundamental inclinations.
So the question to ask is, “Why do all the AI companies seem to think it’s less ideal for the AI to ask clarifying questions?”
One part of the reason is that it’s a lot easier to do single turn reinforcement. It’s hard to judge whether a chatbot’s answer is going to end up being helpful if it’s current turn consists of just a clarifying question.
Yes I assumed it was a conscious choice (of the company that develops an A.I.) and not a limitation of the architecture. Although I am confused by the single-turn reinforcement explanation as while this may increase the probability of any individual turn being useful, as my interaction over the hallucinated feature in Instagram attests to, it makes conversations far less useful overall unless it happens to correctly ‘guess’ what you mean.
Why don’t LLM’s ask clarifying questions?
Caveat: I know little to nothing about the architecture of such things, please take this as naive user feedback if you wish, or you could ignore it.
Just now I was asking the Meta AI chatbot how to do an ‘L-Cut’ using the Android Instagram app. It hallucinated for quite a few messages instructions how to ‘overlap’ two video tracks when editing a reel before it finally admitted that no such ability in fact exists in the Instagram App.
My grossly negligent mistake was assuming that a AI LLM with Meta Branding would have current or accurate knowledge of Meta properties and products.
However, imagine that there are two versions of the Instagram App, one that has this feature and one that doesn’t—why didn’t it ask “Just to check, what kind of phone are you using?” (which, also, would allay fears of ‘spying’ and invasion of privacy) and then, and only then give me advice or apologetically inform me that such a feature is not available. (In other branded/commercial chatbots, wouldn’t this be an opportunity for an up-sell?[1])
Maybe I’m not prompting LLMs in the right way, but they almost never ask me questions. And certainly never clarification questions. They make an assumption, and produce a slab of text with a boilerplate “let me know if this helps?” tacked on the end. I assume this is a conscious choice and not a limitation of the technology.
To test my observation I entered the same prompt into three different LLMs:
I just asked Claude Haiku 3 “When was Rosberg’s first Grand Prix?” to see if it assumes I mean Keke Rosberg, or his son, Nico Rosberg.Claude apologies for not having information on Nico Rosberg’s first Grand Prix… and then goes on to state that “Formula One has had multiple drivers named Rosberg compete over the years. More details would be needed to determine the exact driver and their debut race.”
What I find interesting, not one sentence in the answer takes the form of an interrogative!
Next, I asked GPT-4o mini the exact same question, to which it confidently assumes which Rosberg I mean: “Nico Rosberg’s first Formula One Grand Prix was at the 2006 Bahrain Grand Prix, which took place on March 12, 2006. He raced for the Williams team.”
Microsoft CoPilot makes the same assumption, with the traditional tacked on boilerplate question:
”Nico Rosberg made his Formula One debut at the 2006 Bahrain Grand Prix. He went on to have a successful career, culminating in winning the World Drivers’ Championship in 2016. Is there anything else you’d like to know about his career or Formula One in general? [FI car Emoji, Smoke Emoji]”
Meta AI makes the same assumption, and gives a lot more unsolicited information, but it does mention his father:
”Nico Rosberg’s first Grand Prix was the 2006 Bahrain Grand Prix. This marked the beginning of his Formula One career, which would eventually see him drive for Williams from 2006 to 2009 and then Mercedes from 2010 to 2016. During his time with Mercedes, Rosberg won the Formula One Driver’s championship in 2016, a feat that made him and his father, Keke Rosberg, the second-father pairing to achieve this impressive milestone”
Not that it is relevant, but Graham and Damon Hill were the first.
This is not an exhaustive sample by any means, but does confirm my naive observation that LLMs do not ask clarifying questions or use interrogatives in order to better shape their responses to my needs.
I imagine such a commercial hellscape would look a little like this:
”I was just wondering why I wasn’t earning any Ultrabonus points with my purchases”
“Before we continue, could you tell me, do you have a Overcharge Co. Premium savings account, or a Overcharge Co. Platinum savings account?”
”Uhh I think it is a Premium.”
″I’m so sorry. if you have a Overcharge Co. Platinum savings account then you will not be able to enjoy our Overcharge co. ultrabonus points loyalty system. However you may be suprised that for only a small increase in account fee, you too can enjoy the range of rewards and discounts offered with the Overcharge co. ultrabonus points loyalty system. Would you like to learn more?”
These sorts of behavioral choices are determined by the feedback given by the people who train the AI. Nothing to do with the AI’s architecture or fundamental inclinations.
So the question to ask is, “Why do all the AI companies seem to think it’s less ideal for the AI to ask clarifying questions?”
One part of the reason is that it’s a lot easier to do single turn reinforcement. It’s hard to judge whether a chatbot’s answer is going to end up being helpful if it’s current turn consists of just a clarifying question.
Yes I assumed it was a conscious choice (of the company that develops an A.I.) and not a limitation of the architecture. Although I am confused by the single-turn reinforcement explanation as while this may increase the probability of any individual turn being useful, as my interaction over the hallucinated feature in Instagram attests to, it makes conversations far less useful overall unless it happens to correctly ‘guess’ what you mean.