Shipping the Element of speech
Mar 11, 2025
What: Talking to robots
Starting from Assistant and the years it took for us to get to Gemini Live I was the lead design for a nascent Speech team. I worked to demonstrate a world where a user could speak more naturally to address an agent.
For this breakdown I plan to focus on how we actualized linguistic concepts through interaction breakdown, prototypes and body storming. A lot of our work centered around three of the big four with linguistics. Mainly pragmatics or the study of how language is used in context, on a smaller scale sociolinguistics (language & society) and historical linguistics (language change).
Linguistic Challenges | Example | Key IXD needs |
|---|---|---|
Topology (SVO) | Sam apples ate | Inclusivity for ESL |
Prosody (rhythm) | Well, of course | Recognition of tone |
Glottal Stop | "Uh[ʔ]oh" | Pausing before fulfilling |
Honorific | "chan" (diminutive) | Locale based honorific |
Table represents a sample of NLP linguistics and IXD considerations.
Room to finish a thought
The three workstreams shared one intent: let people concentrate on what they want to say. Speak at your own pace gives a thought room to develop. Partial fulfillment makes useful progress while it develops. Pardon the interruption lets someone change direction without starting over.
“Alexa usually interrupts me so I get nervous and say as few words as possible.”
How: Bringing it all together
Utilizing an e2e traditional IXD method of breaking down micro-interactions and addressing the tech, the spec and a degree of what users would experience in their day to day.
We aimed to direct our product roadmap through explorations that get into the nuances of linguistics. When we design all these systems we need to keep in mind there is a tolerance of false accept and that a user may not be pausing and then we make processing longer, etc. so we aim to have aspirations:
Give someone time to finish
A call request takes shape on a smart-home display. The person clarifies the contact, hears an acknowledgment, then repeats the request before the call is placed.
Showing progress and committing to an action are different decisions. The two-second hold explores that distinction, but the repeated request shows the unresolved question: does the acknowledgment help the person continue, or leave them unsure whether they were understood?
Recover a missed interruption
A request to change the volume arrives while the assistant is speaking. The sequence follows the missed interruption, the invitation to try again and the eventual adjustment.
When the assistant misses a turn, recovery becomes part of the interaction. A brief invitation lets the person continue without rebuilding the whole request.
Another storyboard shows an attempted correction that fails: the person has to start again. Together, these examples show the cost of losing the conversational thread—before an action, during a response and when a correction is needed.
Illustrative reconstructions using original storyboard artwork. The speaker icon replays with synthetic voices, including overlapping dialogue; these are not original Google voices or recordings. Motion and timing are newly authored, including the two-second hold described in the call study. Timeline labels interpret the visible interaction, not internal model states. Captions abbreviate the long song announcement. View the original studies.
Why: Many users rule out chatting with agents
Users don't know the rules. We used to expect people to know how a robot should be talked to and generally it was command centric. Even when one off commands are accurately accomplished a users expectations are generally quite low due to lack of shared context awareness.
Wrap up: Three things (what is natural?)
What is natural? In the changing space of LLMs we need to temper this as our expectations change to quickly between task oriented and conversational. It is in these two "modals" that context will be key do users need a full UI? Do they just need better under the hood AI API operators?
Understanding unfinished utterances, buying time and generally delaying helps keep the thread of a users need, may keep a users attention and avoid the paralysis a pregnant pause brings (e.g. "un hunh", "mmm") It also ensures that we do not jump in before truly reflecting.
Listening while speaking or… Interrupting LLMs when they go on too long is paramount and a common pattern that exists to have a natural and straightforward dialog while maintaining context.
Be relevant and be out of the way… for LLMs and for Gemini we have a conversation space and are working through command and control with the retirement of Google Assistant.