Speech interaction studies
Original supporting material for Shipping the Element of speech. These studies preserve the original wording and diagrams.
Defining work streams: Bringing UXR findings, engineering technology and UX together
Pardon the interruption - The UX need was based on the tendency when hearing the response from the agent they are communicating with but the mic generally was not open during speech. Engineering need was how do we utilize our NLP stack to fulfill this… by utilizing NLU or parsing text and using RNN (recurrent neural networks) to comprehend meaning. And the UXR finding kept pointing to people needing less rules in conversation like breaking norms and interrupting the agent.
Partial Fulfillment - The UX need here was to begin to reflect state to show speed and develop an app framework that utilizes a natural interaction and reduces wait time. We eventually filed a patent for this under the idea of "anticipatory computing" or the idea that you could work through an app, building on needs, refining them, etc. similar to scenarios one might recall from Hollywood (Ironman) but more like help me find a restaurant I ate at last night… near kings cross… not inside the station… yes that one!"
Speak at your own pace - The UX need here came from the big roadblock that kept coming up in UXR about how people feel pressured like they are in a spelling bee or similar…
Original interaction studies
Convey degree of certainty - UX - voice plate appears but text is not bold
Understand unfinished utterances - UX - UI pattern of timer loaded and but we don't assume time letting user fill in the rest
Adjust listening endpoint - UX - pause the fulfillment and keep listening until we are certain a user is done
Understand emotional prosody - when uncertain show a disambiguation screen relating to context (e.g. calls)
Buy time - Show UI of action happening but pause the actual call for 2 seconds for user to give them that they are certain
View original storyboard ↗Detect return addressee - possible side-speech update on UI and assistant responds minimally to indicate listening
View original storyboard ↗Resolve inexact object reference - don't be pedantic by pointing out an error, just focus on fulfillment based on context
Listen while speaking - a user can barge in and ask for volume up (e.g. long name of a song)
Correct in-progress action - punt should be helpful and when info isn't available give tips related to similar queries
Modify free text field
Confirmation
Correct completed action - UX here is similar to voice typing and a user says scratch that, etc.