Amazon Sumerian Concierge Experience
Developing Communication Skills for an AI-driven Humanoid Host
Amazon Sumerian Concierge Experience is a personalized virtual concierge demo where Amazon Sumerian's humanoid host Cristine introduces users to the Sumerian team. She greets her teammates by name and shows them their desk location when she recognizes them through a webcam, and she helps visitors learn more about the office space. The demo was built as a use case of virtual hosts at conferences, and the full version walks users through a separate registration flow to enable facial recognition.
This experience is an exploration of whether we can increase user engagement by enabling natural conversation techniques for an AI-driven character. Face tracking is paired with the host's point of interest system so that Cristine can maintain eye contact and turn her body towards users to show that she's paying attention. We used emotion analysis to adapt Cristine's responses to the nuances of the users' facial expressions. Users often commented on their appreciation for subtle adjustments in her speech and her courteous personality in which she responds to them thanking her and not just to their commands.
We've made the demo available as a starter template in Amazon Sumerian by popular demand. Our customers can quickly create a similar experience for smartphones, laptops, and touch screens by simply configuring their AWS resources. Paced caption texts alongside voice, touch, and click interactions are enabled to increase accessibility and to accommodate different environments and hardware.
My role I lead the project from ideation to production as well as managed the internal process to open source it. I conducted research, visual design, voice use interaction, created prototypes, and wrote production code. This project started as a pet project with Jinil J., a software engineer on the team, and it became a showcase demo shared at multiple events such as AWS Summit and Loft. Jinil set up the backend for facial recognition, and I collaborated with him to implement the computer vision features and optimization using web workers and by experimenting with web assembly. I worked with our senior UX designer Courtney A. to conduct user research and refine user interaction.
Contact Please contact the Amazon Sumerian team for any questions related to Amazon Sumerian.
Amazon Sumerian | 2018
Programming LanguagesJavaScript, built with Amazon Sumerian, AWS JavaScript SDK, and jsfeat
Demos + Publicity
AWS Summit - London
AWS Loft - New York
SIGGRAPH 2018 workshop
Demo shown to more than 50 C-level executives
Open source project
Cristine Conversing with a Teammate from the Sumerian Team
The left panel demonstrates facial recognition. Cristine greets me by name and highlights my desk space.
Underlying Technology
The open source version uses Amazon Rekognition [facial recognition], Lex [conversation bot], Polly [text-to-speech], DynamoDB [NoSQL database], and S3 [object storage] from AWS JavaScript SDK as well as jsfeat [face detection] and web audio/RTC APIs. The original demo used serverless computing using AWS Lambda and Open CV for face detection.
Host Pays Attention to the Users
Cristine rotates her body towards the user and makes eye contact so that face tracking feels more natural.
A Showcase Demo and an Open Source Project
This was the first project created with Amazon Sumerian that integrated AWS's AI technologies with a Sumerian host. This project helped us to improve its tool and the host system as well as explore various techniques commonly asked by our customers. Since our customers include software developers, artists, and designers, we paid careful attention to the ease of configuration and extensibility of the open source project from both code and asset management perspectives.
* Image courtesy of Amazon Sumerian