www.ics.com

Integrated Computer SolutionsUnleashing the Power of Modern Devices

DevelopingNatural User Interfaces

in Qt

Presented By

Jeff LeBlancUser Experience Team Lead @ ICS

Software developer for 20+ yearsCertified Qt trainer since 2003

Adjunct faculty at WPI teaching HCI

Contact me: jeffl@ics.com

Seen any good movies recently?

Looking to the Past

• Fiction has inspired innovation and design foryears• Jules Verne inspired Simon Lake• H.G. Welles inspires Robert Goddard

Looking to the Future

Make It So: InteractionDesign Lessons fromScience Fiction (Shedfroff,Noessel 2012)

http://www.scifiinteraces.com

Looking at Hollywood

• What they get wrong• A lot!!

“It’s a UNIX system!”- Jurrasic Park (1993)

Looking at Hollywood• When they get it right…

• Emotional impact, visceral appeal

• Norman describes three levels of emotionaldesign• Reflective• Behavioral• Visceral

Looking at Hollywood

Jayse Hansen - http://www.jayse.tv/

Inspirational Design

Natural User Interfaces

• A Natural User Interface (NUI) is a UI that• Becomes effectively invisible• Is based on natural elements

• Goes back to Steve Mann (70s, 80s)

Inspiration! Can we combine a NUI with AI tocreate an intelligent agent?

Inspirational Design

+ =

Intelligent Agents

• Artificial Intelligence: A Modern Approach(Russell, Norvig 2003)

Intelligent Agents

• Intelligent, or rational, agents work with theirenvironment• Perceive through sensors• Affect using actuators

Apple’s Agents

1987 – Knowledge Navigator

Today - Siri

Gartner Hype State - 2013

Smart machines, including “intelligent personal assistants” are expected to be a majortechnology trend for 2014

Goals of This Project

• Combining many differentmodalities• Input: voice, touch, gesture,

sensors• Output: multi-media

• Produce an experience asnatural as interacting in thereal world with anotherperson

Touch

Touch

• Interacting with a computerusing finger(s) instead ofpointing device (mouse)

• Touch differs from mouse• No tracking state, no hover• Multiple contact points possible• Reduced accuracy, especially at

borders• No physical affordance

Designing for Touch

• Plan for gestures and flicks• Support multi-touch• Design for the “fat finger”

problem• Larger targets, more space

between• Expand touch areas around

targets

Gestures

Gestures• Currently requires additional hardware,

specifically a depth camera such as• Microsoft Kinect• Creative Senz3D camera• Leap Motion

Gesture Cameras

• Stereo

• Infrared / structured light• Kinect

• Time-of-Flight• Creative / Senz3D

Designing for Gestures

• Creating gestural language• Arm fatigue• The “Live Mic” problem

• Unintentional gestures• false positives

• Unrecognized gestures• false negatives

Designing for Gestures

• Techniques to reduce false positives• Reserved / special actions• Clutching• Multi-modal input

• May detract from the naturalness of theexperience

Voice Interaction

Voice Input• Planning for different interaction modes

• Command• Limit vocabulary to intended command words• Upwards of 95% accuracy

• Dictation• Free form, speech to text• Upwards of 85% accuracy, training helps

• Agent• Combination of Command and Dictation• Interactive

Trust and Confidence

• Similar “Live Mic” issues to gesture• False positives

“My fellow Americans, I'm pleased to tell you today thatI've signed legislation that will outlaw Russia. We beginbombing in five minutes.” – Reagan, 1984

• Commands not being recognized• False negatives

Voice Output

• Voice Synthesis / Text-to-speech (TTS)• Degree of naturalness varies• Highly flexible

• Canned responses• Telephony systems• More natural, but limited

Case Study

• Build an “Intelligent” Agent• Model-based agent: store state of the world

• Intelligent Presentation Assistant (IPA)• AKA “Jarvis”

• Advance slides by gestures• Voice triggered actions

Intel Perceptual Computing SDK

• Intel initiative, started in 2012• Free SDK, limited hardware (only supports

Creative camera)• C++ based API• Contains voice, gesture tracking (with limited

fingers), facial recognition

Solution for the Live Mic

Use the SDK’s Facial Recognition capabilityto determine if someone is looking atcamera

Voice Recognition Issues

• Accuracy level has varied betweenversions of the SDK and target hardware

• Current solutions• False positives: reject matches below 48%

confidence level• Command mode• Hardware selection, tuning and filtering

Voice Recognition Accuracy

• In practice, simple commands recognizedwith 40% to 60% confidence

Voice Recognition Accuracy

• In practice, simple commands recognizedwith 40% to 60% confidence

Qt Integration

QML front-end(s), C++ backend

Qt Integration

• Put various recognition routines off in their ownQThreads• doWork() slots have the recognition loops

Qt Integration

• Create signal-based API for recognition• Gestures – swipeLeft(), swipeRight(), etc• Voice – voice(const QString&)• Face – facePresent(bool)

• XML parser for voice grammar• Canned voice output done with QMediaPlayer

Future Work• Explore other cameras / hardware

• Bluetooth headset mics• Kinect

• Native SDK very “Microsoft-centric” API, may need to useQAbstractNativeEventFilter or 3rd party libraries

• Leap Motion• Released in July ‘13, early Qt integration done

• Explore other voice APIs• Microsoft SAPI in progress• Headset mic vs array mic to reduce ambient noise

• Open source solutions for Linux support

Thanks for Attending