best practices for developing natural user interaction with intel realsense using the nuia sdk

Ultracode Munich #8 / Intel® RealSense™

Munich, 31th March 2015

31.03.2015 4tiitoo @ Ultracode Munich #8 / Intel® RealSense™ 1

„4tiitoo’s vision for reshaping how the masses interact with computers could become a reality sooner rather than later.“ – TechCrunch

People are happy with PCs, Laptops and Tablets!

American Customer Sa0sfac0on Index (ACSI) for PCs ranges 77 to 87 out of 100

(Ann Arbor, 2013)*

* For comparison: ACSI for celluar telephones ranged 70 to 72 (AA, 2007) and 71 to 81 (AA, 2013)


People are happy with PCs, Laptops and Tablets!

American Customer Sa0sfac0on Index (ACSI) for PCs ranges 77 to 87 out of 100

(Ann Arbor, 2013)*

Why should we consider

Natural User Interac0on


Image source: http://www.blastr.com/sites/blastr/files/MinorityReportTomCruise-1.jpeg

The impact of Natural User Interaction will be similar to the invention of the mouse and GUIs

Because

Gesture: Humans internalize declara0ve

poin0ng at the age of 1

Speech: Humans can process 400 words per minute (WPM), speak 160 WPM but type only 33 WPM

0

100

200

300

400

500

Proceesing Speach Type

Gaze: The eye can iden0fy an image in 13 milliseconds. Gaze allows real 0me access to our point of focus

Posture: Carpal tunnel syndrome costs the US $ 1.2 to 2.3 bio. per year


Generic enabling

Sensor capabilities

Device Poin5ng Commands Text Natural Input speed Coverage

Mouse + 0 -‐ -‐ -‐ -‐ Touchpad + 0 -‐ -‐ -‐ -‐ Keyboard -‐ + + -‐ -‐ -‐ Touchscreen + + -‐ + -‐ -‐ Speech -‐ + 0 + + + Gaze + -‐ -‐ + + -‐ Gesture 0 + -‐ + -‐ + Neuro -‐ 0 -‐ + -‐ -‐ 360° HUD NA NA NA + NA +

NUI should combine multiple aspects of human interaction to combine capabilites and increase robustness


Generic enabling

Sensor overview for user interaction

Device < 1 m 1 m to 2 m 2 m – 6 m

Mouse

Touchpad

Keyboard

Touchscreen

Speech

Gaze

Gesture

Neuro

360° HUD

NUI will eventually free humans of being bound to a device and increase features in all dimensions incl. Screen size


NUI vs. GUI


•  NUI hardware is still in a quite early stage

•  Precision and recognition rate are much lower: A gesture may not be recognized immediately/definitely and a gaze click might not exactly be at the place the user looks at

•  Interfaces can be more intuitive, but on the other hand also not as clear as traditional user interfaces: Which voice commands or gestures are available?

•  Direct manipulation of objects displayed to the user is possible

Best practices - General

•  Do not try to recreate traditional interfaces: Design for as little learning as possible and use familiar interaction principles

•  Think of usage scenarios, e.g. in open space office a speech interface may not be optimal due to distraction of other users. Similarly, do not require the users to perform the same hand gesture a hundred times since they will tire of it quickly.

•  Actively plan for false positive and false negatives Just like human interaction misunderstandings are common. Today users expect devices to respond error free to any input, so you need strategies:

–  Direct feedback ideally with option to continue or undo

–  Competing detection algorithms

–  Minimize requirements regarding precision

–  “Double check” either by combining sensors or based on history

•  Test, evaluate with several people and environments and tune the user experience


A simple example derived from human communication


RealSense is a greenfield of opportunities and the right usage model is not yet defined

Human communica5on: •  Usually the person looked at

feels adressed •  To adress another person, the

name is used •  If you don‘t feel adressed you

ignore commands

Speech recogni5on with RealSense: RealSense detects that user is facing the computer Ø  Lower required confidence

level for speech commands Ø  Don‘t expect trigger words

Speech recogni5on with RealSense: RealSense detects that user is not facing the computer Ø  Higher required confidence

level for speech commands Ø  Require trigger words

Best practices - Gesture control


•  Use gestures for simple commands and direct manipulation

•  Use natural and intuitive gestures, ideally inspired by RL or at least iconic

•  Performing the gestures should be possible with both hands

•  Keep in mind cross-cultural audiences and user variation, since a gesture can mean very different things in different places and users can have very different habits.

•  Do not occlude the screen with the users hands

•  Give feedback about the camera FOV. The hands should not be "just lost" because the user accidentally left the camera's FOV.

•  Stay consistent: The meaning of gestures should not change wildly with context.

•  Make sure recognition is robust across different platforms with different camera angle and user distance (Notebook, AIO, Add-on)

Best practices – Voice control


•  Use voice for more complex commands and when not within range of a computer’s other sensors

•  Voice commands should be based around natural language usage

•  Use synonyms for the commands to be more flexible and tolerant in interpreting the input

•  Avoid using a single word as command (ideally two or three words)

•  Either avoid dangerous commands or implement confirmation due to possible false negatives

•  Be aware of pauses and interjections during speech input

•  Consider environmental noise and social appropriateness

•  Always show the status of the speech engine (listening, not listening, processing sound)

•  If you implement voice feedback make it short and intelligent

The biggest challenge for the market entry

of new sensor technologies is

the absence of valued use cases.


NUIA Extension technology in a nutshell

ü Dedicated support of existing “killer” apps

ü SDK designed to facilitate enabling of existing apps without code changes

ü Enables rapid prototyping

Missing support

Millions of existing applications …

… wait to be powered by your next gen device


NUIA Core

Combining the Natural User InterAction capabilities of all available sensors

Standard Software

OS

Frameworks

Webcam Webcam SDKs

Speech Speech SDKs

Gesture Gesture SDKs

Gaze Gaze SDKs Neuro

Neuro SDKs ... Other SDKs

Dedicated NUI Software

Generic Interaction Dedicated Extensions

Plugins

4tiitoo

4tiitoo + 3rd party devs


Extensive experience in enabling Applications


Popularity

Business usage

Home usage

More than 100 applications and web applications enabled with NUIA

NUIA Shape Sorter - Overview


About us

4tiitoo

•  Specialized in software for natural user interaction since 2007

•  Team received Gartner “Cool Vendor” award for Human-Machine Interfaces in 2013

•  Follow us via NUIA.com or

facebook.com/NUIAway twitter.com/NUIAway

youtube.com/user/NUIAway linkedin.com/company/4tiitoo

Contact Stephan:

•  [email protected]

•  linkedin.com/in/stephanodoerfer/

•  4tiitoo @ Ultracode Munich Special: Contact me if you are interested to join our NUIA Next Level IDE Support Program


We are hiring!