personal universal controllers: controlling complex appliances with guis and speech

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #1

Jeffrey Nichols and Brad A. MyersCarnegie Mellon UniversityApril 8, 2003

Personal Universal Controllers:Controlling Complex Appliances with GUIs and Speech

CHI 2003 Conference on Human Factors in Computing Systems Ft. Lauderdale, Florida

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #2

The Problem

Appliances are too complex

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #3

The Problem, cont.

Each complex appliance has its own idiosyncratic interface!•Home and Car Stereos•VCRs & Camcorders•Car Navigation Systems•Answering Machines•…

Increasingly Computerized Low Usability

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #4

Our Solution

Separate the interface from the appliance!

Handheld becomes personal universal controller (PUC)Key Features

User interface-independent appliance specification Automatic generation of GUI and speech interfaces

Specifications

Control

Feedback

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #5

Automatic Generation of UIs

Benefits All interfaces consistent for the user

With conventions of handheldOther applications and UI guidelines

Even from multiple manufacturersAddresses idiosyncracy problem!

Multiple modalities (GUI + Speech UI)

Can take into account user preferences

Will work on special purpose devices (for disabled)

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #6

Overview

Appliances Controller Devices

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #7

Demonstration #1

Sony CamcorderPocketPC, Smartphone, Speech, Tablet PC

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #8

- Appliance Adaptors- Comm. Protocol- Interface Generators

Architecture

XML-based

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #9

XML-based Specification LanguageDescribes appliance with these features:

Functions of DeviceState Variables and Commands

LabelingMultiple labels are necessary

GroupingHierarchical groups

Dependency InformationFor enabling and structure

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #10

Demonstration #2

Windows Media PlayerPocketPC, Smartphone

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #11

Demonstration #3

GMC Information System / Climate ControlPocketPC

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #12

Important Work By Others INCITS V2 Standardization Effort

SIG at CHI 2003Toward a Unified Universal Remote Console StandardThursday 9:00-10:30, in Room 122

Xweb (now ICE)[Olsen Jr., UIST 2000]

Stanford iRoom, iCrafter[Ponnekanti, Ubicomp 2001]

Speakeasy[Newman, UIST 2002]

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #13

Future Work Improve quality of generated user interfaces

More and better rules

Specification and generation of more appliances answering machine, navigation system

Consistent interface generation New interfaces modeled on past interfaces

Generating interfaces for the “experience” One interface generated for multiple connected

appliances

General purpose appliance adaptor for HAVi and UPnP protocols

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #14

Acknowledgements

Funding National Science Foundation Microsoft General Motors Pittsburgh Digital

Greenhouse

Equipment Grants Mitsubishi (MERL) VividLogic Symbol Technologies Hewlett-Packard Lucent

PUC Project Members Brad A. Myers Thomas K. Harris Roni Rosenfeld Michael Higgins Joseph Hughes Kevin Litwack Rajesh Seenichamy Mathilde Pignol Stefanie Shriver Jeffrey Stylos Peter Lucas

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #15

Conference on Human Factors in Computing Systems (CHI)April 8, 2003

Thanks!

http://www.cs.cmu.edu/~pebbles/puc/http://www.cs.cmu.edu/~jeffreyn/

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #16

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #17

Rise of Mobile Devices

Mobile devices are becoming cheaper and more common

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #18

Demonstration #4?

Two-way X10 LightingPocketPC, Smartphone

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #19

X10 Demo #1

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #20

X10 Demo #2

Back

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #21

DV Camera #1

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #22

DV Camera #2 Back

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #23

Windows Media Player Back

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #24

GM Driver Information SystemBack

Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 � � �Slide #25

GM Climate Control System Back