studying the use of handhelds to control smart appliances

Jeffrey Nichols • 1 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003

Studying The Use of Handhelds To

Control Smart Appliances

Jeffrey NicholsCarnegie Mellon University

May 19, 2003


The Problem

Appliances are too complex


The Problem, cont.

Each complex appliance has its own idiosyncratic interface!• Home and Car Stereos• Car Navigation Systems• Answering Machines• …

Increasingly Computerized

Low Usability


Our Solution

Separate the interface from the appliance!

Key Features User interface-independent appliance specification Automatic generation of GUI and speech interfaces

Specifications

Control

Feedback


Benefits of Our Approach

Handheld has richer interface technology than appliance can afford Color LCD screen, touch screen, text entry technology

More effort can be put into interface design technology Appliance manufacturer’s must weigh trade-offs

between usability, cost, time-to-market, etc.

Two-way communication channel Better feedback can be provided to the user regarding

the appliance’s state.


Automatic Generation of UIs

Benefits All interfaces consistent for the user

With conventions of handheldOther applications and UI guidelines

Even from multiple manufacturersAddresses idiosyncracy problem!

Multiple modalities (GUI + Speech UI)

Can take into account user preferences

Will work on special purpose devices (for disabled)


Outline

A First Step

User Studies

Current Work


A First Step

Build Reference Interfaces

Remote control interfaces for various appliances that we design manually.

Verify that better interfaces can be created on a handheld

Used for understanding what functional knowledge is necessary to make a good interface.


Reference Interfaces

Interfaces were hand-designed for two appliances and two handhelds Appliances

AIWA CX-NMT70 Shelf Stereo AT&T 1825 Telephone/Answering

Machine Handhelds

Palm Microsoft PocketPC


Palm Interfaces

We initially designed paper-prototype interfaces for Palm

telephone stereo


PocketPC Interfaces

We later implemented interfaces for Microsoft’s PocketPC (simulated remote control).

telephone stereo


Interface Quality?

We iteratively improved the interfaces using heuristic analysis techniques.

We conducted a think-aloud study with several Carnegie Mellon students to find problems in the interfaces.

Lastly, we conducted a user study that compared our reference interfaces with the manufacturer’s interfaces.


Outline

A First Step

User Studies

Current Work


User Studies

Two Studies Study #1:

Paper-Prototype Palm vs. Actual Appliance

Study #2: Functional PocketPC vs. Actual Appliance


User Studies, cont.

Procedure We did a between-subjects study.

Each subject worked on two sets of tasks. In order to minimize subjects, each worked

on both the stereo and the phone.

We controlled for order and interface type.


Evaluation of Task Performance

Three Metrics: Time to complete all tasks

Number of times help was requested How often did the subject need the manual

or online help?

Number of missteps Misstep = the pressing of a button that does

not advance the progress on the current task

No missteps were counted after the user requested help.


User Study #1: PalmOS

Compared paper prototype interfaces with the interfaces of the actual appliances Experimenter changed paper

as subjects tapped

Control of stereo and phone simulated verbally

When the stereo started playing music, the experimenter said “you now hear music from the stereo”


User Study #1, cont.

Participants 13 Carnegie Mellon Graduate Students

Five female, Eight male Enrolled in School of Computer Science Volunteers (unpaid) Seven owned a Palm device One had no Palm experience Four owned Aiwa-brand stereo systems


User Study #1 Results

Users made five times the errors and needed help twice as often with the actual appliances!

All results significant (p < 0.001 for all)


User Study #2

We implemented the interfaces on a handheld and simulated remote control of an actual appliance. Remote control applications built

in Visual Basic on a PocketPC

Control of stereo and phone simulated in software

Feedback appeared to come from the actual appliance


User Study #2, cont.

Participants Twelve students from Carnegie Mellon

Four female, Eight male

Volunteered in response to a newsgroup advertisement

Paid $7 for their participation (30-45 minutes)

All had limited handheld experience

Half (6) owned Aiwa-brand stereos

Two had AT&T digital answering machines


User Study #2 Results

All differences are significant (p < 0.05)

About ½ the time and ½ the errors!


Qualitative Results

Why were the reference interfaces better? Clear feedback and explanation of the

current state was possible.

Elements could be disabled on the screen (graying out)

Functions were separated across multiple screens.


Outline

A First Step

User Studies

Current Work


Current Work

Designed a XML-based Specification Lang Functions of Device

State Variables and Commands

LabelingMultiple labels are necessary

GroupingHierarchical groups

Dependency InformationFor enabling and structure


Current Work, cont.

Built multiple automatic interface generators PocketPC

SmartPhone Tablet PC

(desktop)

Speech


Control of Real Appliances


Acknowledgements

Funding National Science

Foundation Microsoft General Motors Pittsburgh Digital

Greenhouse

Equipment Grants Mitsubishi (MERL) VividLogic Hewlett-Packard

PUC Project Members Brad A. Myers Thomas K. Harris Roni Rosenfeld Michael Higgins Joseph Hughes Kevin Litwack Rajesh Seenichamy Mathilde Pignol Stefanie Shriver Jeffrey Stylos Peter Lucas


Thanks!

For more information see… http://www.cs.cmu.edu/~jeffreyn/

http://www.cs.cmu.edu/~pebbles/puc/

Or e-mail me at… [email protected]


Actual Appliance Interfaces

Lots of Problems Poorly labeled and overloaded buttons Insufficient feedback

Timer example Programming the speed-dial

Phone has technical separation between phone and answering machine


Qualitative Results

Grouping controls is important Groups define which elements are

placed adjacent to each other and how elements are separated onto pages.

Groupings vary between devices and interface styles.


Qualitative Results, cont.

Dual-associated functions are hard to make obvious for users The record button is associated

with both tapes (record onto) and each of the other modes (recorded from).

Some users expected the first mapping to used, whereas the controller used the second mapping.



Tree-based structures are not sufficient for fully describing an interface Some interface concepts, especially dual-

associated functions, break the tree because they may interact with the children of several different elements within the tree

The record button breaks the stereo’s tree structure because it is globally accessible but has different local effects.



A single function may map to multiple interface widgets (and vice versa) Example: One state variable could be used

to represent all of the playback states of a tape player. The play, stop, fast-forward, and rewind buttons all act on this single variable.


Applying These Results

We are actively applying these results to the design of the specification language A tree-grouping structure is augmented

with a dependency graph to help describe dual-mapped functions

Ranking relationships within groups using “priorities”

We will also apply them in the design of the automatic layout engine


Future Work

Build the specification language and automatic generation engine


A Hard Problem…

Automatically generating a good user interface is hard, but we think we can do it for several reasons:

Remote controls are a special class of user interface that use relatively simple interaction techniques.

Buttons, text fields, and other standard widgets.

Our approach differs from earlier work…


The Approach

Study Interfaces Functional knowledge of the appliance

What must the appliance tell the handheld about itself so that a “good” interface can be constructed.

Design and Layout How do we turn the knowledge about the

appliance into a usable interface?

Design a specification language

Build an automatic interface generator


Our Progress…

Study Interfaces Functional knowledge of the appliance

What must the appliance tell the handheld about itself so that a “good” interface can be constructed.

Design and Layout How do we turn the knowledge about the

appliance into a usable interface?

Design a specification language (in progress)

Build an automatic interface generator


Problems with User Study #1

Paper-prototype study introduced a high possibility of experimenter interference.

Solution Create an environment that completely

simulates what one might experience using a personal universal controller

Interfaces running on an actual handheld Interfaces should appear to control an

actual appliance

studying the use of handhelds to control smart appliances

Documents