jeffrey nichols user interface software & technology (uist) october 30, 2002 slide #0 jeffrey...
TRANSCRIPT
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 1
Jeffrey Nichols and Brad A. MyersCarnegie Mellon UniversityOctober 30, 2002
Generating Remote Control Interfaces for Complex Appliances
The Fifteenth Annual Symposium on User Interface Software & Technology (UIST) 2002 Paris, France
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 2
The Problem
Appliances are too complex
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 3
The Problem, cont.
Each complex appliances has its own idiosyncratic interface! Stereo systems Telephones VCRs Alarm Clocks …
Increasingly Computerized
Low Usability
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 4
Our Solution
Separate the interface from the appliance!
Handheld becomes personal universal controller (PUC)Key Features
Interface-independent appliance specification Automatic generation of GUI and speech interfaces
Specifications
Control
Feedback
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 5
Automatic Generation of UIs
Benefits All interfaces consistent for the user
With conventions of handheld Even from multiple manufacturers
Addresses hotel alarm clock problem!
Multiple modalities (GUI + Speech UI)
Can take into account user preferences
Will work on special purpose devices (for disabled)
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 6
Important Work By Others INCITS V2 Standardization Effort
Alternative Interface Access Protocol (AIAP)[Zimmermann, CHI 2002]
User Interface Modeling Language (UIML)http:/www.uiml.org/
Xweb (now ICE)[Olsen Jr., UIST 2000]
Stanford iRoom, iCrafter[Ponnekanti, Ubicomp 2001]
Speakeasy[Newman, UIST 2002]
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 7
- Comm. Protocol- Interface Generators- Specification Lang.
Architecture
XML-based
- Appliance Adaptors
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 8
Language Design
Approach Create reference interfaces
AIWA Shelf Stereo AT&T Telephone/Answering
Machine
Test interfaces with subjects Users twice as fast and made
half the errors with reference interfaces as compared to manufacturers’ interfaces
Analyze interfaces for functional information
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 9
Language Elements
State Variables and Commands Represent functions of appliance State variables have types
Boolean, Enumeration, Integer, String, etc. Variables sufficient for most functions but not all
“seek” button on a Radio
Label Information One label not suitable everywhere
The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and
text-to-speech information
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 10
Language Elements, cont.
Group Tree Specify
organization of functions
We use n-ary tree with variables or commands at leaves
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 11
Language Elements, cont.
Dependency Information Formulas that specify when a variable or
command is active in terms of other state variables
Equals, Greater Than, Less Than Linked with logical operators (AND, OR)
For example, <and>
<equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals></and>
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 12
Interface Generators
Generators for Two Modalities Graphical
Implemented for PocketPC in Java 1.1 Uses dependency information to generate panel
structure of interface
Speech Implemented using Universal Speech Interface
(USI) techniques [Rosenfeld 2001] Uses dependency information to disambiguate
shortcut words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 13
Graphical Interface Generator
Focuses on panel structure of user interface
Small groups of controls have basic layouts
Complexity comes from structure of groups
Structure can be inferred from dependency info!
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 14
Inferring Structure
Find sets of variables that are “mutually exclusive” Every variable in a set will
never be active at the same time as a variable in another set
Create structure with sets, using overlapping panels
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 15
Choosing Panel Typesa)
b)
c)
full screen
tabbed
partial screen
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 16
Making the Interface Concrete
Finish conceptual layout Choose controls (decision
tree) Choose row layouts
(one column, two column, etc.)
Allocate space Examine panel contents and
choose sizes
Instantiate and place controls
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 17
Generating Speech Interfaces
Automatically build USI tree from dependencies Allows verbal navigation of functional groups
Automatically generate grammar for parser Phrases for query and control
“What is playmode?”“Set playmode to play”“play”
Automatically generate language model and pronunciation for recognizer
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 18
Controlling Appliances
We have built adaptors for many actual appliances
Sony Digital Camcorder X10 Lighting Audiophase Shelf Stereo AudioReQuest MP3 player Software Media Players (WinAmp, WMP)
Written specifications for others Elevator Telephone/Answering Machine
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 19
Demonstration
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 20
Future Work Improve quality of generated user interfaces
Back-tracking, etc.
Specification and generation of list interfaces Necessary for many appliances
Adaptive interface generation New interfaces modeled on past interfaces
General purpose appliance adaptor for HAVi and UPnP protocols
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 21
Acknowledgements
PUC Project Members Brad A. Myers Michael Higgins Joseph Hughes Thomas K. Harris Roni Rosenfeld Mathilde Pignol Stefanie Shriver Peter Lucas Kevin Litwack
Funding National Science Foundation Pittsburgh Digital
Greenhouse Microsoft
Equipment Grants Mitsubishi (MERL) VividLogic Symbol Technologies Hewlett-Packard Lucent
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 22
User Interface Software & Technology (UIST)October 30, 2002
Thanks!
http://www.cs.cmu.edu/~pebbles/puc/http://www.cs.cmu.edu/~jeffreyn/
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 23
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 24
Determining Panel Structure
1. Find states that others depend upon
2. Search for the group that contains this state
3. Determine mutually exclusive sets within group children
4. Assign sets to panels Based upon types, tree
properties
Power
Mode
Band
PlayState
CDTrack
FMStation
FMPresets
AMStation
AMPresets
Radio
CD
Mode == Any
Mode == Radio
Mode == CD
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 25
Choosing Panel Types
Rules for Choosing Panels If…
state has boolean type all controls are one set
Create… Two full-screen overlapping
panels.
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 26
Choosing Panel Types, cont.
Rules for Choosing Panels
If… state has enumerated type
there is a set for every value of the state
Create… a tabbed panel
Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 27
Choosing Panel Types, cont.
Rules for Choosing Panels If…
there are mutually exclusive sets of controls
Create… overlapping panels a control for the state not on
the overlapping panel