1 toolkits for physical objects (tuis) jack li advanced user interface software november 30 fall...
Post on 15-Jan-2016
227 views
TRANSCRIPT
1
Toolkits for Physical Objects (TUIs)
Jack Li
Advanced User Interface Software
November 30
Fall 2004
2
Outline
Motivation for TUIs
Input Technologies
TUI Toolkits
3
Current Limitations of UIs Desktops and laptops suck in our entire
attention As a result we live in two worlds: the real
physical world and the virtual cyber world Constantly “wired” in order to be in both
worlds Loss of rich interactions found in
everyday things (particularly haptic)
MOTIVATION FOR TUIS
4
Motivation
Driven by ubiquitous computing Both people and technology moving away
from the desktop into the environment Input: different ways of interacting Output: different ways of displaying
MOTIVATION FOR TUIS
5
Tangible User Interfaces TUIs augment the physical world by integrating
digital information with everyday physical objects
Goal: to bridge the gap between bits and atoms 1. Interactive surfaces 2. Coupling information with physical objects 3. Ambient media for background awareness
MOTIVATION FOR TUIS
6
Outline
Motivation for TUIs
Input Technologies
TUI Toolkits
7
Physical Objects as Input
Pretty much anything … walls, papers, utensils, clothes hangers!
Inspiration sci-fi novels, own imagination everyday life, current practices
INPUT TECHNOLOGIES
8
Devices Cameras Electronic tags (RFID technology) Sensors
Light Motion Heat Force
Motors Orbs
INPUT TECHNOLOGIES
9
Framework for the GUI
(a)model-view-controller
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
INPUT TECHNOLOGIES
10
Framework for the TUI
(b)MCRpd: model-control-representation(physical and digital)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
INPUT TECHNOLOGIES
11
Outline
Motivation for TUIs
Input Technologies
TUI Toolkits Phidgets (2001) Calder (2004) Papier-Mâché (2004)
12
Toolkit Goals
Same as when GUIs were built Make easy for non-specialized experts to
create (should not need a vision or hardware guru on the team)
Hide dirty implementation details Abstract proper functionality with well-
defined API
13
Overview
First toolkit for physical objects Reusable physical devices with proper
abstraction for creating a wide variety of applications
An API that had the right amount of abstraction
PHIDGETS
14
GUI Differences
Connection manager Way of detecting when devices go on/off
Identification Every physical object should have a virtual
counterpart Simulation
A way to test code even if phidget not present
PHIDGETS
15
ArchitecturePHIDGETS
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
16
The Physical DevicePHIDGETS
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Primitive device components Sensors Motors Switches
Circuit board CY7C63000 USB micro-controller
Communications layer USB communication standard
17
Wire ProtocolPHIDGETS
Low-level software in Windows and microcontroller for communication that’s hidden from end programmers
Commands from phidgets Unique Id numbers Device type (servo, powerbar) Specific events (tag for RFID reader)
Commands from Windows Set device state (output like servo)
QuickTime™ and aTIFF (LZW) decompressorare needed to see this picture.
18
Computer SoftwarePHIDGETS
PhidgetManager Connection manager
• Event onAttach()• Event onDetach()• Int count• Enum item
Phidget-specific devices All have:
• String deviceType• Boolean isAttached• Long serialNumber
RFID• onTag()
Servo• onPositionChanged(), motorPosition(index)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
19
Simple ApplicationPHIDGETS
Phidget Manager
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Phidget ServoQuickTime™ and a
TIFF (LZW) decompressorare needed to see this picture.
20
Outline
Motivation for TUIs
Input Technologies
TUI Toolkits Phidgets (2001) Calder (2004) Papier-Mâché (2004)
21
Main goals
Primarily geared towards interaction/product designers
Support for more refined products Support on the hardware side
CALDER
22
Supported DevicesCALDER
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Small size is big win!
23
Fluid prototypingCALDER
Objects easily embedded!
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
24
Communication
Wired PIC16C745 C with USB interface engine Same as Phidgets (and iStuff)
Wireless Tradeoff between size and power (perhaps
Moore’s law will remedy) Small batteries, antenna, and pulse-width
modulation
CALDER
25
Software
Surrogates (GUI widgets) to manipulate physical object output
CALDER
26
Outline
Motivation for TUIs
Input Technologies
TUI Toolkits Phidgets (2001) Calder (2004) Papier-Mâché (2004)
27
Overview
Emphasis on software side side Input model that provides common
programming abstractions Focus on objects being lightweight
PAPIER-MACHE
28
Software Abstractions Phob—encapsulates information related to this
physical object All—source producer, time RFIDPhob—tag VisionPhob—mean color, bounding box, rotations
PhobListener—listens for Phobs produced phobAdded(), phobUpdated(), phobRemoved()
PhobProducer—actual devices that generates events
PhobManager—maintains PhobProducers
PAPIER-MACHE
29
A Common Event Model PhobListener is the interface among all devices Methods are the same
phobAdded(), phobUpdated(), phobRemoved() Vision
phobAdded() when new object detected, phobUpdated() when old object changed (size, color), phobRemoved() when old object disappears
RFID (different from what the paper says) phobAdded() when tag detected, no phobUpdated(),
phobRemoved() when tag disappears Facilitates easier retargetting and substitutions (allows
for more exploration and faster, prototyping)
PAPIER-MACHE
30
Marble Answering Example
1. Get the manager that keeps track of all input devices (PhobProducers)RFIDManager manager = RFIDManager.getManager();
2. Set up a map associating all objects with an audio clipAssociationFactory factory = new AudioClipFactory(); AssociationTreeTable assocMap = new
AssociationTreeTable(factory);
3. Attach this map so new objects are assigned new clips and old ones get played manager.addPhobListenerForAllProducers(assocMap);
PAPIER-MACHE
31
A Behavior: Associations
A common behavior we noticed New objects get stored/assigned, old
objects get retrieved a database of sorts An extension of PhobListener
PAPIER-MACHE
32
A Behavior: Associations
AssociationTreeTable addClassifierFactoryPair(Classifier, Factory)
ObjectClassifier isMemberOfClass(Phob phobToCompare)
AssociationFactory createAssociationEltForPhob(Phob phob)
PAPIER-MACHE
33
Monitoring WindowPAPIER-MACHE
34
Device state—connection mgrPAPIER-MACHE
35
Camera FeedbackPAPIER-MACHE
36
Association MapPAPIER-MACHE
37
ExtrasPAPIER-MACHE
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
38
Summary
39
Major Toolkit differences Phidgets
Complete, balanced, varied assortment of physical objects easy to program with
Calder Small size allows for more fluid prototyping at
the PHYSICAL level Papier-Mâché
Unified event model allows easier development and retargetting at the SOFTWARE level
40
Conclusion
TUIs seek to augment the physical world with digital information
Toolkit support allows non-experts to develop TUI application, like GUI toolkits
Successful toolkits must manage physical connections, gather/alter relevant physical state information, and provide good feedback for these added complexities
41
Future work
Input adaptation M to N mapping of input devices
Input models What other kinds of behaviors can be
abstracted Will be more clear with the development
of more applications
42
References Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits
and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States
Ullmer, B. and H. Ishii, Emerging Frameworks for Tangible User Interfaces, in Human-Computer Interaction in the New Millennium, Addison-Wesley. pp. 579 -- 601, 2001.
Saul Greenberg, Chester Fitchett. "Phidgets: easy development of physical interfaces through physical widgets," Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida
Scott R. Klemmer, Jack Li, James Lin, James A. Landay. "Papier-Mache: toolkit support for tangible input," Proceedings of the 2004 conference on Human factors in computing systems, p.399-406, April 24-29, 2004, Vienna, Austria
Lee, J.C.; Avrahami, D.; Hudson, S.E.; Forlizzi, J.; Dietz, P.H.; Leigh, D.L., "The Calder Toolkit: Wired and Wireless Components for Rapidly Prototyping Interactive Devices", Designing Interactive Systems (DIS), ISBN: 1-58113-787-7, pp. 167-175, August 2004