toolkits for physical objects (tuis)

1

Toolkits for Physical Objects (TUIs)

Jack Li

Advanced User Interface Software

November 30

Fall 2004

2

Outline

Motivation for TUIs

Input Technologies

TUI Toolkits

3

Current Limitations of UIs Desktops and laptops suck in our entire

attention As a result we live in two worlds: the real

physical world and the virtual cyber world Constantly “wired” in order to be in both

worlds Loss of rich interactions found in

everyday things (particularly haptic)

MOTIVATION FOR TUIS

4

Motivation

Driven by ubiquitous computing Both people and technology moving away

from the desktop into the environment Input: different ways of interacting Output: different ways of displaying

MOTIVATION FOR TUIS

5

Tangible User Interfaces TUIs augment the physical world by integrating

digital information with everyday physical objects

Goal: to bridge the gap between bits and atoms 1. Interactive surfaces 2. Coupling information with physical objects 3. Ambient media for background awareness

MOTIVATION FOR TUIS

6

Outline

Motivation for TUIs

Input Technologies

TUI Toolkits

7

Physical Objects as Input

Pretty much anything … walls, papers, utensils, clothes hangers!

Inspiration sci-fi novels, own imagination everyday life, current practices

INPUT TECHNOLOGIES

8

Devices Cameras Electronic tags (RFID technology) Sensors

Light Motion Heat Force

Motors Orbs

INPUT TECHNOLOGIES

9

Framework for the GUI

(a)model-view-controller

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

INPUT TECHNOLOGIES

10

Framework for the TUI

(b)MCRpd: model-control-representation(physical and digital)

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

INPUT TECHNOLOGIES

11

Outline

Motivation for TUIs

Input Technologies

TUI Toolkits Phidgets (2001) Calder (2004) Papier-Mâché (2004)

12

Toolkit Goals

Same as when GUIs were built Make easy for non-specialized experts to

create (should not need a vision or hardware guru on the team)

Hide dirty implementation details Abstract proper functionality with well-

defined API

13

Overview

First toolkit for physical objects Reusable physical devices with proper

abstraction for creating a wide variety of applications

An API that had the right amount of abstraction

PHIDGETS

14

GUI Differences

Connection manager Way of detecting when devices go on/off

Identification Every physical object should have a virtual

counterpart Simulation

A way to test code even if phidget not present

PHIDGETS

15

ArchitecturePHIDGETS

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

16

The Physical DevicePHIDGETS

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Primitive device components Sensors Motors Switches

Circuit board CY7C63000 USB micro-controller

Communications layer USB communication standard

17

Wire ProtocolPHIDGETS

Low-level software in Windows and microcontroller for communication that’s hidden from end programmers

Commands from phidgets Unique Id numbers Device type (servo, powerbar) Specific events (tag for RFID reader)

Commands from Windows Set device state (output like servo)

QuickTime™ and aTIFF (LZW) decompressorare needed to see this picture.

18

Computer SoftwarePHIDGETS

PhidgetManager Connection manager

• Event onAttach()• Event onDetach()• Int count• Enum item

Phidget-specific devices All have:

• String deviceType• Boolean isAttached• Long serialNumber

RFID• onTag()

Servo• onPositionChanged(), motorPosition(index)

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

19

Simple ApplicationPHIDGETS

Phidget Manager

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Phidget ServoQuickTime™ and a

TIFF (LZW) decompressorare needed to see this picture.

20

Outline

Motivation for TUIs

Input Technologies

TUI Toolkits Phidgets (2001) Calder (2004) Papier-Mâché (2004)

21

Main goals

Primarily geared towards interaction/product designers

Support for more refined products Support on the hardware side

CALDER

22

Supported DevicesCALDER

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Small size is big win!

23

Fluid prototypingCALDER

Objects easily embedded!

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

24

Communication

Wired PIC16C745 C with USB interface engine Same as Phidgets (and iStuff)

Wireless Tradeoff between size and power (perhaps

Moore’s law will remedy) Small batteries, antenna, and pulse-width

modulation

CALDER

25

Software

Surrogates (GUI widgets) to manipulate physical object output

CALDER

26

Outline

Motivation for TUIs

Input Technologies

TUI Toolkits Phidgets (2001) Calder (2004) Papier-Mâché (2004)

27

Overview

Emphasis on software side side Input model that provides common

programming abstractions Focus on objects being lightweight

PAPIER-MACHE

28

Software Abstractions Phob—encapsulates information related to this

physical object All—source producer, time RFIDPhob—tag VisionPhob—mean color, bounding box, rotations

PhobListener—listens for Phobs produced phobAdded(), phobUpdated(), phobRemoved()

PhobProducer—actual devices that generates events

PhobManager—maintains PhobProducers

PAPIER-MACHE

29

A Common Event Model PhobListener is the interface among all devices Methods are the same

phobAdded(), phobUpdated(), phobRemoved() Vision

phobAdded() when new object detected, phobUpdated() when old object changed (size, color), phobRemoved() when old object disappears

RFID (different from what the paper says) phobAdded() when tag detected, no phobUpdated(),

phobRemoved() when tag disappears Facilitates easier retargetting and substitutions (allows

for more exploration and faster, prototyping)

PAPIER-MACHE

30

Marble Answering Example

1. Get the manager that keeps track of all input devices (PhobProducers)RFIDManager manager = RFIDManager.getManager();

2. Set up a map associating all objects with an audio clipAssociationFactory factory = new AudioClipFactory(); AssociationTreeTable assocMap = new

AssociationTreeTable(factory);

3. Attach this map so new objects are assigned new clips and old ones get played manager.addPhobListenerForAllProducers(assocMap);

PAPIER-MACHE

31

A Behavior: Associations

A common behavior we noticed New objects get stored/assigned, old

objects get retrieved a database of sorts An extension of PhobListener

PAPIER-MACHE

32

A Behavior: Associations

AssociationTreeTable addClassifierFactoryPair(Classifier, Factory)

ObjectClassifier isMemberOfClass(Phob phobToCompare)

AssociationFactory createAssociationEltForPhob(Phob phob)

PAPIER-MACHE

33

Monitoring WindowPAPIER-MACHE

34

Device state—connection mgrPAPIER-MACHE

35

Camera FeedbackPAPIER-MACHE

36

Association MapPAPIER-MACHE

37

ExtrasPAPIER-MACHE

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

38

Summary

39

Major Toolkit differences Phidgets

Complete, balanced, varied assortment of physical objects easy to program with

Calder Small size allows for more fluid prototyping at

the PHYSICAL level Papier-Mâché

Unified event model allows easier development and retargetting at the SOFTWARE level

40

Conclusion

TUIs seek to augment the physical world with digital information

Toolkit support allows non-experts to develop TUI application, like GUI toolkits

Successful toolkits must manage physical connections, gather/alter relevant physical state information, and provide good feedback for these added complexities

41

Future work

Input adaptation M to N mapping of input devices

Input models What other kinds of behaviors can be

abstracted Will be more clear with the development

of more applications

42

References Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits

and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States

Ullmer, B. and H. Ishii, Emerging Frameworks for Tangible User Interfaces, in Human-Computer Interaction in the New Millennium, Addison-Wesley. pp. 579 -- 601, 2001.

Saul Greenberg, Chester Fitchett. "Phidgets: easy development of physical interfaces through physical widgets," Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida

Scott R. Klemmer, Jack Li, James Lin, James A. Landay. "Papier-Mache: toolkit support for tangible input," Proceedings of the 2004 conference on Human factors in computing systems, p.399-406, April 24-29, 2004, Vienna, Austria

Lee, J.C.; Avrahami, D.; Hudson, S.E.; Forlizzi, J.; Dietz, P.H.; Leigh, D.L., "The Calder Toolkit: Wired and Wireless Components for Rapidly Prototyping Interactive Devices", Designing Interactive Systems (DIS), ISBN: 1-58113-787-7, pp. 167-175, August 2004