physical user interfaces what they are and how to build them saul greenberg university of calgary
DESCRIPTION
Physical User Interfaces What they are and how to build them Saul Greenberg University of Calgary. Physical User Interfaces. Special purpose computer-controlled devices that can be situated in a real-world setting. typically designed for particular contexts and uses. Physical User Interfaces. - PowerPoint PPT PresentationTRANSCRIPT
Physical User InterfacesWhat they are and how to build them
Saul GreenbergUniversity of Calgary
Physical User Interfaces
Special purpose computer-controlled devices that can be situated in a real-world setting.
• typically designed for particular contexts and uses.
Physical User Interfaces
Foundations for• Ubiquitous Computing• Tangible Media• Context aware computing
Styles of use• foreground interaction• ambient displays• information appliance• collaborative interaction• physical controls
Ubiquitous Computing
• A less-traveled path I call the invisible; its highest ideal is to make a computer so imbedded, so fitting, so natural, that we use it without even thinking about it.
• Provide hundreds of wireless computing devices per person per office, of all scales … It is invisible, everywhere computing that does not live on a personal device of any sort, but is in the woodwork everywhere.
Source: Mark Weiser’s UbiqCom web site
Mark WeiserXerox PARC
Ubiquitous Computing
Invisible Everywhere Computing
• invisible: o tiny, embedded, attachable…
• everywhere: o wireless, dynamically configurable,
remote access, adapting…
• aka Pervasive Computing
Source: Mark Weiser’s UbiqCom web site
Mark WeiserXerox PARC
Ubicomp is Situated Computing
Makes use of simple shared context
• space • time• proximity• affordances
Participation in the context• is physical• is out here with us• is in many small and large
places, including trivial ones
Source: Mark Weiser’s UbiqCom web site
Ubicomp Technology Trends
Source: Mark Weiser’s UbiqCom web site
Ubicomp Technology Trends
Processors: • cheap, small, dedicated, microprocessors
Low Power• small batteries, solar (?)
Wireless• 802.11, Bluetooth, infrared, mobile telephony, …
Displays • very small (inches) to very large (walls)
Peripherals• sensors, actuators, motors, …
Run-time systems• Linux on a chip, Windows CE, downloadable executables…
Source: Mark Weiser’s UbiqCom web site
ParcTab
Mobile hardware • infrared• room-sized cells• location information
A portable GUI device• small case with belt clip, ½ size of PDAs• touch sensitive 128x64 pixels display • 3 finger-operated mechanical buttons (chorded)• piezo-electric speaker• low power needs (~ 1 week between charges )• can be used in either hand by rotating display
Source: Mark Weiser’s UbiqCom web site
Tangible User Interfaces
From ‘painted bits’ to ‘tangible bits’• genre change• give physical form to digital information• physical objects, surfaces, and spaces become
tangible embodiments of digital information.
seamlessly couple the dual worlds of bits and atoms • input: grasp and manipulate• output: change of physical properties of object
Source: Tangible Media Group web site
Hiroshi IshiiMIT
Context-Aware Computing
Context as information• … characterizes a situation of a person, place or object relevant
to the interaction between a user and an applicationo location o identity o state and activites of people, groups o state of computational and physical objects
Context-aware computing• uses contextual information to
o selectively present information and serviceso automatically execute a serviceo attach context information for later retrieval
Source: Dey, Abowd and Salber, HCI Journal 2001
Styles of use
Foreground interactions
Conscious intentional interactions• graspable objects • augmented surfaces• link between physical and virtual actions
• exploit human senses of touch and kinesthesia.
MusicalBottles - controlling sound, light
InteractionOne opens and closes bottles to control digital contents e.g., opening it tells a story
Music bottlesmovement and uncorking of the bottles controls the different sound tracks and the patterns of colored light rear-projected onto the table’s translucent surface.
Source: Tangible Media Group web site
MusicalBottles - controlling sound, light
Ishii, MITSource: Tangible Media Group web site
IAMASCOPE
Wooden Mirror - wooden pixels, image
Ishii, MITDaniel Rozin, NYU
Wooden Mirror - wooden pixels, image
Ishii, MIT Daniel Rozin, NYU
Triangles – building a digital story
When triangles connect together, they trigger digital events.
These events influence the progress of a non-linear story, or allow users to organize media elements in order to create their own story space.
Extracted from Tangible Media Group web site
Triangles – building a digital story
Extracted from Tangible Media Group web site
Ambient displays
Background information • communicate digitally-mediated senses of activity and presence
at the periphery of human awareness. • ambient light sound, airflow, water movement, object motion… • peripheral displays
Extracted from Mark Weiser’s UbiqCom web site
Extracted from Mark Weiser’s UbiqCom web site
Pinwheels - ambient digital wind
Extracted from Tangible Media Group web site
Pinwheels - ambient digital wind
Extracted from Tangible Media Group web site
Information Perculator
bubbles of digital patterns
Heiner, Hudson & Tanaka
Information Perculator
bubbles of digital patterns
Heiner, Hudson & Tanaka
Ambient Light Display
light reflection from water onto ceiling
Ambient Room
ACM CHI’98, Tangible Media Group
Personal Ambient Display
Small, physical devices worn to display information to a person in a subtle, persistent, and private manner.
Ambient information is displayed solely through tactile modalities such as heating and cooling, movement and vibration, and change of shape.
Extracted from Tangible Media Group web site
Physical controls
Physically-mediated computer controlled interactivity
Marble Answering Machine
Incoming voice messages are physically instantiated as marbles.
The user can grasp the message (marble) and drop it into an indentation in the machine to play the message.
The user can also place the marble onto an augmented telephone, thus dialing the caller automatically.
Durrell Bishop
TouchCounters
Computational tags track the usage of physical objects.
TouchCounters sense activity through magnetic, acceleration, and infrared sensors, and indicate their status on bright LED displays.
TouchCounters can be networked to a web server that generates use histograms for each object.
Extracted from Tangible Media Group web site
TouchCounters
Extracted from Tangible Media Group web site
Tagged Objects
ACM CHI’99, Xerox Parc
mediaBlocks
Extracted from Tangible Media Group web site
SenseTable
A system for tracking the positions and states of multiple objects wirelessly on a flat surface.
Objects can be equipped with various controls -- dials or buttons -- which can be monitored in real-time.
When coupled with a projector, the system can display information about the objects on or near the objects themselves.
Extracted from Tangible Media Group web site
HandScape
A vectorizing digital tape measure for digitizing field measurements, and visualizing the volume of the resulting vectors with computer graphics.
Using embedded orientation-sensing hardware, it captures relevant vectors on each linear measurements and transmits this data wirelessly to a remote computer in real-time.
Extracted from Tangible Media Group web site
Manipulative User Interfaces
Customizable Physical Interfaces
ACM UIST 2002, Greenberg & Boyle
Collaborative interactions
Physical objects connecting collaborators• collaborative interaction• awareness
Adapted from Tangible Media Group web site
PingPong Plus
features a "reactive table" that incorporates sensing, sound, and projection technologies.
Projectors display patterns of light and shadow on the table; bouncing balls leave images of rippling water; and the rhythm of play drives accompanying music and visuals.
Extracted from Tangible Media Group web site
PingPong Plus
Extracted from Tangible Media Group web site
InTouch – collaborative haptics
Force-feedback technology is employed to create the illusion that people, separated by distance, are interacting with a shared physical object.
When one of the rollers is rotated, the corresponding roller on the other distant object rotates in the same way. Two people separated by distance can then play…
Extracted from Tangible Media Group web site
InTouch – collaborative haptics
Extracted from Tangible Media Group web site
Bench
…two cold steel benches located in different cities.
When a person sits on one of these benches, a corresponding position on the other bench warms, and a bi-directional sound channel is opened.
At the other location, after feeling the bench for "body heat," another person can decide to make contact by sitting near the warmth.
Initially the sound channel is distorted, but as the second party lingers, the audio channel clears.
--summarized by Ishii and Ullmer
Anthony Dunne and Fiona Raby at the RCA
Physical but Digital Surrogates
Rotating figurine servo motor
Tippable figurine light sensors
Proximity detector ultrasonic sensor
Hydra unitvideo, camera, speakers, microphone
Physical but Digital Surrogates
Privacy preserving media space
Roomware i-land
Computer-augmented room elements• like doors, walls, furniture (e.g. tables and chairs) with integrated information
and communication technology.
From the GMD Darmstadt web site on I-Land
Roomware i-land
Dynawall
From the GMD Darmstadt web site on I-Land
Roomware i-land
CommChair
From the GMD Darmstadt web site on I-Land
Roomware i-land
ConnecTableBy moving multiple
ConnecTables together, they can be arranged to form a large display area. Integrated sensors measure the distance between the ConnecTables and initiate the automatic coupling of the displays
From the GMD Darmstadt web site on I-Land
Roomware i-land
InteracTable•touch-sensitive plasma-display (PDP) is integrated into the table top•Border for leaning…
From the GMD Darmstadt web site on I-Land
Roomware i-land
Phillips – Intelligent Information Surfaces
From the Philips Lime Video CD
Tokens
Phillips – Intelligent Information Surfaces
From the Philips Lime Video CD
Designing out of the box
The problem:• programming / designing with physical devices is hard
o circuit design (electrical engineering)o microprocessor interface to digital/analog deviceso ‘wire’ interface (serial, USB, wireless, IR…)o wire protocolo connection/disconnection/intermittent connectivityo software to use deviceso maintenance and extensibility
• simple things take a long time to do• most people don’t bother
Solution 1: Interdisciplinary team
Works, but…• Still takes time• When one of the team
leaves, knowledge islost
• systems hard tomaintain &extend
Hydra unitvideo, camera, speakers, microphone
Rotating figurine servo motor
Tippable figurine light sensors
Proximity detector ultrasonic sensor
Solution 2: Hack existing devices
Programmable Embodied Agents (Kaminsky et al)
• hacked Microsoft Actimates
Solution 2: Hack existing devices
Programmable Embodied Agents (Kaminsky et al)
• Arm position -> quantity• Squeezing hand/leg -> counting• Movement-> task progress
Proxy for other person squeeze hand, other’s hand goes up)
Event monitoring Signal document is printing, then complete Barney Email biff
Solution 3: Phidgets
Physical Widgets• simple, easy to program devices and component-based software
with well-defined API• building blocks for physical interfaces
• analogous to GUI widgets
Phidget Examples
PhidgetServo:• Control 1 or 2 servo motors
PhidgetPowerbar• Control power state of outlets on a power bar
PhidgetInterfaceKit• 8 simple input and outputs plus 2 sensors• A constructor kit
PhidgetProximitySensor• Returns how close something is to it
PhidgetMotionDetector• Periodically returns the amount of motion in a space
Related areas
Mobile ComputingAugmented RealityContext-aware computingReactive EnvironmentsUbiquitous MediaCooperative Buildings
Digital Inputs - switches
Rocker Toggle Push button Push-Pull
Rotary Slide Tactile Keylock
Analog inputs: Environmental Sensors
Distance
Light MotionPressureTemperature
Voltage Weight
rangefinder
Distance
proximity
Analog inputs: Input Sensors
Force Mini-joystick
Potentiometers
multi-turnsingle-turn
Capacitive
encoder
Accelerometer
slider
Analog inputs: Input Sensors
Bend
Force
Tilt
Camera
Custom Input: Identification
RFID Tags and Antenna Bar Code Scanner
Custom inputs
Camera
Digital Outputs – low power
Lights
LEDslamps
Relays Solenoids
Output: Motors
Position: 0-180o Servo
Rotate by steps: +/- xo
Stepper
SpeedDC Motor
Output: Displays
Text LCD
Numeric-alpha
Graphics (not yet)
Approaches: PIC Micro-controller
Single programmable chip computer with:• CPU, RAM, ROM, I/O, serial/parallel ports, A/D and D/A converters
Need to know:• basic circuit design• basic electronics
o resistor, capacitor, diodes, transistors…
• low level programming• PIC hardware details• …
Powerful, flexible, but• learning curve• excessive time in low level details• complex things are hard• serial
e.g. www.voti.nl/swp
Approaches: Basic Stamp
Pre-built boards• Pic microcontroler• pre-wired circuits and connectors• boards designed for different uses
Still need to know • electronic components • electronic circuitry • PBasic language: stamp-specific
instruction set
Still flexible, but• learning curve still there• time in low level details• complex things are hard• serial
Approaches: HW as SW components
Phidgets and Making Things• dedicated devices• some plug and play electronics• under direct computer control• well-defined component-based software
o Interface via APIs, objects, and/or widgets
Need to know • conceptual knowledge of device• high level programming language• software component documentation
High design flexibility, low electronic flexibility• very low learning curve• design by combining and varying• time in conceptual design, not electronics
Approaches: Programmable Brick
Lego Mindstorms• robotics (downloadable code)• computer control as well• proprietary RCX microcontroller• infrared communication• 3 input ports
o light/touch sensors…• 3 output ports
o motors, lights…• Lego building blocks• children’s programming language for downloading but
o well-defined SDKo 3rd party access from standard languages
Need to know • SDK / language
High design potential for robotics• limited input/output (3+3)• expensive for basic set, plus add-ons
e.g. http://www.crynwr.com/lego-robotics/
www.mastincrosbie.com/mark/lego/images/grabberarm1.jpg
e.g. http://www.crynwr.com/lego-robotics/