programming for interactivity ixd2 2012 - interaction design ume¥
TRANSCRIPT
Thomas Lövgren
Interactive Developer, designer & programmer
Umeå Institute of Design, 2012-10-15
Programming for Interactivity
Terminology, techniques & design/interaction-concepts
(self-study)
Self-study outline
In this self-study the student will have a chance to study the following topics:
• Touch-input categories
• TUIO
• Touch technologies
• Resistive and capacitive screens
• Multi-touch ”All point screens” (The Future of User Interface Design)
• Gestures & Recognition
• Adobe Flash CS5/CS5.5 & AIR Multi-touch API
• GestureWorks
• Links & Resources
Next lecture will be:
ActionScript 3.0 Multi-touch pt.2
Designing, coding & publishing for a touch-device
Introduction to Multi-touch pt. 1 with ActionScript 3: Terminology & techniques
Multi-touch
(Technique 1)
(Technique 2)
(Technique 3)
What is Multi-touch?
Multi-touch is an enhancement to touch-screen technology, which provides the user the
ability to apply multiple finger-gestures simultaneously onto an electronic visual-display to
send complex commands to the device
Multi-touch
Touch Event: A term used to describe when a system knows that an object has touched the touch
device
Single Touch: Single Touch occurs when a finger or stylus creates a touch event on a touch-
device so it is detected by the touch controller and the application can determine the X,Y
coordinates of the event
Multi-touch: An interactive technique that allows single or multiple users to control graphical
displays with more than one simultaneous finger (+2 fingers)
Gesture: A physical movement that can be sensed, and often an action assigned to it. Some
common gestures are single finger panning, swipe, and two finger zoom-pinching
Gesture-enhanced single-touch: Also known as "Dual Control", "Gesture Touch" and often
"Dual-Touch" describes the ability of a touchscreen to register certain two-finger gestures, even
though the display hardware does not have full Dual-Touch capabilities
Touch-input categories
Multi-touch
TUIO (Tangible User Interface Objects) is an open framework that defines a common protocol
and API for tangible multi-touch surfaces
The protocol encodes control-data (position, size, and relative velocity of blobs) from a tracker
application and sends it to a client application
TUIO is based on Open Sound Control: An emerging standard for interactive environments
The Flash ActionScript 3 TUIO library supports true Multi-touch (TUIO client)
• Positive: Can be used for large custom made Multi-touch installations
• Negative: Requires in general a lot of ’tweaking’ to get working properly
TUIO: overview
http://www.tuio.org
Multi-touch
Technique 1
Bending Wave
Infrared (Grid / Cameras) (IR)
Optical (Camera)
Projected Capacitive (Matrix)
Resistive
Surface Acoustic Wave (SAW)
Surface Capacitive
Touch technologies
http://solutions.3m.com/wps/portal/3M/en_US/TouchTopics/Home/Technologies/
Multi-touch
Resistive touch-screens are composed of two flexible sheets coated with a resistive
material and separated by an air gap or microdots
When contact is made to the surface of the touch screen, the two sheets are pressed
together, registering the precise location of the touch
Advantages
• Most widely used touch technology
• Can be activated by bare finger, gloved hand, or stylus
• Low cost
Disadvantages
• Top sheet is highly susceptible to scratches, cuts and cigarette burns
• Transmission typically in 80% to 85%
Touch functionality
• Single touch
Examples
• Nokia 5800 and Samsung Omnia
Resistive screens
Multi-touch
In Capacitive screens a small voltage is applied to the four corners of the screen
The human body also conducts electricity so when a person touches the screen with their
finger, there is a change in the capacitance of the screen where the person touched it
The computer then registers the exact location of the persons touch
Advantages
• Very light finger touch required for activation
• Technology with fastest touch response time
• Transmission typically 88% to 92%
Disadvantages
• Severe scratch can affect operation within the damaged area
• Supports only finger or tethered pen
• Cannot be used in all weather scenarios
Touch functionality
• Multi-touch
Examples
• Apple iPhone, Samsung Galaxy S, Tablets etc
Capacitive screens (Surface Capacitive Touchscreens)
Multi-touch
The iPhone touch-sensitive screen includes a layer of capacitive material
The iPhone's capacitors are arranged according to a coordinate system
Every point on the grid generates its own signal when touched and relays that signal to the
iPhone's processor
This allows the phone to determine the location and movement of simultaneous touches in
multiple locations
The processor uses software to analyze the raw data and determine the features of each
touch (size, shape, location…)
How the iPhone works (Capacitive touch-screen)
http://electronics.howstuffworks.com/iphone.htm
Multi-touch
Single-touch touch-screens, which optimize design space and
provide unlimited numbers of buttons, still only allow users
to select one function at a time
With multi-touch gesture touch-screens, users can perform
more complicated inputs like sizing photos, rotating and
adjusting web pages
Multi-touch all-point touch-screens tear down the boundaries
of user input, which is only limited by the designer’s
imagination (design your own complex gestures, 10 fingers
etc)
Multi-Touch All-Point Touch-screens: The Future of User Interface Design
http://mobiledevdesign.com/tutorials/multitouch-touchscreens-0723/
Multi-touch
Gestures are the synthesis of multiple touch events into a single event
Gesture recognition is a topic in computer science and language technology with the goal
of interpreting human gestures via mathematical algorithms
Gesture recognition can be seen as a way for computers to begin to understand human
body language (humans to interface with the machine)
Basic gesture recognition can be done in three steps:
• Capture user input
• Homogenize and normalize input
• Match to a predefined pattern
The Touch Gesture Reference Guide is a unique set of resources for software designers
and developers working on touch-based user interfaces
Gestures: Gestures & Recognition
http://blog.sqrtof5.com/?p=173 http://www.lukew.com/ff/entry.asp?1071
Multi-touch
The latest Flash Multi-touch API supports Single-touch, Multi-touch and Gestures on various
platforms, browsers and devices (Windows 7, Flash Player +10.1, AIR +2.0, Android & iOS etc)
The technique can handle as many touch points as the touch-device and underlying operating
system supports
All Touch-inputs are very similar to the standard AS3 Mouse/keyboard - events, which makes
it easy and intuitive to work with
The AS3 classes that handles Touch-inputs are:
TouchEvent, TransformGestureEvent, GestureEvent, PressAndTapGestureEvent, GesturePhase
Supported Gestures on various platforms: PAN, ROTATE, ZOOM, TWO_FINGER_TAP, GESTURE_PRESS_AND_TAP
The Adobe AIR technology also support Multi-touch for
publishing on: Desktop, Android, iOS and TV
Note! In this course we are going to focus on
Flash/Air for Android (See Multi-touch pt2 for more info)
Adobe Flash CS5/CS5.5 & AIR Multi-touch API: Overview
http://www.adobe.com/devnet/flash/articles/multitouch_gestures.html
Multi-touch
Technique 2
GestureWorks is the most advanced solution for creating true multi-touch applications with
Adobe Flash, AIR and Flex (AS3 programming)
Open source framework that provides a simplified and standardized approach to multi-
touch application development
Supports over 200 gestures, multiple & simultaneous gestures and ”custom gestures”
Multi-touch authoring (Flash and Mac) solution for Windows 7-compatible multi-touch
devices (Android and iOS)
Multi-touch simulation functionality
GestureWorks is developed by IDEUM
GestureWorks: Overview
http://www.gestureworks.com
Multi-touch
Technique 3
NUI group (Natural User Interface group)
http://nuigroup.com
Multi-touch terminology
http://wiki.nuigroup.com/Multi-Touch_Terminology
Adobe Flash CS5/5.5 & AIR Multi-touch API
http://www.adobe.com/devnet/flash/articles/multitouch_gestures.html
GestureWorks (Flash & AIR MT with AS3)
www.gestureworks.com
DreamTouch
http://www.dreamtouchblog.com/
Microsoft Surface
http://www.microsoft.com/surface/en/us/default.aspx
Designing for Multi-Touch, Multi-User and Gesture-Based Systems
http://drdobbs.com/cpp/216402697?pgno=1
IDEUM: Building a High-Resolution Multi-touch Wall
http://www.ideum.com/blog/2010/11/building-a-high-resolution-multitouch-wall-part-1/
Links & Resources
Multi-touch
NUITEQ (Snowflake - suite)
www.nuiteq.com/
TUIO
www.tuio.org