James Haliburton, Concept Lead

Dynamic UIs

CONTENTS

•Today & Tomorrow’s Mobile

•Future Visions

•Emerging Tech

•Dynamic UIs• Legacy of Mobile• What’s in a phone today• Patterns for Achieving Dynamic UIs• Transitions are the new state• Design tools & processes

•Inspirational Seminars

What is a Dynamic UI?

A dynamic UI experience is about valuing the in-the-moment use of the mobile phone, and not only the product’s overall usefulness. A dynamic UI creates a pleasurable experience.

What is a Dynamic UI?

4 Principles•Provide apt feedback•Invite playful, non-task based behavior•Respond to “incorrect” interaction•Design forms of interaction which support natural behavior

Philosophy about the future

At TAT we believe that it is necessary to examine and innovate the future of technology. We are examining strong use cases which justify and direct the need for this development. We are designing for these scenarios now, as the

future is arriving quickly.

TODAY’S MOBILE

TODAY’S MOBILE

•Capacitive touchscreens•Multitouch•GPS•Bluetooth•Wifi

•Accelerometers•Compasses•High res•More processing power than it took to put a man on the moon!

TOMORROW’S MOBILE

Augmented reality

projected UI:s

GyrosGestures3D

Augmented reality

projected UI:s

GyrosGestures3D

UI/UX in mobile moving forward

Changed Input MethodsWe have recently moved from phones like the Sony Ericsson T610 with indirect input (keys) and a color graphical output to the time of the iPhone. Changes in the UI are partially enabled by the change from indirect input to direct input (touch).

But do our designs actually take into account that we have constructed new types of devices?

FUTURE VISIONS

FUTURE VISIONS

FUTURE VISIONS

EMERGING TECH

PROJECTOR PHONES

PROJECTOR PHONES: FUTURE

PROJECTOR PHONES

A Radical Shift•Not merely a matter of scaling up existing UIs and mobile content

•Offers a paradigm shift in the way we interact

•Phones have a social presence in our lives more than other technology

•They are not fixed to a location or specific task - they aim to be flexible and align to our behavior and to enhance our experiences

•Our behavior and experiences are more often than not, situated amongst other people

•Private device in a public setting causes a disparity and disconnect

•The key difference with procam enabled UIs will be that we can begin to truly interact, work, and play together. We will not be forced to remove ourselves from a conversation to check a mapping application or record a phone number - these can be collaborative actions.

•The UI can become a conversational tool.

FLEXIBLE DISPLAYS

18

A multimodal user interface is a UI that involves more than one modality, or human sense

A multimodal UI is thus a UI that provides feedback to several human senses, e.g. visual, auditory and haptic (tactile) feedback

MULTIMODAL

19

Sounds can be added to augment the visual design, adding a new dimension to user immersion

Compared to haptics, sounds can convey information efficiently, e.g. through sound effects, music or even speech

Globe Sejf Trum

MULTIMODAL

20

Sound can help users to better understand the effects of their interactions, especially when using touchscreens

Visual design themes can have corresponding sound themes, enhancing the metaphor of the GUI

Water Fire Paper

MULTIMODAL

21

As proven by film and computer games, sounds can be very expressive

Sounds should be subtle and varied, to avoid annoying people around the person using the device

Example: Asia cute UI with sound

MULTIMODAL

22

MULTIMODAL: FUTURE

23

A UI ECOSYSTEM

Mobile UIs are no longer just about button input and visual outputNew input paradigms require new approaches to output, for example:

Touchscreen input -> usability need for haptic and/or sound feedbackGesture input (via touchscreen or motion sensor) -> expectation of visual output that

follow gestures realistically, e.g. through physics modeling3D aesthetics and realism -> increasing expectations that UIs should mimic the real

world, e.g. by including sound and haptics

In a well designed device, output is closely linked and matched to user inputChanging one part of a UI will affect other aspects

MULTIMODAL

AUGMENTED REALITY

What is a gesture?•Essentially any movement can be a gesture•Rich and subtle•Occurs over space and time•Until recently we’ve been limited mostly to 1-dimensional button pressing•But touch brings in new possibilities

GESTURES

A model for understanding gestures•Symbolic•Iconic•Metaphoric

Symbolic•3x3 Grid•Must learn what the gesture means•“automagical”•Gesture is symbolic of the action occuring

GESTURES

A model for understanding gestures•Symbolic•Iconic•Metaphoric

Iconic•Uses “real world” gestures matched to actions on the wcreen•Flipping like you do your CD collection•Motion matches something we already know how to do

GESTURES

A model for understanding gestures•Symbolic•Iconic•Metaphoric

Metaphoric•Uses a familiar metaphor•Doesn’t necessarily match real world interaction•Needs support of sound, and visuals

GESTURES

A model for understanding gestures

GESTURES

SYMBOLIC ICONIC METAPHORIC

•Pro-user•magic quality•good for supporting surprise•can be seperate from the graphic UI•‘But supported by feedback in other UI elements•Can be difficult to remember•Gesture is its own layer

•Instantly understood•slower to perform than arbitrary gestures•context dependent•highly satisfying•can find inspiration in industrial design affordances•gesture is the meaning

•easily remembered•dependent on other UI elements•can be used to discover new functionality•gesture is enabler

3D & PHYSICS

3D & PHYSICS

3D & PHYSICS

Flexible Information Visualization (FIV) Describes the elements of 3D which allow a UI to better display large amounts of data, like playlists, in interesting ways relevant to the user and to the application being used.

Naturalized Interaction (NI) Describes the elements of a 3D UI which minimize how much a user needs to think about how theyʼre using an application, and allows them to simply use it.

Visual Style and Feedback (VSF) Describes how 3D can add a “wow effect” to a product and better differentiate itself from competitors.

Declutter

Optimize

Focus

Zoom & OverviewFIVRealtionships

Reduce Cognitive Load

Flatten Learning Curve

Show Options

Show AffordancesNIWOW - Effect

Visual ImmersionFeedback

Holistic & Seamless

VSF

3D & PHYSICS

3D & PHYSICS

QUALITIES FROM THIS NEW TECHNOLOGY

Integrating our devices with the real world•We can use our devices as social tools•They can be flexible to our needs•Encourage fun and playfulness•Designs can be rich and creative

TODAY’S MOBILE

•Capacitive touchscreens•Multitouch•GPS•Bluetooth•Wifi

•Accelerometers•Compasses•High res•More processing power than it took to put a man on the moon!

WHAT’S THE RESULT OF ALL THIS TECH?

A nicer box with a longer feature list.Minimal UI innovation.

WHAT’S THE RESULT OF ALL THIS TECH?

What about the UI & UX?

UIs are still based around static screen states.

This is legacy of WIMP (windows, icons, menus, pointing device), but we’ve been talking about post-WIMP for 15 years.

This makes sense with simple task based operations. Screen states support consistency and simple flows.

WHAT’S THE RESULT OF ALL THIS TECH?

PC Mobile Evolution So Far

Windows Static screens

Icons Icons, Widgets

Menus Menus in soft keys

Pointing Device (mouse, trackpad)

D-Pad, Trackball, Touch pointing

WHAT’S THE RESULT OF ALL THIS TECH?

Input innovation has been big (qwerty, touch, accelerometer, predictive text)

•But output and presentation have been mostly "cosmetics" (if that)

•iPhone did a big jump in creating a more natural interface (Realtime, direct manipulation)

•There are more steps in making devices less like machines - interface with real world feel (dynamic)

DYNAMIC UIs - Reacts to real world behaviour

DYNAMIC UIs - Persistency of elements

DYNAMIC UIs - Soft Values, Generative visuals

DYNAMIC UIs - Movable light sources

DYNAMIC UIs - Adaptive Elements

DYNAMIC UIs - Situated Change

Sony Ericsson S500i

Theme changes appearance

depending on time of day, date and special events.

Appearance changes based on the time of day and special events.

DYNAMIC UIs - Personalized Space

DYNAMIC UIs - Personalized Behavior

50

Water Fire Paper

DYNAMIC UIs - Multimodal Design

Dynamic sound can help user to better understand the effects of their interactions, especially when using touchscreens.

DYNAMIC UIs - Analogue Manipulation, Physics, Transition Free

DYNAMIC UIs

TRANSITIONS ARE THE NEW STATE

Dynamic UIs support real behaviour.

This makes sense when you use technology that connects your device to the real world.

UI Paradigm Transition Type User Reaction

Static Screen No Transition Where am I?

Static Screen Full screen transition Something happened, I’m somewhere else

Static Screen Smart full screen transition I understand where I’ve arrived

Dynamic UI Link flow transition I can do some new things now

Dynamic UI Transformation of UI elements I can repurpose these UI elements for what I want

Dynamic UI User controlled space I’m casually browsing, but have control to do what I like

DYNAMIC UIs

TRANSITIONS ARE THE NEW STATE

Dynamic UIs support real behaviour.

This makes sense when you use technology that connects your device to the real world.

DYNAMIC UIs

Dynamic UI Qualities Static State Based UI Qualities

Adaptive Fixed

User expectations are different PC based expectation

Metaphoric, natural behavior Abstract behavior

Empathetic Unforgiving

Creates dialogue with user Directing

Explorative Functional

Playful Task based

Holistic Fragmented

Can communicate soft values Hard values

DYNAMIC UIs

Dynamic UI Patterns

Persistency of UI Elements

Generative Visuals

Apt Reactions to Physical Input

Movable Light Sources

Contextual Change

Personalization of Space

Adaptive UI Elements

Synchronized Multimodal Design

Analogue Manipulation Patterns

Free Moving Elements With Physics Models

DYNAMIC UIs

Design tools & processes

Clear mental / behavioural models•The device has a persona•Range of behaviour and personality is the consistency which pulls a dynamic UI together - aligns to situations•Can derive both design patterns & tech requirements

Agile Work Process•All stakeholders involved•Horizontal, not vertical process•Rapid iterations•Important to sketch on device!

Concept Sketching

v2v2

v1

•wireframes•functional specifications•paper prototypes•flash movies•UI flows•etc.

Typical design production and documentation has difficulty communicating the qualities of a dynamic user experience.

The tools we use for sketching also leave us short as designers in actually testing the experience of what the UI will be like.

We should be able to experience and communicate the rich behavior and interactions of our UIs.

Spatial metaphors can be used to reduce cognitive load since you can work with familiar principles.

Using a metaphor can make it easier to achieve consistency throughout the UI.

It is the first step in creating a consistent behavioral model.

Communicating 3D or behvioral UIs within teams can be troublesome, to have a spatial metaphor can make discussions more direct.

A shape alone is not a metaphor.

Sketching of 3D UIs using spatial metaphors

However, don’t go overboard! It’s still a UI and you can break the rules!

Communicating camera angles in 3D UIs

Concept Sketching

Concept Sketching

Shape vs. Metaphor

•Metaphors have behaviour

•You can put your shape in an environment

•Simple is good

•Spatial metaphors are all about being consistent

•Like personas for interaction behaviour

•You are testing how the UI behaves and feels

Concept Sketching

Traditional work process

Planned

Traditional work process

Actual

Iterative work process with Cascades

DYNAMIC UIs

Patterns & Tools

3D, Touch, GesturesPhysicsGenerative movement & animationMultimodal designClear mental models, Agile work processes

A dynamic UI experience is about valuing the in-the-moment use of the mobile phone, and not only the product’s overall usefulness. A dynamic UI creates a pleasurable experience.

DYNAMIC UIs

There is no best practice.

Our UI design patterns will come from practices of physical computing, industrial design, and good field research - not the desktop PC.

Patterns & Tools

3D, Touch, GesturesPhysicsGenerative movement & animationMultimodal designClear mental models, Agile work processes

Inspirational Seminars

•3D•Android•Multimodal•Advanced Displays•Physics•Sketching & Prototyping

Creating Real Customer Value With 3D

This seminar explores the use of 3D graphics in UIs on mobile devices

• Inspirational examples• Framework for analyzing and working with 3D• What’s the real customer value of 3D?• Provide a 3D experience from low-end to high-end devices• A 3D roadmap to the future• TAT designs & concepts for you

Typically the palette of "3D techniques" has been comprised of lighting, shadows, focus, depth, camera angles and similar visual elements. At TAT, we believe that 3D is more than 3D graphics.

We understand "3D" in user interfaces as a paradigm where the appearance of the interface displays three-dimensional or real-world qualities, while the technology behind the visualization is not necessarily built up as a real 3 dimensional simulation. To benefit from the full potential of 3D in mobile user interfaces we also need to consider the total user experience.

Android Seminar & Workshop

TAT presents an inspiring look at the options with Android today and tomorrow.

• Inspirational examples for you• Differentiating your UI on Android• Configurations for 3D, hardware acceleration, silo apps, more• Suggested roll-out and strategy for this new platform• How does the Cascades UI Framework enhance Android?• How to get better performance and enhance hardware efficiency• Hands-on demonstration of Android development

The new kid on the block is making waves, but how do we harness all that Android has to offer? What is Android? What can it do for you today? What can it do for you tomorrow?

By providing the interaction design and the visual look & feel of the G1, TAT has been part of Android right from the beginning. Already we have deep experience and insight into both the technical and UX opportunities available and can help you push the limits of 3D, unique transitions, work processes, prototyping, and more.

Mixing pragmatism and inspiration TAT will help you address key issues for your organization with Android.

Multimodal UIs - User Experience and WOW

An exploration of a cohesive use of sound, haptics, & animation

• Inspirational examples• How can sound and haptics enhance UX?• A taxonomy for multimodal UIs• Usability factors & aesthetic concerns• Synchronizing animations with sound and haptics• TAT designs & concepts for you

Like films and computer games, the mobile UI can become more engaging and immersive by including sounds and haptics.

Sound and haptics can improve usability on touchscreen devices where visual feedback is not always sufficient.

To achieve a true WOW factor, sound, haptics, and visuals must all be synchronized together.

ProCams and the Future of Mobile Displays

Novel displays and the potentials of ProCams (projectors combined with cameras)

• Inspirational examples• Overview of emerging display technologies• Opportunities and challenges of new displays• Different aspects of interaction with projected UIs• New interaction paradigms of flexible and transparent displays• TAT designs & concepts for you

Projected graphics will enable mobiles to become far more social than today. Collaboration can be enabled by using the projection as input surface

Since pressing buttons will interfere with projections, other ways of interaction, such as camera captured gestures or speech input, become necessary

Flexible and transparent displays open up new interaction possibilities, such as “bend to zoom”

Realism and Physics

New levels of realism in UIs enabled by physics modeling

• Inspirational examples• How can physics and realism enhance UX?• To what parts of a UI can physics be applied?• How to design realistic output that corresponds to input gestures• Benefits and drawbacks of realism• Simulating physics on low end and high end devices• TAT designs & concepts for you

Mobiles with touchscreen or motion sensor input can become more intuitive and impressive if graphics behave realistcally, modeling physics.

UIs that model physics are responsive to subtle gestures, which makes them inviting and playful.

When used right, realistic UIs can both enhance usability and add a considerable WOW-factor.

Thinking Beyond Flash Prototyping

How to capture creativity and design strong products through efficient prototyping

• Inspirational workflow examples• What is a sketch? What is a prototype?• An overview of prototyping tools and methods• Communicating ideas through a multitude of sketches• Useful skills for efficient prototyping• TAT process and tools for prototyping

Introducing new prototyping methods can open up new opportunities byincreasing product differentiation

At TAT we believe that many important aspects of new products and services can be designed and evaluated before entering a resource-heavy prototyping phase.

By introducing practices for improved sketching at your company, a more creative and flexible prototyping culture will emerge.

TAT believes that artefacts radically improve communication and that all aspects of products, services, processes and business models can be visualized.

THANK YOU!