Designing a Moving Experience

Andrew Fisher @ajfisherBe Responsive, 10 September 2013

Hi! My name is Andrew Fisher and I’m an interaction researcher. Tonight I’m going to talk to you about how to design a moving experience for the next 25 minutes.

8 years difference

I imagine everyone has seen this picture by now. In this 8 years between popes it shows a striking technological change. However what’s interesting for me is that this 8 year period depicts a fundamental change in behaviour too.

2005

Reuters / Jerry Lampen

This pic shows it better. Look down here at what people are holding...

Cameras you need to LOOK through to see what you’re shooting.image: http://www.businessinsider.com.au/vatican-square-2005-and-2013-2013-3

2013

AP

Here, there are more cameras because there are more phones. And the one person with an iPad.

But we’ve moved from look through to look at because now EVERYTHING has a screen.

In the span of 8 years, the behaviour of hundreds of millions of people has changed irreversibly after being culturally ingrained for about a hundred years.

These pics show how behaviour is changing and is changed by new interaction types afforded by our devices.

image: http://www.businessinsider.com.au/vatican-square-2005-and-2013-2013-3

Time at human scale

Flickr (cc): Pink Sherbet Photography

So apparently the average span of a pope is about six and a half years.

As a timespan it’s nice to work with as it doesn’t feel as long as a decade. It’s the division of time we use when we grow up that covers pre school, junior school and senior school. It’s a very human time scale and it’s also long enough to see changes happen.

image (cc): http://www.flickr.com/photos/pinksherbet/179279964/

Moore’s Law

7 years also represents about a 10x increase in electronic capability according to Moore’s law.

So this time period is a very interesting one to observe for many reasons.

Not many years ago...

flickr (cc) Zitona

6-7 years ago we didn’t have iPhones or Android phones.

We barely had voice recognition at all, let alone on our phones.

The nintendo Wii was just showing us a future where waving your arms around at a machine wasn’t a sign of madness but something quite normal people would be happy to do.

But I feel we’ve gone off track. Responsive design has focussed too much on our devices and on technical solutions and too little on the behaviour that goes alongside.

image: (cc) http://www.flickr.com/photos/zitona/2098321266/

Design for humans

flickr (cc) jonrawlinson

So let’s look forward and see what the next 6 years will bring us and how we will design moving experiences in the future.

Tonight I want you to think about how responsive design goes beyond just our technology. About how we design for humans and their behaviour not the devices themselves so that experiences are more contextually aware.

To do that we are going to cover:image (cc): http://www.flickr.com/photos/london/106913032

Agenda

1. New device types2. New interaction methods3. Computing as substrate

How new device types will influence our design.

What new methods of interaction should we consider

And how computing will become embedded into our environment.

New devices

flickr (cc) x-ray delta one

Lets start with new devices coming up over the next handful of years. What will they do? How will they change our behaviour?image (cc): http://www.flickr.com/photos/x-ray_delta_one/4870722078

Glass-like

flickr (cc) guiseppe.costantino

Google is betting the future on Glass and there’s a lot of noise about this right now because it taps into an Augmented Reality meme going back at least as far as terminator. image (cc): http://www.flickr.com/photos/69730904@N03/8813585832

Niche applications

flickr (cc) loiclemeur

There will be some very good, although highly niche use cases for glass like devices. Though I can’t see them being used all the time as they are intrusive and likely to be psychologically draining.

image (cc): http://www.flickr.com/photos/loiclemeur/8699901706/

Retail applications

flickr (cc) Zemlinki!

Mach 4 X-Trainer$189.99★★★★☆

Other prices:$149.99 (Amazon)$134.56 (Shoes)

I can see good use cases in retail, where you pick up a product and shows us competitive pricing, features and customer ratings.

There’s some domain specific benefits here but they mostly already work on mobile however they give you the benefit of efficiency because you hands become more free.

In this time span there’s likely to be a lot of hype but a lot of disillusionment as well.image (cc): http://www.flickr.com/photos/zemlinki/196284587/

Microscreens

flickr (cc) Ivva!

Next up are microscreens and in particular smart watches. These are much closer to reality and there are plenty of companies playing here right now with products or prototypes.

Smart watches lay in an interesting area of ambient technology at the intersection of relatively high information density and a high degree of actionability.

This is an interaction sweet spot because you can get good information in front of the person and get attention but then also action it whether on the watch or through an immediate secondary device like a phone or a computer.

image (cc): http://www.flickr.com/photos/ivva/7157969502/

Possible contender?

(c) Samsung

In my view Samsung’s prototype shown last week misses the mark here because they’ve made it a not-quite phone and it’s big and ugly. A smart watch doesn’t need to be massive because it’s not a primary interface. It doesn’t need all the features of the phone.

The winner in this space will be smart, beautiful and not a phone wannabe.image (c) Samsung

Interactive spaces

finally we have interactive spaces.

As spaces become more connected they will give us data, visualisation opportunities and become more interactive.

This will raise some interesting challenges around control and visualisation but also things like privacy.

Novel interfaces

ajfisher Lab at Rockwell

Similarly how do you take your content and display it at the size of a building?

We’re seeing the first versions of this as tech demos, art installations and single products but expect to see this ramp up significantly over the next few years.image left (cc) ajfisherimage right (c) lab at rockwell http://lab.rockwellgroup.com/work/plug-in-play.html

How to prepare

So how do we prepare for these insane levels of fragmentation are going to occur - we already have to cover most of these? What resolution is a building?

The main way to start is by having excellent content and application structure.

Atomise your content

Breaking your content up atomically is critical. This allows it to become portable and unbinds it from your display methods

If this sounds familiar it’s because it is. This is content first design. The natural outcome of designing content first means it becomes a lot more portable.

Content choreography

You must consider content choreography so you can ensure the relationships between your content atoms are maintained.

Content sets

S M L XLFinally, you’ll need to think about content sets which I won’t go into too much tonight.

Conceptually these are like image sets but applied to all of your other content as well.

You need to consider what the content atom looks and behaves like as it changes from through small, medium, large and extra large contexts. How do aspects of the content get prioritised within each of these contexts?

This means our content responds much more cleanly across devices and can scale from the micro to the massive much more easily and we can finally decouple ourselves from specific device resolutions.

Contextual APIs

As we start thinking about content in these terms it begins to look more and more like a contextual API.

A good current example of this is Gmail’s Actions system. The way you interact with it varies across device contexts but the purpose remains consistent and it doesn’t take much to imagine this being extremely useful on a smart watch for example..

New interactions

flickr (cc) Mary Scheirer

Now we’ve looked at new devices, what is happening interaction wise?

We are now at a point where touch is a fairly legitimate input method, though strangely we’re doing it quite poorly on the web. image: h"p://www.flickr.com/photos/maryscheirer/4906553164/

Touch

Touch is slowly starting to make us rethink the way we design interactions because a lot of our standard interaction methods are being removed such as the hover context, but there’s still plenty of navigation systems that rely on it.

Voice

flickr (cc) Brett Jordan

We’re on the cusp of voice being useful. It’s now accurate enough to do useful things like search, create reminders or get directions however most people get a bit self conscious about using it in public.

How many of you have used voice for real though?

A handful - even a couple of years ago that probably would have been zero and even if there was one they would have been too embarrassed to put their hand up.image (cc): http://www.flickr.com/photos/x1brett/8464723074

Gesture

flickr (cc) Brett Jordan

Gesture is getting better too.

The Wii made it culturally acceptable, Microsoft made the technology work without a controller and now LeapMotion, PrimeSense and others are making it affordable and useful everywhere.

Gestural resolution

(c) Thalmic Labs

We’ve also got wacky products like Myo coming down the line as well which will increase the range of gestures we will be able to reliably detect.Video: (c) Thalmic Labs: https://www.thalmic.com/en/myo/

Make this better

So what can we do to prepare?

The first thing is that as web people we need to get so much better at touch in a composite setting. That means when touch and traditional inputs overlap on the same experience such as viewing a full “desktop” site on a tablet that doesn’t have a pointer interaction method.

Do this now

Stay in line with the standardsDesign for bigger target areasUnbind UI events from element events

Everything you build needs to understand touch events now and migrate to the newer standards as they become viable.

But in addition from a design perspective, go for naturally bigger targets, look at things like quick touches triggering a hover state.

Consider having libraries of effects for example popping a menu but then multiple interaction methods that do it. Stop binding UI events directly to the element events that trigger them.

Zero computer users

flickr (cc) Johan Larsson

We must get better at this than we are now.

Go get an iPad for a day and use it to do absolutely EVERYTHING. This is your user in another 18 months. See how bad the experience is and do something about it.image (cc): http://www.flickr.com/photos/johanl/5619897608/

Use your voice

Google

With voice, Google has an excellent speech recognition API. Play with it. Look at how you can enhance things like site search using speech recognition. This will help make your site more accessible - particularly if someone is physically impaired.

Don’t forget that even mobile users find typing difficult and error prone.

Wave your arms

LeapMotion

Gesture is a bit harder because it’s tiring for the user. However when we’re dealing with larger displays such as TVs or buildings, gesture can be extremely intuitive as an interaction method.

Get a Kinect or a leap motion and start playing with it. Image (c): leapmotion.com

What experiences?

So what types of experiences can we create?

Each one of touch, voice and gesture has limitations on their own but combined they become more useful and natural.

Combo multipliers

For example you can use touch or gesture to group things together and say “Send these to John”.

The combination of these affordances becomes more useful especially in not-quite-traditional computing contexts such as controlling a large display.

Retail

With devices like a leap motion, swishing through a product list becomes very simple just by swiping your hand and not too tiring. Other gestures could trigger content enhancements such as popping additional product details for example or throwing things into your shopping cart.

So there’s a lot of potential to be had by overlapping new interaction methods.

Computing substrate

flickr (cc) Wonderlane

At the outer edge of out time period we start to look at computronium or computing as substrate. This creates extremely elastic and slippery user experiences as the notion of when you’re computing breaks down heavily.Image: http://www.flickr.com/photos/wonderlane/3198166347

Too many devices

One of my pet annoyances is I have all these devices. I use all of these every single day and that’s not counting the iPhone I took the picture with.

Many of them do functionally similar things but some are better at certain tasks than others.

The only thing making this marginally better is that I can install chrome on some which gives me a degree of consistency and portability from one device to another.

Devices per user are beginning to proliferate very rapidly.

Resizable screens

Samsung Flexible OLED concept

So as a result we bump into another techno-cultural meme which is the resizing display - the elastic piece of glass. And there are people legitimately researching this tech.

But it misses the point.video: http://www.youtube.com/watch?v=f4AhTiQkWwk

Context drives uses

Flickr (cc): Henry Stratford

It’s the context shift that makes us want a different experience. When you’re standing on a crowded tram, a phone is perfectly useful. When I’m on a sofa a tablet is better.

I can send an email from either of these without too much trouble but if I sit down to write a chapter of a book I really do need a bigger screen, a windowing environment and definitely a keyboard.

But why do these have to be fundamentally different devices?image (cc): h"p://www.flickr.com/photos/henry_stradford/5348910688/sizes/o/

Mobile at the centre

Why don’t these other devices just provide additional affordances to my primary computing device - which let’s face it, is going to be my mobile phone?

Ubuntu Edge

(c) Ubuntu / Canonical

Ubuntu Edge - while it failed as a kickstarter - was heading in this way where the phone became the centre.

We don’t NEED multiple discrete devices that mostly do the same thing. However, our phone doesn’t need to BE a PC, it just needs to be able to use the resources of the PC when we want it to.image (c) canonical

Chromecast

(c) Google

Subsumption of environmental resources is a way off yet but I think Chrome is starting to show the way here. Think about multi-device sync and chrome cast. These are subsumption technologies.

Chrome doesn’t care about the underlying hardware, it just moves the experience where the user wants it to go. And this is just the start of this trend with the worst technology to do it.

How to prepare

So how do we prepare for this world?

Mainly we start by considering how our applications can change based on affordances presented to them that are changing. We need to consider state changes and make them work much much better.

Detecting context shifts

How do I know when someone has moved from a high bandwidth connection to a low one? How do I know if they’ve shifted from a touch context to a pointed one?

How do I know if someone has moved from a small resolution to a large one AFTER the page render has occurred?

We need to start thinking about responding to these context shifts because they will be common sooner than you think.

Retail

Retail has great potential here for me.

I can start an experience on my phone when I’m on the tram. I come home and it can shift to my TV so I get the benefit of a bigger display. At this point I can use gesture to start navigating through products and adding them to my cart.

Finally I use voice, or my onscreen or physical keyboard to put in my details and make a purchase. This is what a moving experience starts to become, a frictionless transition from my set of interaction methods to another as required by the user.

Continued change

So this was my view on the next handful of years.

We know will see a huge proliferation of devices leading to a lot of fragmentation.

Gesture, voice and touch will become more common and preferred as interaction methods.

Finally we’ll see capability being subsumed by primary devices that will enable movable and elastic experiences.

Humanist technology

flickr (cc) Ding Yuin Shan

Of course, none of this is set in stone, but one thing is certain: the web will be the main vehicle for this to occur.

The web is the only scaleable set of technologies that has connected a billion people and is capable of connecting 6 billion more.

However, our notions of what is a web browser are breaking down and even native apps still consume web technologies. The web has never been a monoculture and the proliferation of devices, interaction forms and technologies will create an even more diverse ecosystem.image (cc): http://www.flickr.com/photos/90461913@N00/7901102136/

Hard work is coming

flickr (cc) spinster cardigan

I can’t say the next handful of years will be easy as a designer or a developer. The shift to responsive design as a result of mobile has shown very clearly that the appetite for change is waning - particularly in the corporate world.

However these technologies are coming and they will reshape our behaviour. In 8 years we went from looking through to looking at to take a picture. Human behaviour is shifting rapidly, and we either embrace that or get left behind.image (cc): http://www.flickr.com/photos/84906483@N08/8747337118

Designing a Moving Experience

Andrew Fisher @ajfisherBe Responsive, 10 September 2013

Thankyou.

Get in touch:Andrew Fisher (CTO JBA)andrew@jbadigital.comhttp://twitter.com/ajfisherhttp://ajfisher.me/