first personstatic.lukew.com/fpi_09242012.pdftap double tap swipe/drag flick pinch spread press...

1

USER INTERFACESFIRST PERSON

LUKE WROBLEWSKI BREAKING DEVELOPMENT

@lukew

2

ABSTRACTION• User interfaces give you access to computing capabilities• By abstracting away all the messy stu!

3

PUNCHED CARD INTERFACES• Lived outside the computer• Had specific formats that needed to be learned• Had to be punched by hand or on a key punch machine• Had to be fed into a computer• Output was delivered on paper

Flickr photo by Chris Campbell

4

PUNCHED CARD INTERFACES

5

Punched Cards Interfaces

INPUT OUTPUT

LAYERS OF ABSTRACTION

6

COMMAND LINE INTERFACES• Input typed directly on computer• Required specific commands & syntax• Objects and actions hidden behind cryptic commands

Flickr photo by Moriarty

7

./bagcheck bag additem name=’toast’

8

9

Punched Cards Interfaces

INPUT OUTPUT

Command Line Interfaces

LAYERS OF ABSTRACTION

10

GRAPHICAL USER INTERFACES• Made objects and actions visible• Enabled manipulation through icons, controls, and mouse• Resulted in lots of time interacting with UI components

11

12

Punched Cards Interfaces

INPUT OUTPUT

Command Line Interfaces

Graphical User Interfaces

LAYERS OF ABSTRACTION

13

14

15

NATURAL USER INTERFACES• The content itself serves as the interface• Interact with guessable, physical, and realistic gestures• Make use of skills learned through a lifetime of living

16

TapDouble TapSwipe/DragFlickPinchSpreadPressRotate

17

18Source: http://live.leapmotion.com/about.html

• Control a computer in three dimensions • With your natural hand and finger

movements

19Source: http://live.leapmotion.com/about.html

20Source: http://bit.ly/L2Sb6B

21

22

23

24

Punched Cards Interfaces

INPUT OUTPUT

Command Line Interfaces

Graphical User Interfaces

Natural User Interfaces

LAYERS OF ABSTRACTION

25

Punched Cards Interfaces

INPUT OUTPUT

Command Line Interfaces

Graphical User Interfaces

Natural User Interfaces

LAYERS OF ABSTRACTION

First Person User Interfaces

26

NEW INPUT METHODS

Mouse Click Wheel Multi-Touch

“Each new input method led to not just a disruption but new platforms and new

business models.”

Source: http://bit.ly/udBlqO

27Source: http://tnw.co/yUafAx

1975 1977 1979 1981 1983 1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011

Share of Personal Computing

ATARI

TRS-80

COMMODORE

AMIGA

WINTEL

APPLE

Source: http://bit.ly/xDzag2

Mouse

28Source: http://tnw.co/yUafAx

ATARI

LG

SAMSUNGNOKIA

APPLE

SE

RIM

HTC

Q2/2007 Q1/2008 Q4/2008 Q3/2009 Q2/2010 Q1/2011

Mobile Phone Profits

Source: http://bit.ly/udBlqO

Multi-Touch

29

Punched Cards Interfaces

INPUT OUTPUT

Command Line Interfaces

Graphical User Interfaces

Natural User Interfaces

First Person User Interfaces

LAYERS OF ABSTRACTION

30

Allow people to digitally interact with the real world as they are currently experiencing it.

FIRST PERSON USER INTERFACES

31

Sensor Capabilities• Location detection: GPS, WiFi, cell towers• Orientation: direction from a digital compass• Device positioning & motion: from an accelerometer• Audio: input from a microphone; output to speaker• Video & image: capture/input from a camera• Device connections: through Bluetooth between devices• Proximity: device closeness to physical objects• Ambient Light: light/dark environment awareness• RFID reader: identify & track objects with broadcasted identifiers• Multi-touch sensors• Haptic feedback: “feel” di!erent surfaces on a screen• Biometrics: retinal, fingerprint, etc.

32

FIRST PERSON USER INTERFACES• Interaction through a set of always-on sensors• Instant, relevant output based on your current position• Or based on who or what is near you

Source: AcrossAir

33

1. Navigate the space around you2. Augment your immediate

surroundings3. Interact with nearby objects

or people

FIRST PERSON USER INTERFACES

34

1. Navigate the space around you2. Augment your immediate

surroundings3. Interact with nearby objects

or people

FIRST PERSON USER INTERFACES

35

36

37

38

39

40

41

Indoor Location

• Google Maps for Android: 2D floor plans, WiFi

• Nokia: 3D models, Bluetooth

• Broadcom: WiFi, Bluetooth, NFC and gyroscope, magnetometer, accelerometer, and altimeter

your entry point (via GPS), then count your steps (accelerometer), direction

(gyroscope), & altitude (altimeter)

42

Location Detection

arrange list based on location

43

44

Device Motion

accelerometer

• Measures motion of device (roll & pitch) & force of gravity

• Linear acceleration

magnetometer

• Digital compass

• Magnetic field strength

gyroscope

• Measures rate of change around an axis

• Angular speed

6-axis motion and orientation sensing9

Source: http://bit.ly/wVgWAG

46

47

48

49

50

51

Multiple Sensors

“uses every sensor in your handset to track your activities, location and environment.”

52

1. Navigate the space around you2. Augment your immediate

surroundings3. Interact with nearby objects

or people

FIRST PERSON USER INTERFACES

53

54

Augmented Outdoors

55

56

When discovered by users boosted their sustained tra"c by 40 to 50 percent.

“It was sort of beyond our expectations. We had no idea.”

Yelp CEO, Jeremy Stoppelman

57

MARKERS

OVERVIEW &ORIENTATION

DATA WINDOW

58

MARKERS

OVERVIEW &ORIENTATION

DATA WINDOW

IBM Seer at Wimbledon

59

MARKERS

OVERVIEW &ORIENTATION

ZOOM/SCALE

DATA WINDOW

Wikitude AR Guide

60

MARKERS

OVERVIEW &ORIENTATION

Marker Overload

61

FIXED LOCATION

SPATIAL GRID

TRANSPARENT UI

LOCK DATA

62

FIXED LOCATION

TRANSPARENT UI

OVERVIEW &ORIENTATION

FIXED LOCATION

MARKERS

63

OVERVIEW &ORIENTATION

TRANSPARENT UI

64

HIDE WHEN NOT IN LINE OF SIGHT

3D MARKERS

65

66

Usage Curve

Emergent

Growth

Mature

Declining

Wearables Smartphones &Tablets

Laptops Desktops

67

Usage Curve

Emergent

Growth

Mature

Declining

First Person Interfaces

Natural User Interfaces

Graphical User Interfaces

Command Line Interfaces

68

1. Navigate the space around you2. Augment your immediate

surroundings3. Interact with nearby objects

or people

FIRST PERSON USER INTERFACES

69

Make Contacts

70

Process Receipts

71

Deposit a Check

72

File Taxes

Buy Something

73

Buy Something

74

Translate Text

75

76

77

Take a Payment

78

79

Identify Music

80

Identify Shows

81

82

83

Facial Detection in iOS 5

Faces within a picture, including the locations of eyes & mouths

84

Facial Recognition

85

1. Navigate the space around you2. Augment your immediate

surroundings3. Interact with nearby objects

or people

FIRST PERSON USER INTERFACES

86

Awkward Interactions

AWKWARD INTERACTIONS

SMALL SCREEN, BIG WORLD

Flickr photo by Nokia Point & Find

87

Nerd.Found.

Flickr photo by Nokia Point & Find

88

89

Near Field (NFC)

• Close-range radio communication between phones/devices, tags

• Require a distance of 4cm or less to initiate a connection

• Can read RFID tags (one-way)

90

91

Near Field (NFC)

92

93

94

95

Further reduce abstraction by allowing people to interact digitally with the real world as they are currently experiencing it.

FIRST PERSON USER INTERFACES

1. Navigate the space around you2. Augment your immediate surroundings3. Interact with nearby objects or peopleNEED DESIGN HELP

96

THANKS

LUKEW.COM @LUKEW