1

Ken HinckleyPatrick BaudischGonzalo RamosFrancois Guimbretiere

Microsoft Research

Scriboli: High Performance Pen Interfaces

2

• Copy + Paste: 1. Select Lasso Mode

The Traveling (pen) SalesmanProblem

3

The Traveling (pen) SalesmanProblem

• 2. Circle objects

4

The Traveling (pen) SalesmanProblem

• 3. Invoke Edit Menu

5

The Traveling (pen) SalesmanProblem

• 4. Choose Copy Command

6

The Traveling (pen) SalesmanProblem

• 5. Invoke Edit Menu again

7

The Traveling (pen) SalesmanProblem

• 6. Choose Paste

8

The Traveling (pen) SalesmanProblem

• 7. Acquire the copy

9

The Traveling (pen) SalesmanProblem

• 8. Drag copy to desired position.

10

The Traveling (pen) SalesmanProblem

• The well traveled pen.

11

• Waste time moving the pen• Focus attention on small targets• Same steps every time

• But what did I forget?

The Traveling (pen) SalesmanProblem

12

The Traveling (pen) SalesmanProblem

• 9) Forgot to reselect PEN mode. Oops.

13

• Speed without keyboard hotkeys• Cognitive footprint diminishes with use

•Recall / declarative procedural skill•Minimal demands on visual attention•Repeatable motions for the user•Predictable & dependable system behavior

• Expressiveness•Support many command structures &

apps•Economy of design

•only a few things to learn

Scriboli Design Goals

14

SKETCH – Brown University

• Adding 1 more gesture breaks everything• hard to learn, gestures specific to 3D editing

15

Recognize Pen Gestures?• Classic problem: “ink” or “gesture” ?

•Hard problem: any ‘gesture’ could be ink• Inferred decision comes too late

•real-time interactive feedback difficult

• Make it a physical skill•All we need to solve this is ONE BIT of info•physical button on bezel •habit-forming, deterministic, low

attentional demand•nonpreferred hand, no trip to “lasso mode”

16

Ink/Gesture study w/ U. Washington

• 5 Techniques:• Barrel Button• Tap+hold• NPH button• Pen Pressure• Pen Eraser

17

Phrase Structure for Scriboli

Separator(start)

Press “Gesture”

SeparatorScope/Command

SeparatorCommand/Parameters

Separator(Done)

Scope Command Parameters

Springboard modeCan’t get stuck, unavoidable feedback

18

Separator(start)

Press “Gesture”

SeparatorScope/Command

Pigtail

SeparatorCommand/Parameters

Separator(Done)

Scope Command Parameters

Fast, based on muscle “memory”Reliable and easily chunked by users

Phrase Structure for Scriboli

19

Phrase Structure for Scriboli

Separator(start)

Press “Gesture”

SeparatorScope/Command

Pigtail

SeparatorCommand/Parameters

Crossing boundary

Separator(Done)

Scope Command Parameters

Merge command selection and direct manipulation

FlowMenu [Guimbretiere 00], Control menu [Pook 00]

20

Phrase Structure for Scriboli

Separator(start)

Press “Gesture”

SeparatorScope/Command

Pigtail

SeparatorCommand/Parameters

Crossing boundary

Separator(Done)

Scope Command Parameters

One stroke for scope, command, and parametersNon-modal system (quasi-modal)

21

• A delimiter is “something different”• lexical structure of stroke

• e.g. self-crossing gesture stroke

•DEMO / VIDEO :delimiter techniques

Delimiters for scope selection + marking

22

Summary: Grammar for pen input• Scriboli proposes fundamental building

blocks• Links together object, verb, & indirect

object in fast, fluid, unambiguous command phrases

Written textWritten text

separate wordssentence startsentence enddelimits clause“The cat sat on the mat”

Punct.Punct.

spacecapitalsperiodcommaobjectverbindirect obj.

Scriboli EquivalentScriboli Equivalent

Pen up/pen downGesture button downGesture button upPigtail (delimiter)Scope (lasso, line, …)Marking menuCrossing manip. phase

23

Stitching: Pen Gestures that Span Multiple Displays

Microsoft ResearchKen HinckleyGonzalo RamosFrancois GuimbretierePatrick BaudischMarc Smith

24

Wireless Network 2004 = Horseless Carriage 1904?

• Are Wireless Networks really just wired networks without the wires?

• Or are they something completely different?

25

Wireless Device Soup: Which links does the user want?

26

• Users need techniques to intuitively form bridges between devices• How do users name the devices to

connect?• What is type / purpose of the connection?• Parameters? (Who copies what, to

where?)

Name That Device

27

• Pen stroke that spans displays

• Move the pen

• Cross over bezel

• Finish stroke on nearby tablet

• System infers connection

Stitching

path taken by the pen

transferred pictures

path taken by the pen

transferred pictures

path taken by the pen

transferred pictures

28

• AVI’04• ACM Advanced Visual Interfaces 2004

VIDEO

29

Establishing a Connection• Recognizes timing & dynamics of pen trajectory

• There is nothing special about the pen!• Wireless signal strength determines nearby devices

Tablet 1 Tablet 2

Δt

30

How Do Users Share Physical Space?• SociologyProxemics: How people use

space• Invisible bubble that surrounds an individual

31

6. How Do Users Share Physical Space?• SociologyProxemics: How people use

space• Invisible bubble that surrounds an individual

• Homework assignment:• Sit right next to someone at

airport (when it is not necessary to do so)

• Time how long it is before the other person leaves

32

5. How Do Users Share Physical Space?• SociologyProxemics: How people use

space• Invisible bubble that surrounds an individual

33

34

• Don’t require contact : touching is taboo• “Do they have to be right next to one another?”:

10-40cm• “wide screen would be nice for collaboration”• “no two faces trying to peek at only one screen”

Proxemic Lessons for Stitching

35

• Don’t require contact• “Do they have to be right next to one another?”: 10-40cm• “wide screen would be nice for collaboration”• “no two faces trying to peek at only one screen”

• Give users flexibility to be involved, or not• Intimate Spaces: Combining screens. For close

collaboration with friend or trusted colleague• Personal Spaces: Tablets can be separated by up

to 30” yet still possible to stitch to give files to colleague, etc.

• Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach

Proxemic Lessons for Stitching

36

• Multiple Devices: Cooperative Stitching

Ongoing work

User 1

User 2

Use

r 3U

ser 4

37

• Multiple Devices: Cooperative Stitching

Ongoing work

User 1

User 2

Use

r 3U

ser 4

38

Questions?

39

Results – Completion Time

• Left bar – main block; right bar – repeated block• Dotted bars are Pigtail2 design iteration

1.171.01

1.47

1.260.95

1.381.23

0.84

1.92

1.56

0

0.25

0.5

0.75

1

1.25

1.5

1.75

2

2.25

2.5

Button Button-RI Handle Handle-RI Pigtail Pigtail-RI Pigtail2 Pigtail2-RI Tout Tout-RI

Button Handle Pigtail Pigtail2 Timeout

40

Results – Error Rate

• Error rate of selecting wrong marking direction

4.6%

8.1%

2.7%

0.8%

6.0%

2.5%

4.0%

0.0% 0.4% 0.3%0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

6.0%

7.0%

8.0%

Button Button-RI Handle Handle-RI Pigtail Pigtail-RI Pigtail2 Pigtail2-RI Tout Tout-RI

Button Handle Pigtail Pigtail2 Timeout

41

• Handle can get in the way• Adds some visual clutter• Must check to be sure landed on handle

• Pigtail handles more than one scope elegantly

• Self-referential gesture

Handle vs. Pigtail