interacting with

large screens

patrick baudisch

microsoft researchvisualization and interaction research

large screensbuilding a large display

focus plus context screens

interacting using the mouse

high-density cursormouse ether

interacting using pen & touch

drag-and-poptablecloth

(halo) return

large screens and multimon

... are coming

[Jon Peddie ResearchDec, 2002 N=6652]

No Multimon30%

Plan to Use Multimon

38%

Use Multimon32%

• information mural[Guimbretière, Winograd]

• on large screens optical flow helps navigation [Tan 2001]

• large screens help productivity tasks [Czerwinski 2003]

• multi-monitor setups: access palette windows in Photoshop, CAD… [Grudin 2001]

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

focus plus context screens

conclusions

• Hardware– At least one hi-res

display– At least one larger low-

res display

• Software– scaling of the display

content is preserved– resolution varies

skip to dnp

setup

application scenarios

video

field study: users & tasksSubject’s task Document/view

Sta

tic do

cum

en

ts

Web designer Page: 800 pixel

Mechanical engineer Polybot segment: 5cm

Graphic designer Poster: 1m

Architect in remodeling Building: 50m

Photogrammetry (2) Highway 2 miles

Geographic info. system County: 80km

Chip designers (2) Wafer: 12cm

Dyn

am

ic

Air traffic ctrl. tool builder Zone: 50km

Ego shooter gamer Surrounding: 360º

Submarine ROV op. Surrounding: 360º

Strategy gamers (2) Map: 30k pixel

Smallest detail Ratio

Table detail: 1 pixel 800

Clearance: 0.03mm 2,000

Align: 0.5mm 2,000

Accuracy: 1cm 5,000

Accuracy: 1 inch 100,000

Land boundaries: 0.5m 160,000

Grid: 0.5m 240,000

Plane distance in 25m steps 2,000

Aiming: 0.1º 3,000

Use arms: 1mm/0.05º 8,000

Aiming: 1 pixel 30,000

focus plus context screen VisualizationSame # of pixels

fisheye

55

overview plus detail

44

Display technology

homogeneous resolution

44wall-size, hi-res display

44

What participants

used

What participants

used

Available to½ of participants

Available to½ of participants

… andcurrentsolutions

experiment 1:• 3 interfaces:

• focus plus context screen• overview + detail• homogeneous

• 2 tasks

• 12 subjects from Xerox PARC• Within subjects, counter-balanced• Same number of pixels

task 1: closest hotel

8 maps per interface

F+C screen and O+D use same magnification factor

task 2: verify connections

Verify a different set of 24 connections on the board

0

100

200

300

400

500

600

700

Map task Board task

zooming + panning

overview+detail

focus+context screen

resultsA

vera

ge

ta

sk c

om

ple

tion

tim

es

in s

eco

nd

s

21% faster21% faster 36% faster36% faster

manually zooming

takes time

manually zooming

takes time

visually switching reorientation

visually switching reorientation

visually more ambiguous

visually more ambiguous

experiment 2:driving simulation

120 sec sequence100 fields of nails; 30 rocks; tradeoff

resultsM

ea

n n

um

be

r o

f co

llisi

on

s su

bje

cts

cau

sed

• Sweet spot:flight simulation, unmanned vehicles…

0

5

10

15

20

25

run-over nails rocks hit

overview+detailfocus+context screen

Error rate only 1/3 of two-

monitor setup

Error rate only 1/3 of two-

monitor setup

Subjects preferred thef+c interface

Subjects preferred thef+c interface

observations

• low-res periphery is often ok

• on a wall-size display: If periphery was hi-res I still could not read it due to distance

focus-plus-context interaction

toc

• mouse • pointer moves too fast, users lose track of it• pointer behaves in unpredicted ways when

crossing screen boundaries

• touch/pen input• users cannot reach their content anymore

ktop?ktop?we observed: lots of input problems

conclude

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

high density cursor

conclusions

problem: mouse across long distances

• large screen longer distances

higher mouse acceleration

• temporal aliasing: 500 pixels jumps

• lack of visual continuity & weak stimulus users lose track of the pointer

the problem will get worse

• cursor update is limited by screen refresh rate

• screen refresh rate has actually decreased (LCDs)• larger screens + lower refresh rate status quo• future: even larger screens problem will get worse

how it works

previous cursorposition

current cursorposition

mouse

motion fill-in cursors

current framefill-in cursors

previous frame

inserts cursor image between actual cursor positions the mouse cursor appear more continuous

this is not the mouse trailthe windows

mouse trail…• makes mouse

trail last longer• drawback: cursor

images lag behind

...is not high-density cursor

• hd cursor makes mouse trail denser

• lag-free: mouse stops => cursor stops

video

benefitsprevious cursor position

mouse

motion

current cursor position

fill-in cursors

current framefill-in cursors

previous frame

mouse

motion

1. mouse cursor appear more continuous easier to track the cursor

2. higher “visual weight” easier to re-acquire the cursor

designs alternatives

acceleration

• reference: exponential acceleration

a

b

d

e

f

c

h

g

frame

designs alternativesa

b

d

e

f

c

h

g

frame acceleration

• motion blur with higher weight

designs alternativesa

b

d

e

f

c

h

g

frame acceleration

• temporal super-sampling vs. motion blur

chose discreet version1. latest cursor position is always shown blur-free and in full opacity2. appearance that users are familiar with today3. computationally less expensive

chose discreet version1. latest cursor position is always shown blur-free and in full opacity2. appearance that users are familiar with today3. computationally less expensive

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• density = detectability vs. intrusiveness

transfer function

mouse speed

distancebetweencursorimages

onset threshold(configurable)

cursor trail provides no speed cues

hd c

urso

r has

no

effe

ct

transfer function(configurable)

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• optional cursor growth

user study• conducted pre-study to define interface candidates• interfaces: control vs.

high-density cursor (conservative, tripleDensity, plusScaling) • fitts’ law task• triple-mon: button located at 5” to 40” distance• participants: 7 external participants, 5 coworkers

• hypotheses• high-density cursor faster• the greater the distance the

greater the effect• tripleDensity and plusScaling

faster than conservative

resultsti

me %

rela

tive t

o r

eg

ula

r cu

rsor

90

92

94

96

98

100

102

target distance (mm)125 250 500 750 1000

regular mouse cursor

high-density cursors

speedupup to 7%

conservative

+ scale

+3-dense

short distance

subjective satisfaction

Condition Liked Most Liked Least

control 0 7

HD_conservative 2 3

HD_tripleDensity 2 0

HD_plusScaling 6 0

• most participants did not notice that cursor was different!“did that condition use a different mouse acceleration?”…

• lesson we learned:display frame rate is not a hard limit

toc

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

mouse ether

conclusions

mouse ether

target

start

tasks with incorrecttasks with incorrectvisual feedbackvisual feedback

mouse across multimonproblem 1: different dpi

targetstart

low dpihigh dpi

mouse across multimonproblem 2: the gap

target

start

contents

• problem: why the cursor gets warped

• eliminating warping with mouse ether

• calibration

• user study: up to 28% faster in acquisition task

• conclusions: need to test fitts’ law

why the cursor gets warped

targetstart

low dpihigh dpi

target

start

system oblivious of• dpi differences

system oblivious of• gaps

mouse ether

target

start

targetstart

low dpihigh dpi

target

start

targetstart

low dpihigh dpi

with mouse ether

now users can acquire targets on direct path

now users can acquire targets on direct path

calibration

step 1step 2step 3

the math

screen i

screen j

(cx,cy)

(ex,ey,)

on in

it

+(dx,dy)every frame

(cx,cy)

(ex,ey,)’ ’

’ ’

off-screen travel

target

start

with mouse ether

more off-screen travel

target

start

blocked

same dpi

user’s perspective system’sperspective

target

blocked

high dpilow dpi

start

start

target

target

start

same dpi

with mouse ether

target

high dpilow dpi

start

start

target

name ether

taming off-screen travel

• iteration 1:restrict to convex hull

• iteration 2:allow entering ether only for the purpose of transit• towards another screen• at sufficient speed

• otherwise screen boundaries “hard”

start

target

user study

• 2x 18” screens• 1280x1024 and

800x600 resolution• eight participants• WinFitts software• 72 movements

(=4 trials x 9 paths x 2 directions)

results

High-Low Low-High0

200400600

8001000

120014001600

1800

Mov

emen

t Tim

e (m

s ±S

EM

)

Direction of Movement

Control

MouseEther

with mouse ether up to 28% faster

with mouse ether up to 28% faster

discussion

• multimon offers incorrect visual cues• this hinders successful target acquisition

• mouse ether improves acquisition speed by matching pointer behavior to visual cues

• off-screen travel allows for direct target acquisition

• next steps: investigate target acquisition under contradictory visual feedback ( Fitts’ law?)

toc

target

start

pen/touch input on walls-size displays

• mouse input can be mapped to any screen geometry

• not so for “direct” input technique, such as touch and pen input

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

drag-and-pop

conclusions

a technique for people with limited reach…

• user may not be able to reach content

• user may not be able to read content

drag-and-pop…

• let’s focus on a specific case for a moment: extend basic actions drag-and-drop and picking

scenario 1: long distances

dragging is designed for small screens…… but becomes time-consuming on large screens

scenario 2: dragging + bezels

dragging across bezels in display wall is no problem for the mouse…

…but a big problem when using pen/touch input

drag-and-pop: demo• users starts

dragging icon towards a distant folder or application

• icons of compatible type come towards mouse cursor

• user drops icon with minimal motion

• targets retractdrag-and-pop works across bezels

• drag-and-pop generalizes direct manipulation• bring content to the user• let the user interact with it• send content back

scenario 1: long distances

dragging on large screens

scenario 2: dragging + bezelsdragging acrossbezels in display wall

the displays we used…

design

selecting candidates

• initialize• all icons are candidates

• filter• eliminate icons with non-matching file types• eliminate icons that are too close• eliminate icons outside target angle• if necessary, restrict to some hard limit

preserving layout

• snap to grid• eliminate empty

rows and columns

• translate back• place center of

bounding box in front of user

• closer for experts

the rubber band

• animationdid not work

• “frozen”motion blur

• narrow midriff• suggests elasticity• clue for distance

• simplified version

getting it out of the way

• to rearrange icons on the desktop (overloaded):• any mouse motion moving away from the

“popped-up” icons de-activates drag-and-pop• introduce flick gesture into mouse motion

pre-study

• 3 layouts for study: sparse (11), frame (28), cluttered (35)

• 15 single, 6 dual, and4 triple monitor users

• overall resolutions 800,000 pixels to 3,900,000 pixels

• (= 66% more than the display wall used in the experiment).

user study

• participants: 2 female, 5 male• dynaWall

• 3 Smartboard• 15’ long (4.5m)• 3 x 1024x768 pixels

• native code not stable enough Macromedia Flash version

• task: drag icons into matching folder• highlighting disappeared when started• each desktop: 11-35 icons + 10 icons to be filed

results

• faster with drag-and-pop• error rate higher with drag-and-pop• most of the effect caused by the bezels

0 1 20

2

4

6

8

10

12

14

Number of Bezels Crossed

Drop

Pop

Control

Drag-and-pop

3.7 times3.7 timesspeedupspeedup

0

10

20

30

40

0 500 1000 1500 2000 2500Target Distance (pixels)

Drop

Pop

Control

Drag-and-pop

subjective satisfaction

• > 6 (out of 7)• “I liked using drag-and-pop”• “I always understood what was happening when drag-and-pop

was on”,• “I would use drag-and-pop for large displays.”

• < 3 for• “It took a long time to get used to drag-and-pop”• “It was hard to control what the targets did when drag-and-pop

was on.”

• drag-and-pop interface causes less manual stress and fatigue than the control interface

• lesson learned: the shortest connection between two points on a display wall is not a straight line (fixed)

drag-and-pick

problem• launch app or open file

drag-and-pick• user drags “background”• all icons in that direction

move to the cursor• user drags % releases

mouse over it• target is activated

current work: push-and-pop

outperforms pantograph-based techniques

0

1

2

3

4

5

6

7

8

9

10

11

12

198 455 480 504 920 1075 1123 1173 1651 1695 1850 1897Distance (pixels)

Tim

e (

s)

drag & drop

push & throw

pick & drop

acc. push & throw

drag & pop

push & pop

0

1

2

3

4

5

6

7

8

9

10

260 586 678 712 1363 1592 1624 1660 2476 2505 2734 2766

Distance (pixels)

Tim

e (s

)

drag & drop acc. push & throw

push & throw pick & drop

drag & pop push & pop

insert halo

outperforms pantograph-based techniques

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

tablecloth

conclusions

tableclothgeneralizing drag-and-pop to arbitrary interactions

for users withouttable manners

object I would like to interact with

window panning endswhen window edgereaches edge of screen

tablecloth panningallows panning any screenobject to any screen location

design challenge

• how to invoke tablecloth?• while dragging an object

1st design: crossing menu

dragging

2nd design: dual touch

jumpback

panning

pan

jump

dragging

video

focus-plus-context interaction

the distinction of screen space into focusand context regions is always there(focus plus context screens only emphasize it)

• focus-plus-context interaction• bring contents to user• interact• send contents back

toc

halo

discussion

• distance to peripheral space and human peripheral vision make peripheral content hard to read

• making periphery all hi-res does not solve this problem

• we need to support users using appropriate interaction techniques

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

read more

UIST 2001CHI 2002

INTERACT2003

CG&A 2005

INTERACT 2003GI 2005

CHI 2004

patrickbaudisch.com/projects

thanks!• try it out www.patrickbaudisch.com

• thanks to: • focus plus context screens: victoria

bellotti, nathan good, paul stewart, pamela schraedley

• high-density cursor: ed cutrell, george robertson

• mouse ether: ed cutrell, ken hinckley, rob gruen

• drag-and-pop: & ed cutrell, dan robbins, peter tandler, ben bederson…

• tablecloth: george robertson

can we push this further?

• (unless the app has fixed focus of attention) bring content to the user on demand

• that’s great—so maybe the periphery does not need to be full resolution after all

• can we push this even further? can we create a visual periphery without any peripheral screen space?

halo

store content in periphery

<view halo section is the small screen deck>

access off-screen content

• make halo arcs (or city lights) clickable to pan display to that location

vs.

• preserve users’ spatial memory by using drag-and-pop instead: bring peripheral content to PDA screen

Seamless integration of displays

a b

Context

No reflections on focus screen

Focus

If I had to commercialize today…

• Build all-analog immersive video link• Immersive telepresence,• Remote operated vehicles, drones• Immersive VR• Remote medicine

ContextWall

Extra

(a) locate (b) closest

(d) avoid(c) traverse

How does it work?

The scaling software

• Display image on two display units of different resolution• Similar to two-headed display• but display units are overlapping• and one of them has to be scaled down

• (Related work “Flux capacitor”[Dr. Emmett Brown, 1985])

Linux/VNC

app

focus

contextinput

server

clip

scale

viewer

viewer

Image viewer

context

Photoshop

scale

ACDsee

ACDsee

.gif

.gif

mousefork

input

focus

related work• acquiring distant targets

• move cursor with eye gaze (Sibert and Jacob, 2000), Magic pointing (Zhai et al., 1999)• flick snaps cursor to target (Dulberg et al. (1999)• sticky icons capture cursor (Swaminathan and Sato, 1997)• throwing gets across long distances (Geißler, 1998) • expanding targets save space on screen (McGuffin and Balakrishnan, 2002)• drag-and-pop (baudisch et al 2003)

• enhance detectability of the mouse cursor• <ctrl> for radar animation (Microsoft, Steve Bathiche)• cursor growth (Kensington Mouseworks 2001)• mouse trail for slow response LCDs (e.g. MS Windows)• liveCursor points in the direction of its motion (Ben Bederson)

• motion blur and temporal supersampling• reduce temporal aliasing, such as stroboscope, e.g. wheel spokes• rendering a scene multiple times (Dachille and Kaufman, 2000)• improve the perceived responsiveness of graphics apps (Conner and Holden, 1997)• help users anticipate motion (Chang, 1993; Thomas & Calder, 2001)

design goals

• for users who track the cursorenhance the predictability of the cursor path• enhanced trail density/continuous blur• smooth interpolation of the cursor path• preservation of trail density as a cue for cursor speed.

• for users who reacquire the cursorincrease the detectability of the cursor (visual weight)• enhanced trail density• enhanced cursor opacity• and cursor scaling.

• preserve responsiveness

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• distance between cursor images as cue for mouse speed

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• smooth interpolation

bezier interpolation

cursor position

1. linear interpolation

2. attraction point

3. interpolate

pre-study

• goal: define interfaces for user study• participants: 14 coworkers

• informal procedure• try out high-density cursor• try out different settings (density, onset…)• choose “favorite” setting

• resulting interface parameters• 12-17 pixels/frame vs. 35 pixels/frame• distance = sqrt(n)• cursor growth on or off

touch/pen input breaks

touch/pen-input + multimon

touch and pen input renaissance• PDAs• Tablets• Liveboards /

Smartboards

multi-display systems• DynaWall, iRoom

Smartboard wall• connect tablet to

external screen• …

scenario 1: tables + screen

tablet users scribble with pen… but filing iconsinto folder on external monitor requires mouse

scenario 1: tablet + screen

filing icons into folder on external monitor

related work• techniques for transferring information

• drag-and-drop avoids hidden clipboard (e.g. Xerox Star)• hyper dragging (Rekimoto, 1999)• pick-and-drop (Rekimoto, 1997)+ take-and-put (Streitz et al., 2001)

• overcome large distances• magic pointing (Zhai et al., 1999) requires an indirect input device• gesture input techniques (Rubine, 1991)• throwing (Geißler, 1998) for reliable target acquisition?• laser pointers to acquire targets on a Smartboard (Myers et al. 2002)

• mouse-based interaction techniques• lodestones and lay lines (Jul, 2002)• flick (Dulberg et al., 1999)• sticky icons (Swaminathan and Sato 1997)

related work

• driving directionsvs. route planning aids

• overview-plus-detail• focus-plus-context

• pointing into off-screen space

inside applications…

• drag-and-pop workseven if target is• occluded• clipped• closed (folder)

• use the concept tofile emails?

intrusion border

handle

space for arcs…

and for corner arcs

reserve space for content

arc length = distance

handling many objects

• find best (restaurant): relevance cut-off

• see all (dangers): merge arcs

a first attempt… city lights