Willem Mestdagh - Kevin Cordemans

PapersMS Research

PlayAnywhere:Compact Interactive Tabletop Projection-Vision System []

Andy Wilson

Thinsight:Integrated Optical Multi-touch Sensing through Thin Form-factor Displays []

Bill Buxton, Steve Hodges, ...

PlayAnywherePlayAnywhere:

A Compact Interactive Tabletop Projection-Vision System

Andy Wilson at Microsoft Research, Redmond

2005

Introduction

Looks great...

But... Calibration? Installation? Portability? Cost?

Mainstream Consumer AcceptanceMulti-touch: new sensing and display technologies

Ubiquitious computing

PortabilityEase of installation

Ease of use: any surface?Calibration

Cost

Devise an easy-to-use, low-cost multi-touch system...

PlayAnywhereFront-projected computer vision-based

interactive table system which uses a new projecting technology to obtain an exceptionnaly self-contained form factor

Requirements:Quick set up: no calibration beyond factoryNo “installation”Usable on any flat surfaceCompact but still displaying and

sensing over a large surface (form factor)General purpose(Low cost)

Future?Canesta projection keyboard

PlayAnywhereWhat is it?Known problems in existing systemsHow does it work?

PlayAnywhere configuration Projector Camera and IR Illuminant

Image Processing Image Rectification Touch and Hover Analysis of shadow PlayAnywhere Visual Code Flow Move

Computer Vision Pros and Cons

Known problems in existing systems

PlayAnywhereWhat is it?Known problems in existing systemsHow does it work?

PlayAnywhere configuration Projector Camera and IR Illuminant

Image Processing Image Rectification Touch and Hover Analysis of shadow PlayAnywhere Visual Code Flow Move

Computer Vision Pros and Cons

PlayAnywhere Configuration

All-in-One

IR illuminant

Projector (with mirrors)

Camera with IR-passing-filter

Prototype

ProjectorNEC WT600 DLP

40’’ diagonale afbeeldingVoordelen van de plaats van de projector

Gevaarlijke plaatsing projector is vermeden Wanneer camera en projector samen zijn geplaatst

geen nood om de camera te hercalibreren bij verplaatsing van het toestel

• Nadelen Beeldkwaliteit kan niet gegarandeerd worden,

want projectie-oppervlak is niet gekend

Camera en IR IlluminantDe projector belicht de scene met zichtbaar

stralingDe illuminant belicht de scene met infrarode

stralingCamera ontvangt alleen gereflecteerde

infrarode stralen dankij de infrared-passing-filter die aan de camera bevestigd is(Camera blind voor reflecties van de projectorstraling)

How does it work?Image Processing

Preparation: Image RectificationComputer Vision Techniques: Analysis of

ShadowHover and TouchFlow MovePlayAnywhere Visual Code

Image Rectification (1)

Remove distortion via standard bilinear interpolation techniques

Parameters for transformations need only be determined once (factory):Lens specs known Fixed topology device-plane (assumption)

Setup Good thing Bad thing

Wide lens Maximize usable surface Creates barrel distortion

Oblique position of camera

Optimizes form factor (compact device)

Produces projective distortion

Image Rectification (2)

Image Rectification (3)Input image and projected image are in a one-to-

one correspondence, meaning:A rectangular object on the surface appears as a

rectangular object in the input image at the same (scaled) coordinates.

Rectified image suited for computer vision techniques

Limitation: objects further away from the unit will appear at a lower resolution(Minimum effective resolution is less than that of the acquired image)

Hover and TouchDetect touch without relying on special

instrumentation of the surface, so that the device may operate on any available flat surface (goal)

PlayAnywhere: exploite change in appearance of shadows as an object approaches the surface

No tracking of fingers, ...

Hover and Touch

thresholding operation

HoverCandidate finger positions:highest on each of the distinct shadows which enter the scene from the bottom of the image

TouchAnalysis of shadow shapes:threshold width of finger shadow at a location slightly below the topmost point

Hover and TouchTo limit false positives, candidate finger must lie on

a shadow that extends to the bottom of the image

Objects on table can corrupt touch detection by mimicing arm shadows, meaning their shadows extends to the

bottom of the image

Presently, only one finger per hand(more fingers requires more spohisticated analysis)

Precision limited by image resolution

Flow MoveSupport direct manipulation of virtual objects

(doc’s, photo’s)Gestures that are natural to a specific activity

E.g. resizing and rotating photo’s as if they were lying on a table

PlayAnywhere: Optical flow techniquesBased on local texture on the movig objectNo absolute position-based tracker, no widgetsInstead simple statistics summarizing the flow

field, computing only relative motion information

Optical Flow (1)Optical Flow computed from the most recently

acquired rectified image I(t) and the previous image I(t-1)

Mask images with grid

Image (t-1) Image (t)

For each (x,y) on a regular grid, the integer vector quantity (dx, dy) is determined, such that: image patch P centered on (x, y) at time t-1 most closely matches image patch P’ centered on (x+dx, y+dy) at time t.

Image patches are compared by computing pixelwise absolute differences (low values close match).

If image patch P closely matches image patch P’, there is a good chance that the two patches are representing the same content, shifted to a different gridpoint in the interval.

(dx, dy) gives the direction of this shift.

Optical Flow (2)

Image(t)(x+dx, y+dy)Image patch P’

Image(t-1)(x, y)Image patch P

Resulting vector

Optical Flow (3)For a given patch in the image, select (dx, dy) that minimizes

Vector field can be used as an argument in a function to infer meaning from user’s

gesture

Resulting vector field

PlayAnywhere Visual CodesOok mogelijk te werken met infrarood

systemen in objecten mogelijkheid tot werken met barcodes (derde vb met barcode)

Resultaat http://youtube.com/watch?v=YRfWhUnYyY8

Nu : Thingsight

Beschrijving“Optical sensing system, fully integrated into

a thin form factor display”Scherm met multi-touch systeem2 mogelijkheden om multitouchsystemen uit

te werken : - een arrangement van electroden op het

oppervlak- werken met cameras : voor of achter het

scherm

VergelijkingBeide Touch-algoritme op basis van

schaduwinval van handenBeide Handig te gebruiken en te verplaatsenPrincipe werking ongeveer gelijk

verschil : thinsight werkt met elektroden op het scherm, playanywhere werkt met camera’s boven het scherm

Beiden hebben de mogelijkheid tot werken met externe objecten met een infrarode bron

Voordeel bij thinsight : geen oppervlakte voor apparaat nodig