Bi-manual

An interaction paradigm for augmented reality live

performance

Alexis Clay - Jean-Christophe Lombardo - Nadine Couture - Julien Conan

Human-Computer Interaction, Tourism and Cultural Heritage (HCITOCH 2012)

4th March 2011

2

Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Feedback from the audience:

120 submitted forms

4th March 2011

3

Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

80% return rate

90% found it innovative

Audience bright interest for 3D ribbons following the dancer’s

hands

Feedback from the audience:

120 submitted forms

4th March 2011

4

Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Feedback from the audience:

120 submitted forms

4th March 2011

5

Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

>90% wish to witness the evolutionof the show

6

1 VIRTUAL SCULPTURE

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING

5 PERSPECTIVES

Summary

Hand Interactions Stereoscopic Display

CAVE and development tools

Live Performance Application

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Prototype1

Prototype2

73D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Sketching SurfacesDeisinger et al - 2000

BLUIsculpt systemArctic Region Super Computing Center - 2002

State of the Art

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE

8

Surface DrawingSchkölne, Pruett, Schröder- 2001

Sketching SurfacesDeisinger et al - 2000

Freehand InteractionLarge Space Interaction and

visualization

Direct Metaphor Virtual matter seems to come out

right under the user’s hand

Large Space Painting Dance application

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Outcomes

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE

1

2

9

Schkolne et al. Surface Generation

Hands’ Joints Path

Application to DanceLive Performance

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Our Objective

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE

103D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Our Objective

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE

Credits: Samsung Galaxy SIII advertisement - 2012

11

Hand conformation

Wrists’ positions

Articulations and tip positions

Metaphor

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE2 MANUAL

INTERACTIONS

Basic InformationGenerate Virtual Objects

12

1/2s or 1/10s sample Surface between t and t-1

Movement parallel to the palm’s surface

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE2 MANUAL

INTERACTIONS

Shapes of the generated surface

1

Curves Surfaces

1rst metaphor: Shapes

13

Volume section = user hand visual surface2

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE2 MANUAL

INTERACTIONS

2nd metaphor: Volumes

2

Movement perpendicularly to the palm’s surface

14

More accurate Same as a tinier brush

Automatic closed surfaces fullfilment

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE2 MANUAL

INTERACTIONS

3rd metaphor: Curves

3

Index/Middle finger conformation

15

Thumb positions

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE2 MANUAL

INTERACTIONS

Pen-writing metaphor

« Draw »

« Free To Move »

16

Main interests

Simple

Direct metaphor

Natural modes

Hand orientation / Fingers tucked

1

2

3

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE2 MANUAL

INTERACTIONS

173D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE3 CAVE 3D PAINTING

CAVE 3D PAINTING PROTOTYPE STAGE 3D PAINTING PROTOTYPE

Immersion in virtual space Co-located interactions Display Accuracy Development Tools

Two users simultaneously max Cost 3mx3mx3m

3D Display in a Large Space Multi viewers Reduced cost

Dancer Blind From His Creations Occlusion Problem One Projection Screen

18

Hand IR Trackers

Glasses with Positions Trackers

Virtual Platformwith AR-Dtrack

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 3 CAVE 3D PAINTING

Instrumented Space

19

isiVR

Multi-display Cluster Position Tracking

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 3 CAVE 3D PAINTING

Scalable and Interactive Virtual Reality

Immersive 3D Visualization FrameWork

isiVR Features

Immersive ToolsReady-to-useOpenSceneGraph

Render Engine

20

Inputs OutputsBehaviors

Facilitate Additions/Modifications

Immersive ToolsReady-to-useOpenSceneGraph

Render Engine

openSource license

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 3 CAVE 3D PAINTING

isiVR

21

Flat Polygons

Control Points

Bézier Surfaces according toHand’s Movement

Positions: Middle Finger Distal Phalanx Wrist

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 3 CAVE 3D PAINTING

Schkölne Paradigm Application

223D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 3 CAVE 3D PAINTING

Surface Generation by Hand in a CAVE

233D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE3 CAVE 3D PAINTING

CAVE 3D PAINTING PROTOTYPE STAGE 3D PAINTING PROTOTYPE

Immersion in virtual space Co-located interactions Display Accuracy Development Tools

Two users simultaneously max Cost Effective 3mx3mx3m

3D Display in a Large Space Multi viewers Reduced cost

Dancer Blind From His Creations Occlusion Problem One Projection Screen

243D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 4 STAGE 3D PAINTING

Prototype Scheme

Signals received are different format

Instrumented User

25

17 Motion Trackers

Positions x,y,z(t)Orientations

SIGNALS

Gyroscope, magnetometers, and

accelerometers signals

Articulated and Rigid Kinematic Chain

IndividualBody Segments

Mtx Trackers

Xbus Master

Master Slave

XSens

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 4 STAGE 3D PAINTING

Motion Capture Suit

26

SIGNALS

0 < Flexion < 1

Erase Objects

Point an object

Choose color

Display Palette

Choose color

1

0

2

3

4

56

7

9

10

11 12138

Thumb_NearThumb_FarThumb_IndexIndex_NearIndex_FarIndex_MiddleIndex_NearIndex_FarIndex_RingRing_NearRing FarRing_LittleRing_NearRing_Far

123456789

1011121314

5DT14 Flexion Trackers

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 4 STAGE 3D PAINTING

Data Gloves

273D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE

Render EngineVisual Programming Editor

Procedural ApproachSDK

Behaviors Indirect Basis

Inputs Management Building Blocks

VIRTOOLS

4 STAGE 3D PAINTING

28

OBJECT

Actions Reactions

Loop Process

Settings

Variable Configuration

Custom Dialog Box

Message Handler

Building Blocks

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE

VIRTOOLS

4 STAGE 3D PAINTING

PROCESSING

ENTITY CREATION

BUFFER

COORDINATES Hands

x,y,z(t) vertices

Object Mesh

Lights

x,y,z(t+1) vertices

Material Texture

Camera

Shader

Main Object Sub Object

293D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE

VIRTOOLS

4 STAGE 3D PAINTING

303D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE

Obje

ctGN

Mes

hGN

curv

eA(t)

curv

eB(t)

curv

eC(t)

curv

eD(t)

posA

(t-1)

posB

(t-1)

posC

(t-1)

posD

(t-1)

Mat

eria

lGN

Text

ureG

NOp

acity

Mod

eOp

acity

Aver

age

Mat

eria

lPal

ette

posA

(t)po

sB(t)

posC

(t)po

sD(t)

VIRTOOLS

4 STAGE 3D PAINTING

313D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 4 STAGE 3D PAINTING

32

4.Tip

3.Distal

2.Medial

1.Proximal

More Details

Save CPU Resources

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 4 STAGE 3D PAINTING

Tesselation Post-Process1)

2)

3)

Tesselation

333D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 4 STAGE 3D PAINTING

Occlusion SCENE

AUDIENCE

Projection Screen

Constraint

Semi-transparent projection screen (between dancer and

audience)

343D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 4 STAGE 3D PAINTING

« La Mer », Debussy - 1905

353D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 4 STAGE 3D PAINTING

36

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING

1 VIRTUAL SCULPTURE 5 PERSPECTIVES

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Current Work and Applications

Current Work

Credits: Proteigon - 2011

Dancer’s Movement Parameters:

Space Time Weight Flow

“Computational models of expressive gesture in multimedia systems”,

G.Volpe et al, 2003

37

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING

1 VIRTUAL SCULPTURE 5 PERSPECTIVES

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Perspectives

Creations Visual FeedbackCreations Haptic Feedback

38

Authors

Alexis ClayJean-Christophe LombardoNadine CoutureJulien Conan

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Bi-manual

An interaction paradigm for augmented reality live

performance

Alexis Clay - Jean-Christophe Lombardo - Nadine Couture - Julien Conan

Human-Computer Interaction, Tourism and Cultural Heritage (ESTIA - INRIA)