bi manual 3d painting - an interaction paradigm for augmented reality live performance - slideshare
Post on 25-Jun-2015
269 Views
Preview:
DESCRIPTION
TRANSCRIPT
Bi-manual
An interaction paradigm for augmented reality live
performance
Alexis Clay - Jean-Christophe Lombardo - Nadine Couture - Julien Conan
Human-Computer Interaction, Tourism and Cultural Heritage (HCITOCH 2012)
4th March 2011
2
Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
Feedback from the audience:
120 submitted forms
4th March 2011
3
Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
80% return rate
90% found it innovative
Audience bright interest for 3D ribbons following the dancer’s
hands
Feedback from the audience:
120 submitted forms
4th March 2011
4
Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
Feedback from the audience:
120 submitted forms
4th March 2011
5
Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
>90% wish to witness the evolutionof the show
6
1 VIRTUAL SCULPTURE
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING
5 PERSPECTIVES
Summary
Hand Interactions Stereoscopic Display
CAVE and development tools
Live Performance Application
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
Prototype1
Prototype2
73D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
Sketching SurfacesDeisinger et al - 2000
BLUIsculpt systemArctic Region Super Computing Center - 2002
State of the Art
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE
8
Surface DrawingSchkölne, Pruett, Schröder- 2001
Sketching SurfacesDeisinger et al - 2000
Freehand InteractionLarge Space Interaction and
visualization
Direct Metaphor Virtual matter seems to come out
right under the user’s hand
Large Space Painting Dance application
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
Outcomes
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE
1
2
9
Schkolne et al. Surface Generation
Hands’ Joints Path
Application to DanceLive Performance
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
Our Objective
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE
103D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
Our Objective
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE
Credits: Samsung Galaxy SIII advertisement - 2012
11
Hand conformation
Wrists’ positions
Articulations and tip positions
Metaphor
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE2 MANUAL
INTERACTIONS
Basic InformationGenerate Virtual Objects
12
1/2s or 1/10s sample Surface between t and t-1
Movement parallel to the palm’s surface
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE2 MANUAL
INTERACTIONS
Shapes of the generated surface
1
Curves Surfaces
1rst metaphor: Shapes
13
Volume section = user hand visual surface2
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE2 MANUAL
INTERACTIONS
2nd metaphor: Volumes
2
Movement perpendicularly to the palm’s surface
14
More accurate Same as a tinier brush
Automatic closed surfaces fullfilment
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE2 MANUAL
INTERACTIONS
3rd metaphor: Curves
3
Index/Middle finger conformation
15
Thumb positions
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE2 MANUAL
INTERACTIONS
Pen-writing metaphor
« Draw »
« Free To Move »
16
Main interests
Simple
Direct metaphor
Natural modes
Hand orientation / Fingers tucked
1
2
3
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE2 MANUAL
INTERACTIONS
173D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE3 CAVE 3D PAINTING
CAVE 3D PAINTING PROTOTYPE STAGE 3D PAINTING PROTOTYPE
Immersion in virtual space Co-located interactions Display Accuracy Development Tools
Two users simultaneously max Cost 3mx3mx3m
3D Display in a Large Space Multi viewers Reduced cost
Dancer Blind From His Creations Occlusion Problem One Projection Screen
18
Hand IR Trackers
Glasses with Positions Trackers
Virtual Platformwith AR-Dtrack
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 3 CAVE 3D PAINTING
Instrumented Space
19
isiVR
Multi-display Cluster Position Tracking
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 3 CAVE 3D PAINTING
Scalable and Interactive Virtual Reality
Immersive 3D Visualization FrameWork
isiVR Features
Immersive ToolsReady-to-useOpenSceneGraph
Render Engine
20
Inputs OutputsBehaviors
Facilitate Additions/Modifications
Immersive ToolsReady-to-useOpenSceneGraph
Render Engine
openSource license
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 3 CAVE 3D PAINTING
isiVR
21
Flat Polygons
Control Points
Bézier Surfaces according toHand’s Movement
Positions: Middle Finger Distal Phalanx Wrist
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 3 CAVE 3D PAINTING
Schkölne Paradigm Application
223D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 3 CAVE 3D PAINTING
Surface Generation by Hand in a CAVE
233D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE3 CAVE 3D PAINTING
CAVE 3D PAINTING PROTOTYPE STAGE 3D PAINTING PROTOTYPE
Immersion in virtual space Co-located interactions Display Accuracy Development Tools
Two users simultaneously max Cost Effective 3mx3mx3m
3D Display in a Large Space Multi viewers Reduced cost
Dancer Blind From His Creations Occlusion Problem One Projection Screen
243D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 4 STAGE 3D PAINTING
Prototype Scheme
Signals received are different format
Instrumented User
25
17 Motion Trackers
Positions x,y,z(t)Orientations
SIGNALS
Gyroscope, magnetometers, and
accelerometers signals
Articulated and Rigid Kinematic Chain
IndividualBody Segments
Mtx Trackers
Xbus Master
Master Slave
XSens
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 4 STAGE 3D PAINTING
Motion Capture Suit
26
SIGNALS
0 < Flexion < 1
Erase Objects
Point an object
Choose color
Display Palette
Choose color
1
0
2
3
4
56
7
9
10
11 12138
Thumb_NearThumb_FarThumb_IndexIndex_NearIndex_FarIndex_MiddleIndex_NearIndex_FarIndex_RingRing_NearRing FarRing_LittleRing_NearRing_Far
123456789
1011121314
5DT14 Flexion Trackers
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 4 STAGE 3D PAINTING
Data Gloves
273D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE
Render EngineVisual Programming Editor
Procedural ApproachSDK
Behaviors Indirect Basis
Inputs Management Building Blocks
VIRTOOLS
4 STAGE 3D PAINTING
28
OBJECT
Actions Reactions
Loop Process
Settings
Variable Configuration
Custom Dialog Box
Message Handler
Building Blocks
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE
VIRTOOLS
4 STAGE 3D PAINTING
PROCESSING
ENTITY CREATION
BUFFER
COORDINATES Hands
x,y,z(t) vertices
Object Mesh
Lights
x,y,z(t+1) vertices
Material Texture
Camera
Shader
Main Object Sub Object
293D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE
VIRTOOLS
4 STAGE 3D PAINTING
303D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE
Obje
ctGN
Mes
hGN
curv
eA(t)
curv
eB(t)
curv
eC(t)
curv
eD(t)
posA
(t-1)
posB
(t-1)
posC
(t-1)
posD
(t-1)
Mat
eria
lGN
Text
ureG
NOp
acity
Mod
eOp
acity
Aver
age
Mat
eria
lPal
ette
posA
(t)po
sB(t)
posC
(t)po
sD(t)
VIRTOOLS
4 STAGE 3D PAINTING
313D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 4 STAGE 3D PAINTING
32
4.Tip
3.Distal
2.Medial
1.Proximal
More Details
Save CPU Resources
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 4 STAGE 3D PAINTING
Tesselation Post-Process1)
2)
3)
Tesselation
333D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 4 STAGE 3D PAINTING
Occlusion SCENE
AUDIENCE
Projection Screen
Constraint
Semi-transparent projection screen (between dancer and
audience)
343D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 4 STAGE 3D PAINTING
« La Mer », Debussy - 1905
353D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL
SCULPTURE 4 STAGE 3D PAINTING
36
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING
1 VIRTUAL SCULPTURE 5 PERSPECTIVES
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
Current Work and Applications
Current Work
Credits: Proteigon - 2011
Dancer’s Movement Parameters:
Space Time Weight Flow
“Computational models of expressive gesture in multimedia systems”,
G.Volpe et al, 2003
37
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING
4 STAGE 3D PAINTING
1 VIRTUAL SCULPTURE 5 PERSPECTIVES
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
Perspectives
Creations Visual FeedbackCreations Haptic Feedback
38
Authors
Alexis ClayJean-Christophe LombardoNadine CoutureJulien Conan
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26
Bi-manual
An interaction paradigm for augmented reality live
performance
Alexis Clay - Jean-Christophe Lombardo - Nadine Couture - Julien Conan
Human-Computer Interaction, Tourism and Cultural Heritage (ESTIA - INRIA)
top related