Download - Bi Manual 3D Painting - An Interaction Paradigm for Augmented Reality Live Performance - slideshare

Bi-manual

An interaction paradigm for augmented reality live

performance

Alexis Clay - Jean-Christophe Lombardo - Nadine Couture - Julien Conan

Human-Computer Interaction, Tourism and Cultural Heritage (HCITOCH 2012)

4th March 2011

2

Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications

3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26

Feedback from the audience:

120 submitted forms

4th March 2011

3



80% return rate

90% found it innovative

Audience bright interest for 3D ribbons following the dancer’s

hands


120 submitted forms

4th March 2011

4




120 submitted forms

4th March 2011

5



>90% wish to witness the evolutionof the show

6

1 VIRTUAL SCULPTURE

2 MANUAL INTERACTIONS

3 CAVE 3D PAINTING

4 STAGE 3D PAINTING

5 PERSPECTIVES

Summary

Hand Interactions Stereoscopic Display

CAVE and development tools

Live Performance Application


Prototype1

Prototype2


Sketching SurfacesDeisinger et al - 2000

BLUIsculpt systemArctic Region Super Computing Center - 2002

State of the Art


3 CAVE 3D PAINTING

4 STAGE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE

8

Surface DrawingSchkölne, Pruett, Schröder- 2001

Sketching SurfacesDeisinger et al - 2000

Freehand InteractionLarge Space Interaction and

visualization

Direct Metaphor Virtual matter seems to come out

right under the user’s hand

Large Space Painting Dance application


Outcomes


3 CAVE 3D PAINTING


SCULPTURE

1

2

9

Schkolne et al. Surface Generation

Hands’ Joints Path

Application to DanceLive Performance


Our Objective


3 CAVE 3D PAINTING


SCULPTURE


Our Objective


3 CAVE 3D PAINTING


SCULPTURE

Credits: Samsung Galaxy SIII advertisement - 2012

http://www.youtube.com/watch?v=jCg6eL8pnnc



11

Hand conformation

Wrists’ positions

Articulations and tip positions

Metaphor


3 CAVE 3D PAINTING


SCULPTURE2 MANUAL

INTERACTIONS

Basic InformationGenerate Virtual Objects

12

1/2s or 1/10s sample Surface between t and t-1

Movement parallel to the palm’s surface


3 CAVE 3D PAINTING


SCULPTURE2 MANUAL

INTERACTIONS

Shapes of the generated surface

1

Curves Surfaces

1rst metaphor: Shapes

13

Volume section = user hand visual surface2


3 CAVE 3D PAINTING


SCULPTURE2 MANUAL

INTERACTIONS

2nd metaphor: Volumes

2

Movement perpendicularly to the palm’s surface

14

More accurate Same as a tinier brush

Automatic closed surfaces fullfilment


3 CAVE 3D PAINTING


SCULPTURE2 MANUAL

INTERACTIONS

3rd metaphor: Curves

3

Index/Middle finger conformation

15

Thumb positions


3 CAVE 3D PAINTING


SCULPTURE2 MANUAL

INTERACTIONS

Pen-writing metaphor

« Draw »

« Free To Move »

16

Main interests

Simple

Direct metaphor

Natural modes

Hand orientation / Fingers tucked

1

2

3


3 CAVE 3D PAINTING


SCULPTURE2 MANUAL

INTERACTIONS




SCULPTURE3 CAVE 3D PAINTING

CAVE 3D PAINTING PROTOTYPE STAGE 3D PAINTING PROTOTYPE

Immersion in virtual space Co-located interactions Display Accuracy Development Tools

Two users simultaneously max Cost 3mx3mx3m

3D Display in a Large Space Multi viewers Reduced cost

Dancer Blind From His Creations Occlusion Problem One Projection Screen

18

Hand IR Trackers

Glasses with Positions Trackers

Virtual Platformwith AR-Dtrack




SCULPTURE 3 CAVE 3D PAINTING

Instrumented Space

19

isiVR

Multi-display Cluster Position Tracking





Scalable and Interactive Virtual Reality

Immersive 3D Visualization FrameWork

isiVR Features

Immersive ToolsReady-to-useOpenSceneGraph

Render Engine

20

Inputs OutputsBehaviors

Facilitate Additions/Modifications

Immersive ToolsReady-to-useOpenSceneGraph

Render Engine

openSource license





isiVR

21

Flat Polygons

Control Points

Bézier Surfaces according toHand’s Movement

Positions: Middle Finger Distal Phalanx Wrist





Schkölne Paradigm Application





Surface Generation by Hand in a CAVE

http://www.youtube.com/watch?v=5JR3A5KQ-dg






SCULPTURE3 CAVE 3D PAINTING

CAVE 3D PAINTING PROTOTYPE STAGE 3D PAINTING PROTOTYPE

Immersion in virtual space Co-located interactions Display Accuracy Development Tools

Two users simultaneously max Cost Effective 3mx3mx3m

3D Display in a Large Space Multi viewers Reduced cost

Dancer Blind From His Creations Occlusion Problem One Projection Screen



3 CAVE 3D PAINTING 5 PERSPECTIVES1 VIRTUAL

SCULPTURE 4 STAGE 3D PAINTING

Prototype Scheme

Signals received are different format

Instrumented User

25

17 Motion Trackers

Positions x,y,z(t)Orientations

SIGNALS

Gyroscope, magnetometers, and

accelerometers signals

Articulated and Rigid Kinematic Chain

IndividualBody Segments

Mtx Trackers

Xbus Master

Master Slave

XSens





Motion Capture Suit

26

SIGNALS

0 < Flexion < 1

Erase Objects

Point an object

Choose color

Display Palette

Choose color

1

0

2

3

4

56

7

9

10

11 12138

Thumb_NearThumb_FarThumb_IndexIndex_NearIndex_FarIndex_MiddleIndex_NearIndex_FarIndex_RingRing_NearRing FarRing_LittleRing_NearRing_Far

123456789

1011121314

5DT14 Flexion Trackers





Data Gloves




SCULPTURE

Render EngineVisual Programming Editor

Procedural ApproachSDK

Behaviors Indirect Basis

Inputs Management Building Blocks

VIRTOOLS

4 STAGE 3D PAINTING

28

OBJECT

Actions Reactions

Loop Process

Settings

Variable Configuration

Custom Dialog Box

Message Handler

Building Blocks




SCULPTURE

VIRTOOLS

4 STAGE 3D PAINTING

PROCESSING

ENTITY CREATION

BUFFER

COORDINATES Hands

x,y,z(t) vertices

Object Mesh

Lights

x,y,z(t+1) vertices

Material Texture

Camera

Shader

Main Object Sub Object




SCULPTURE

VIRTOOLS

4 STAGE 3D PAINTING




SCULPTURE

Obje

ctGN

Mes

hGN

curv

eA(t)

curv

eB(t)

curv

eC(t)

curv

eD(t)

posA

(t-1)

posB

(t-1)

posC

(t-1)

posD

(t-1)

Mat

eria

lGN

Text

ureG

NOp

acity

Mod

eOp

acity

Aver

age

Mat

eria

lPal

ette

posA

(t)po

sB(t)

posC

(t)po

sD(t)

VIRTOOLS

4 STAGE 3D PAINTING

32

4.Tip

3.Distal

2.Medial

1.Proximal

More Details

Save CPU Resources





Tesselation Post-Process1)

2)

3)

Tesselation





Occlusion SCENE

AUDIENCE

Projection Screen

Constraint

Semi-transparent projection screen (between dancer and

audience)





« La Mer », Debussy - 1905

36


3 CAVE 3D PAINTING

4 STAGE 3D PAINTING

1 VIRTUAL SCULPTURE 5 PERSPECTIVES


Current Work and Applications

Current Work

Credits: Proteigon - 2011

Dancer’s Movement Parameters:

Space Time Weight Flow

“Computational models of expressive gesture in multimedia systems”,

G.Volpe et al, 2003

http://www.youtube.com/watch?v=sd9efB4Ano4

http://www.youtube.com/watch?v=sd9efB4Ano4

37


3 CAVE 3D PAINTING

4 STAGE 3D PAINTING

1 VIRTUAL SCULPTURE 5 PERSPECTIVES


Perspectives

Creations Visual FeedbackCreations Haptic Feedback

38

Authors

Alexis ClayJean-Christophe LombardoNadine CoutureJulien Conan


Bi-manual

An interaction paradigm for augmented reality live

performance

Alexis Clay - Jean-Christophe Lombardo - Nadine Couture - Julien Conan

Human-Computer Interaction, Tourism and Cultural Heritage (ESTIA - INRIA)

Download - Bi Manual 3D Painting - An Interaction Paradigm for Augmented Reality Live Performance - slideshare

Top Related