virtual cinematography theory and practice for automatic real- time camera control and directing...

Virtual CinematographyVirtual CinematographyTheory and Practice for Automatic Real-Theory and Practice for Automatic Real-

Time Camera Control and DirectingTime Camera Control and Directing

Liwei He Liwei He Microsoft ResearchMicrosoft Research

http://research.microsoft.com/users/lhehttp://research.microsoft.com/users/lhe

MotivationMotivation

There are 3 elements in computer graphicsThere are 3 elements in computer graphics– lights, scene objects, andlights, scene objects, and

– the camera (focus of this talk)the camera (focus of this talk) Camera control is hard (7 DOFs)Camera control is hard (7 DOFs)

– position (3), direction (3), field of view (1)position (3), direction (3), field of view (1) We may learn from cinematographyWe may learn from cinematography

Roles in cinematographyRoles in cinematography

Low-level responsibilitiesLow-level responsibilities– CameramanCameraman

• Position and move the cameraPosition and move the camera

– Film editorFilm editor• Keep the film to proper lengthKeep the film to proper length• Decide shot transitionsDecide shot transitions

High-level responsibilitiesHigh-level responsibilities– Script writer and directorScript writer and director

• Story-tellingStory-telling

Camera control level 1Camera control level 1

Input: camera position and directionInput: camera position and direction Output: camera transformationOutput: camera transformation Application: low-level graphics library Application: low-level graphics library

routines (Direct3D)routines (Direct3D)

Camera control level 2Camera control level 2

Input: show both A and B, follow C, etc.Input: show both A and B, follow C, etc. Output: camera position and directionOutput: camera position and direction Application: 3D game, computer Application: 3D game, computer

animation packageanimation package

Camera control level 3Camera control level 3

Input: high-level user directionsInput: high-level user directions– show a conversation show a conversation

– show a car chase sceneshow a car chase scene Output: a sequence of level 2 camera specOutput: a sequence of level 2 camera spec Applications: graphical chat, tele-Applications: graphical chat, tele-

conferencing, Virtual Reality gamesconferencing, Virtual Reality games

Camera control level 3 (cont.)Camera control level 3 (cont.)

Encodes cinematographic expertiseEncodes cinematographic expertise– an interesting problem in itselfan interesting problem in itself

Provides an interface that isProvides an interface that is– real-timereal-time

– visually entertainingvisually entertaining

– more informativemore informative

Eyepeep BARG07011 Feldegast

Microsoft V-Chat Lunar Island

Eyepeep says "really"Feldegast says "yup"BARG07011 has joined the conversationEyepeep says "im still older"Feldegast says "lol"DGeste309 does something silly.

BARG07011DGerste309EyepeepFeldegastGrenDayGod

Principles of CinematographyPrinciples of Cinematography

Film structureFilm structure

FilmFilm

......

......

ScenesScenes

ShotsShots

Camera distanceCamera distance

Close upClose up Close shotClose shot Medium shotMedium shot

Full shotFull shot Long shotLong shot

The line of interestThe line of interest

A B

External cameraExternal camera

A B

External cameraExternal camera

A B

Internal cameraInternal camera

A B

Apex cameraApex camera

A B

Moving camerasMoving cameras

A B

pan(A) track(B)

Some rules in film editingSome rules in film editing

Don't cross the line of interestDon't cross the line of interest Avoid jump cuts Let the actor lead Break movement

Don’t cross the line of interestDon’t cross the line of interest

1

2

Some rules in film editingSome rules in film editing

Don't cross the line of interest Avoid jump cutsAvoid jump cuts Let the actor lead Break movement

Avoid jump cutAvoid jump cut

Some rules in film editingSome rules in film editing

Don't cross the line of interest Avoid jump cutsAvoid jump cuts Let the actor lead Break movement

Two-person conversation Two-person conversation

A B

11 22

33 44

2

3

1

A BA B

A

4 B

Three-person conversationThree-person conversation

22

44

33

11

B

A

C

43

2

A B

C

A B

1

A B

C

Bad

Good

x

y

z

U n iv e rse

Three spacesThree spaces

The universe space

Three spacesThree spaces

The eye space (z-axis is the look at direction)

x

y

z

U n iv e rse

zE y e

x y

Three spacesThree spaces

The screen space (screen is ctan(fov/2) away from eye point)

x

y

z

E y eS c re e n

-1 + 1

+ 1

-1

Standard look-at transformationStandard look-at transformation

Given eye position Pfrom, a look at direction

Want a rotation matrix R and a translation to transform a vector from universe space to eye space

x y

TT

x

y

z

U n iv e rse

P f ro m

E y e

x

y

z

U n iv e rse

P f ro m

E y e

Standard look-at transformationStandard look-at transformation

But this is not uniqueT

xy

x

y

z

U n iv e rse

P f ro m

E y e

Standard look-at transformationStandard look-at transformation

But this is not uniqueT

x y

x

y

z

U n iv e rse

P f ro m

E y e

Standard look-at transformationStandard look-at transformation

But this is not uniqueT

yx

x

y

z

U n iv e rse

P f ro m

E y e

Standard look-at transformationStandard look-at transformation

We will need to specify an up vector , usually [0,1,0]U

U

T

x

y

z

U n iv e rse

P f ro m

E y e

Standard look-at transformationStandard look-at transformation

In eye space

Z-axis is

X-axis is

Y-axis is

Rotation matrix

Translation is -Pfrom

T

ZUV ˆˆˆ

VTU ˆˆˆ

TUVR ˆˆˆ33

UU V

T

Internal close-up of actor AInternal close-up of actor A

x

y

z

U n iv e rse

P f ro m

E y eP a t

Look at [xLook at [xatat,y,yatat] transformation] transformation

U V

TGiven eye position Pfrom, look at point Pat, and Pat in screen space [xat, yat]

Want

In universe space

T

A

)(ˆfromat PPunitA

Look at [xLook at [xatat,y,yatat] transformation] transformation

In screen space:

H = [xat, yat, ctan(fov/2)]

Solve from the following equations: x

y

z

E y eS c re e n

-1 + 1

+ 1

-1

P f ro m

P a t

H

1ˆˆ

ˆˆ]0,1,0[ˆ

ˆˆ]1,0,0[ˆ

UT

UAH

TAH

T

External of actor A and actor BExternal of actor A and actor B

x

y

z

U n iv e rse

P f ro m

E y eP B

P A

Look at [xLook at [xAA,y,yAA] and [x] and [xBB,y,yBB]]

U V

T

Given PA,PB in universe space and eye space, and distance of eye position Pfrom and Pa.

Want Pfrom and T

A

Use numerical method:

1. Set Pfrom to 0

2. Solve R using the Look at [xat,yat] method

3. Transform E by inverse of R, getting a new Pfrom approximation

4. Goto 2

Look at [xLook at [xAA,y,yAA] and [x] and [xBB,y,yBB]]

x

y

z

E y eS c re e n

-1 + 1

+ 1

-1

P f ro m

P B

H

P A

Medium panning shot of actor AMedium panning shot of actor A

ReferencesReferences

Jim Blinn’s CG&A ’88 articleJim Blinn’s CG&A ’88 article– Where am I? What am I looking at?Where am I? What am I looking at?

Steve Drucker’s Interactive 3D ’92, ’94, Steve Drucker’s Interactive 3D ’92, ’94, ‘95 papers‘95 papers– Cast camera positioning as an Cast camera positioning as an

optimization problemoptimization problem

System DesignSystem Design

Overall system diagramOverall system diagram

Real-timeApplication

Virtual Cinematographer

Renderer

At each time stepAt each time step

1. Application generates events to VC1. Application generates events to VC– ( subject, verb, object )( subject, verb, object )

2. VC determines– camera specifications

– acting hints

3. Renderer outputs the image

Real-timeApplication

Virtual Cinematographer

Renderer

At each time stepAt each time step

1. Application generates events to VC– ( subject, verb, object )

2. VC determines2. VC determines– camera specificationscamera specifications

– acting hintsacting hints

3. Renderer outputs the image

Real-timeApplication

Virtual Cinematographer

Renderer

At each time stepAt each time step

1. Application generates events to VC– ( subject, verb, object )

2. VC determines– camera specifications

– acting hints

3. Renderer outputs the image3. Renderer outputs the image

Real-timeApplication

Virtual Cinematographer

Renderer

A networked virtual party gameA networked virtual party game

Actors are simulatedActors are simulated

Actor’s mind

LonelinessLoneliness

ThirstThirst

BoredomBoredom

Walk, converse, look around, Walk, converse, look around, drink, etc.drink, etc.

Can be controlled by the usersCan be controlled by the users

Networked virtual party gameNetworked virtual party game

Server

Client

Client

Client

User actionsUser actions

EventsEvents

Inside the Virtual Inside the Virtual CinematographerCinematographer

VC architectureVC architecture

Idioms

Camera modules

VC architectureVC architecture

Idioms

Camera modules

Camera modulesCamera modules

Geometric placement of specific Geometric placement of specific cameras for each shotcameras for each shot

Choose the side of the line of interest Influence acting

Camera module: Camera module: ext(B,A)ext(B,A)

A B

Static camera modulesStatic camera modules

A B

ext(B,A) ext(A,B)

apex(A,B)

int(B)

Moving camera modulesMoving camera modules

A B

pan(A) track(B)

Camera modulesCamera modules

Geometric placement of specific cameras for each shot

Choose the side of the line of interestChoose the side of the line of interest Influence acting

Don’t cross the line of interestDon’t cross the line of interest

1

2

Camera modulesCamera modules

Geometric placement of specific cameras for each shot

Choose the side of the line of interest Influence actingInfluence acting

Acting hintsActing hints

A BA’A’ B’B’

VC architectureVC architecture

Idioms

Camera modules

Previous worksPrevious works

Dave Christianson’s AAAI ’96 paperDave Christianson’s AAAI ’96 paper– Declarative camera control for automatic Declarative camera control for automatic

cinematographycinematography

– An off-line algorithmAn off-line algorithm

– Need to know all events aheadNeed to know all events ahead

– Use plan algorithms in AIUse plan algorithms in AI

Film idiomsFilm idioms

Capture a particular type of sceneCapture a particular type of scene Register relevant eventsRegister relevant events Select shot typesSelect shot types Determine the transitions between shotsDetermine the transitions between shots

Hierarchical idiom structureHierarchical idiom structure

LookaroundLookaround

MasterMaster

ConverseConverse DrinkingDrinking MovingMoving

3Talk3Talk

2Talk2Talk

Hierarchical idiom structureHierarchical idiom structure

Avoids exponential growth of statesAvoids exponential growth of states Allows expertise to be reusedAllows expertise to be reused Provides robustnessProvides robustness

MovingMoving idiom idiom

track(C)pan(A)

ext(A,B) ext(B,A)

apex(A,B) apex(A,B)

MovingMoving idiom idiom

track(C)pan(A)

ext(A,B) ext(B,A)

apex(A,B) apex(A,B)

T > 8

MovingMoving idiom idiom

track(C)pan(A)

ext(A,B) ext(B,A)

apex(A,B) apex(A,B)

T > 8

MovingMoving idiom idiom

track(C)pan(A)

ext(A,B) ext(B,A)

apex(A,B) apex(A,B)D < 10

MovingMoving idiom idiom

track(C)pan(A)

ext(A,B) ext(B,A)

apex(A,B) apex(A,B)

D < 5

MovingMoving idiom idiom

apex(A,B)

D < 5

apex(A,B)

pan(A)

ext(B,A)

T > 8

ext(A,B)

D < 10

T > 8

3Talk3Talk idiom idiom

int(C)

int(C)

ext1to2

2Talk(A,B)

Invoking a sub-idiomInvoking a sub-idiom

int(C)

int(C)

ext1to2

2Talk(A,B)

ext(B,A)ext(B,A)

ext(A,B)

ext(A,B) 2Talk

ContributionsContributions

Real-time camera controlReal-time camera control– lightweightlightweight

– automatic camera placementautomatic camera placement

– automatic shot transitionsautomatic shot transitions Do a reasonable jobDo a reasonable job

– not to replace human in computer animationnot to replace human in computer animation

Work in ProgressWork in Progress

Camera control for chatCamera control for chat

Camera control for live meetingCamera control for live meeting

Image after warpingImage after warping

Camera control for classroomCamera control for classroom

Multiple cameras per classroomMultiple cameras per classroom– Lecturer camerasLecturer cameras

– Audience cameraAudience camera Use audio and vision techniques toUse audio and vision techniques to

– Track the lecturerTrack the lecturer

– Activate the audience cameraActivate the audience camera

Dramatic lightingDramatic lighting

Virtual CinematographyVirtual CinematographyTheory and Practice for Automatic Real-Theory and Practice for Automatic Real-

Time Camera Control and DirectingTime Camera Control and Directing

Liwei He Liwei He Microsoft ResearchMicrosoft Research

http://research.microsoft.com/users/lhehttp://research.microsoft.com/users/lhe