computer graphics inf4/msc computer graphics lecture notes #16 image-based modelling, rendering and...

Computer Graphics Inf4/MSc

Computer Graphics

Lecture Notes #16

Image-Based Modelling, Rendering

and Lighting


13/11/2007 Lecture Notes #16 2

Global Illumination and Image-Based Lighting

• Traditional Computer Graphics involves:– Modelling with matter: geometry with reflectance

properties.

• Image-Based Modelling & Rendering is:– Modelling and rendering with Light, often deriving

geometry and materials in the process.

• Image-Based Lighting allows:– Combination of real and synthetic graphics with

consistent illumination, using images as light sources.



Pursuit of Photorealism

• For complex models, tools can require lot of effort and rendering is very slow.

• Choice: do usual modelling or just take a photo (that would provide photorealism!)

• IBMR is about using these photos and transcending their limitations (eg. Light).

• Don’t need anything like a complete model, but may need a number of photographs…



Pursuit of Photorealism

• Need enough photographs to give coverage.

• Major result is - rendering is faster.

• So Image-Based Modelling and Rendering (IBMR) starts with the answer!

• Research inspired by film industry.

• Related to techniques in Computer Vision.

• New so no taxonomy, just techniques.



IBMR Techniques

• Panoramas – movement within and between panoramic cylinders, eg. Quicktime VR.– Panoramas can be real or synthetic.

• View interpolation –from a few photos, interpolate view from any position (morphing).

• * Lighting – adding real or synthetic objects to a photographed scene, illuminated with scene lights.

• Modelling – extracting models from photos.



“Traditional graphics”



Computer Vision



Vision meets graphics



However ...

• Vision falls short


13/11/2007 Lecture Notes #16 10

And ...

… so does graphics


13/11/2007 Lecture Notes #16 11

Image-Based Rendering (IBR)


13/11/2007 Lecture Notes #16 12

What is an image?

• Collection of radiance values

• radiance along a ray

• 3D position

• 2D direction


13/11/2007 Lecture Notes #16 13

Plenoptic function• Radiance value for all

possible rays = plenoptic function.

• All possible images are a subset of this plenoptic function.

• Too much stuff!• Goal of IBR is generate a

continuous representation of the plenoptic function.


13/11/2007 Lecture Notes #16 14

Plenoptic function

• Radiance is constant along a ray (line)

• 2D position

• 2D direction

Once we know one ‘origin’, we know them all


13/11/2007 Lecture Notes #16 15

What is an image?• Image = rays going through one point

• usually restricted to viewing frustum, but can also be panoramic


13/11/2007 Lecture Notes #16 16

What is an object?• Image = rays going through one point + image

plane

• 2D function (position on image plane)


13/11/2007 Lecture Notes #16 17

What is an object?• Outgoing radiance field of an object

• 2D function (position on surface)


13/11/2007 Lecture Notes #16 18

What is an object?• All light leaving the object


13/11/2007 Lecture Notes #16 19

What is an object?• All light leaving the object

• 4D function (2D position + 2D direction)


13/11/2007 Lecture Notes #16 20

What is an object?• All possible images of an object


13/11/2007 Lecture Notes #16 21

What is an object?• All possible images of an object


13/11/2007 Lecture Notes #16 22

What is an object?• We don’t really need the object


13/11/2007 Lecture Notes #16 23

What is an object?• We don’t really need the object


13/11/2007 Lecture Notes #16 24

Lumigraph / Light Field• Object is only defined by its radiance field

stuff4D function

(Levoy - Cohen et al 96)


13/11/2007 Lecture Notes #16 25

Lumigraph - capture

(Stanford - Levoy et al.)


13/11/2007 Lecture Notes #16 26

Layered Depth Images

(McMillan)

Problem with one photo is “holes” when view point moved:

- this is why a number of photos are generally required

- fill the holes using intensities of neighbouring pixels

- interpolation …


13/11/2007 Lecture Notes #16 27

Image Based Lighting

• Add models or objects to scenes and allow them to be manipulated in the scenes.

• Modelling with light allows added objects to be illuminated consistent with image existent lighting - photorealistic inclusions.

• We start with the answer by finding the scene illumination.


13/11/2007 Lecture Notes #16 28


• Real Scene

• Goal: place synthetic objects on table

(all pictures P. Debevec 98-99)


13/11/2007 Lecture Notes #16 29

Extracting scene lighting

• Capture illumination using illumination sphere


13/11/2007 Lecture Notes #16 30


Real scene


13/11/2007 Lecture Notes #16 31

Image Based Lightingcaptured illumination field


13/11/2007 Lecture Notes #16 32


Real scene

light based model

local scene

synthetic objects


13/11/2007 Lecture Notes #16 33

Image Based Lighting• Use renderer - compute effects of synthetic

objects on local scene

local scene (brdf estimated)synthetic objects (brdf known)

light based model


13/11/2007 Lecture Notes #16 34


• Render into the scene

background


13/11/2007 Lecture Notes #16 35


• Render synthetic objects


13/11/2007 Lecture Notes #16 36

Image Based Lighting• Effect of local scene on real scene


13/11/2007 Lecture Notes #16 37

Image Based Lighting• Add differences to image


13/11/2007 Lecture Notes #16 38

Reconstruction from images?

• Computer Vision related.• Constructs new views and extracts models.• Epipolar geometry

– expresses relationships between points in different images.

• Difficult to predict full impact in CG.• State of the Art

– 3D reconstruction from uncalibrated images.


13/11/2007 Lecture Notes #16 39

References

• You won’t find this material in the major graphics texts since it is new and not yet mainstream but I recommend the following:– The Computer Image, Watt & Policarpo, Addison-

Wesley 1998.

– SIGGRAPH courses for the past few years, we have some on CD rom and on-line within EdVEC and not immediately available to Informatics machines – ask me if you have a strong interest and need.

computer graphics inf4/msc computer graphics lecture notes #16 image-based modelling, rendering and...

Documents

computer vision slide

traditional graphics

synthetic graphics

lighting slide

image plane slide

panoramic slide

surface slide

short slide