advanced image based rendering techniques€¦ · advanced image based rendering techniques ... to...

19
Advanced Image Based Rendering Techniques Herbert Grasberger * Institute of Computergraphics & Algorithms, TU Vienna, Austria June 13, 2006 Abstract Image Based Rendering is a rendering approach based on light rays. No geometry has to be known to construct images of a certain object, only many reference images have to be available. This Document covers some advanced techniques of Image Based Rendering, like Plenoptic Modeling, where the Plenoptic Function is simplified in a certain way. Another advanced technique is Hardware Lumigraph Rendering where the texture mapping qualities of current 3D Accelerator Hardware are used to reconstruct a Lumigraph. The last technique is the Delta Tree, a storage efficient representation of reference images. The Document finishes with some applications of Image Based Rendering in film and animation. 1 Introduction Traditional geometry based rendering approaches rely on a model of the geometry to be drawn in some way. This is often a problem when an object in the real world needs to be modeled. To create a model for image based rendering, we just need a set of pho- tos from the object, captured at special positions around it. The resulting data still has to be processed for space and rendering speed reasons, but the model is already complete at that point. The initial techniques were pub- lished in 1996: Lightfield Rendering [Marc Levoy ’96] and The Lumigraph [Steveb J. Gortler ’96]. The techniques described in this paper are based on those two approaches to Image Based Rendering. 2 Plenoptic Modeling Plenoptic Modeling is an Image Based Rendering method based on sampling, reconstructing and resampling the Plenoptic Function. The method was first proposed in [McMillan & Bishop ’95]. * [email protected], 532, 0325080 1

Upload: others

Post on 30-Apr-2020

22 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

Advanced Image Based Rendering Techniques

Herbert Grasberger∗

Institute of Computergraphics & Algorithms, TU Vienna, Austria

June 13, 2006

Abstract

Image Based Rendering is a rendering approach based on light rays.No geometry has to be known to construct images of a certain object,only many reference images have to be available. This Document coverssome advanced techniques of Image Based Rendering, like PlenopticModeling, where the Plenoptic Function is simplified in a certain way.Another advanced technique is Hardware Lumigraph Rendering wherethe texture mapping qualities of current 3D Accelerator Hardware areused to reconstruct a Lumigraph. The last technique is the Delta Tree,a storage efficient representation of reference images. The Documentfinishes with some applications of Image Based Rendering in film andanimation.

1 Introduction

Traditional geometry based rendering approaches rely on a model of thegeometry to be drawn in some way. This is often a problem when an objectin the real world needs to be modeled.To create a model for image based rendering, we just need a set of pho-tos from the object, captured at special positions around it. The resultingdata still has to be processed for space and rendering speed reasons, but themodel is already complete at that point. The initial techniques were pub-lished in 1996: Lightfield Rendering [Marc Levoy ’96] and The Lumigraph[Steveb J. Gortler ’96]. The techniques described in this paper are based onthose two approaches to Image Based Rendering.

2 Plenoptic Modeling

Plenoptic Modeling is an Image Based Rendering method based on sampling,reconstructing and resampling the Plenoptic Function. The method was firstproposed in [McMillan & Bishop ’95].

[email protected], 532, 0325080

1

Page 2: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

2.1 Plenoptic Function

The Plenoptic function is a parameterized function which describes the ra-diance of light from any given direction (θ, φ) at any given point of view(Vx, Vy, Vz) at any time (t) at any wavelength (λ).

p = P (θ, φ, λ, Vx, Vy, Vz, t) (1)

Figure 1: Visualisation of the Plenoptic Function [McMillan & Bishop ’95]

A complete sample of the plenoptic function is a full spherical map for agiven viewpoint and time value.In Plenoptic Modeling a simplified version of the Plenoptic Function is used.Whereas the original Plenoptic Function includes a time parameter and allwavelengths of light, the version used in Plenoptic Modeling is timeinvariantand is reduced to three wavelengths which are 560nm (red), 530nm (green)and 430nm (blue).

2.2 Plenoptic Sample Representation

One problem is the internal representation of the Plenoptic Function. Thereare several possibilities, like spherical, cubic or a cylindrical representations.The spherical version would be the most natural one, but it lacks a storagesuitable representation. This is due to various disortions which arise whenprojecting a sphere onto a plane.A cubic projection would be better, especially when it comes to disortions,

2

Page 3: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

but on the corners of the cube, it is overdefined.The cylindrical projection was chosen because it can easily be unrolled ontoa plane. The cylinder has no endcaps, which limits the vertical field of view.

The advantage of the cylindrical projection is the simplicity of aquisi-tion. Only a video camera and a tripod is required. Two captured planarprojections of such a scene are related by a certain transformation. u

vw

=

a11 a12 a13

a21 a22 a23

a31 a32 a33

xy1

(2)

x′ =u

wy′ =

v

w

In this formula, x and y represent the pixel coordinates of an image I andx′ and y′ the corresponding pixel in an Image I ′.

With the restriction, that the pictures are aquired in a panning move-ment, the focal length and the position of the focus point are the same ineach image.As a result two images differ only by a rotation in the vertical direction.In order to project two or more pictures onto the cylindrical projectionthe most common technique uses four common points between two images.With this 4 common points a mapping can be computed. It is possible withthis method to map the Nth image onto image N − 1. In this approach itis not necessary to determine an entire camera model.This set of transformations Hi can be split up into an intrinsic transfor-mation Si which is only determined by camera properties and the extrinsictransformation Ri. The extrinsic transformation is determined by the rota-tion around the camera’s center of projection.

u = Hix = S1RiSx (3)

Due to this decomposition the projection and the rotation components ofthe transformation are decoupled. The intrinsic part is invariant over allof the images, which enables splitting up the problem of the cylindricalprojection into the determination of the extrinsic rotation Ri followed bythe determination of the intrinsic projection component Si.Due to the invariance of the intrinsic part the whole mapping between theith and the jth image of the series can be written as:

I ′i = S−1Ry+1Ry+2Ry+3 . . . RyIj (4)

By modifying S the images can be reprojected onto arbitrary surfaces. Dueto this fact, all of the acquired images can be mapped onto a cylinder easily.

3

Page 4: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

2.3 Determining Image Flow Fields

It is possible to use more than one cylinder for reconstructing a certainviewpoint. If you only use one cylinder the result is simlar to QuickTimeVR, which means that no real motion is possible, only zooming is done here.With more than one cylinder it is possible to achive a depth impressionwhen doing motion. In order to work with more than one cylinder, it isnecessary to estimate the relative positions of the cylinders and to computethe depth information through disparity images.

In order to compute the relative position between two cylinders, corre-sponding points have to be found and marked manually. The relativepositions of center of projections can be determined automatically, but theycan only be computed to a scale factor. This is due to the fact that it isnot possible to determine from a set of pictures, if the observer is lookingat an object or a whole scene. This is why corresponding points have to bemanually specified.If the same camera properties were used for the two cylinders, the unknowntransformation between these two cylinders can be reduced to a translationand rotation. The parameters have to be estimated by optimizing the raysbetween the corresponding points, where the value to optimize is the sumof distances between the rays. When this is done, only one parameter of thetransformation matrices is missing. The missing parameter is the distancebetween the eye points of the cylinders, which must be estimated physically.

Epipolar methods enable us to identify two equal points on different cylin-ders and compute a sinusoid like curve between those two points. Thisenables us to establish image flow fields.There are several methods to compute such a flow field:

• correlation methods

• relaxation methods

• method of differences

2.4 Plenoptic Function Reconstruction

The core of an Image Based Rendering algorithm is the computation of newviewpoints. With Plenoptic Modeling computing a new viewpoint, which isrepresented by another cylinder, out of two cylinders A,B and the disparityimage is done with an algorithm that is also able to describe occlusion andperspective effects.In figure 2 the point P in the scene has the angle θ concerning the cylinderA. With consideration of the disparty disp(A,B, p) = α between A and B,the point P has the angle θ + α concerning B. As a result, it is possible

4

Page 5: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

Figure 2: Reconstrucing a new Cylinder [McMillan & Bishop ’95]

to compute with the positions of A and B and the angles the position ofP on the new cylinder V and the angle θ + β, where β = disp(A, V, p).[McMillan & Bishop ’95]Actually the P isn’t computed exactly, but only the angles and positions areused to compute V . Due to some transformation techniques it is possibleto compute the transformation interactively. It is also possible to do theprojection onto a plane in the same computation step.

One problem of this algorithm can be, that more than one pixel of theold cylinders might be projected onto the same pixel in the new cylinder,but it is possible to use the painter’s algorithm in order to draw the rightone of the old pixels. In this algorithm the whole scenery is painted back tofront, which can be done even if the object geometry is not known.This can be established by projecting the eye’s position onto the cylinder’s

surface and dividing it into four toroidal sheets. In order to achive the rightprojection the sheets with more distance to the eye point have to be drawnfirst (see figure 3).

2.5 Conclusion

With Plenoptic Modeling it is possible to capture and to reconstruct thewhole scenery with ordinary hardware. The only restriction of the viewingangle is because of the missing endcaps of the cylinder.It shares the same problems with other Image Based Rendering techniques,like the problem that the quality of the images depends on the original

5

Page 6: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

Figure 3: Image flow field [McMillan & Bishop ’95]

footage.

3 Hardware Lumigraph Rendering

The most important Image Based Rendering methods Lumigraph[Steveb J. Gortler ’96] and Light Field Rendering [Marc Levoy ’96] needmany resources. It is possible to improve the performance of the Lumi-graph using current 3D hardware [M.F.Cohen ’97].The Lumigraph is based on the Plenoptic Function, which is reduced to a4D function. The principle of the Lumigraph is intersecting a ray with a pairof planes. The ordinary Lumigraph uses more than one pair, to illustratethe possibilites of hardware optimization, only one pair will be used in thisexample.Due to the finite context of the computer the Lumigraph has to be dis-cretized. It hast been found out, that 32x32 nodes on the (s,t) plane and256x256 nodes on the (u,v) plane give good results1

3.1 Reconstruction using Texture Mapping

In the reconstruction process with 3D hardware, the planes are used as tex-tures, because interpolation would need too much computing power com-pared to the texture mapping approach. The whole (u,v) plane is used as atexture and as a result triangles instead of pixel are drawn. Interpolation isdone with blending.Each vertex of a (s,t) triangle has a corresponding triangle in the (u,v) plane,which are drawn 3 times with different alpha values. If the alpha values are

1Because those values were published in ’97, it can be assumed that today the valuescan be leveled up, due to more computing power; as for consistency I will use the valuesof ’97

6

Page 7: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

1.0 at the corresponding triangle and 0.0 at the two other ones, the originalimage is reconstructed.The texturecoordinates of the (u,v) texture are computed through intersect-ing the rays starting at the camera.In order to draw the whole image, each triangle of the (s,t) texture has tobe drawn 3 times, which can be achieved with current 3D hardware withoutany problems.The bottleneck of this method is the memory. When a full Lumigraph with

Figure 4: Reconstruction as Alpha Blended Texture Triangles[M.F.Cohen ’97]

6 planes and RGB color values is used, more than one gigabyte memory isused2. There are several methods to optimize this approach.

3.2 Reducing Data

In general dynamic adjusting the tesselation is used to compress the (s,t)plane. The limited set of (s,t) nodes enables to draw fewer polygons.

2256x256x32x32x6x3 bytes = 1.125GB; due to the fact, that the bottleneck of all IBRmethods is the texturesize and quality, higher texture sizes are recommended and as aresult the memory consumption will exceed the sizes of current 3D accelerators memory

7

Page 8: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

3.2.1 Subsampling

The easiest way of compressing the (s,t) plane is simply the reduction ofthe resolution by a factor of 2. This reduces the size by 4, but the achievedimage is going to look blurred, due to the alpha blending described above.To compensate the loss of picture quality, it is possible to use mip-mappingat the (s,t) plane.

3.2.2 Fixed pattern of (s,t) nodes

Most images have their most important information in the center, becausethe viewer concentrates on this area. As a result, it is possible to use onlythe nodes in the center of the (s,t) plane, e.g. it is possible to pick onlythe 9 most centered nodes and to reconstruct the image with these nodes.These 9 nodes are blended with the alpha values like above (1,0,0). In orderto complete the picture fictitious nodes are assumed in the rest of the (s,t)plane. These nodes are only drawn two times, because no texture is assignedhere.See figure 5 for an example. Triangle I is drawn first using the (u,v) plane

Figure 5: Reconstruction using the 9 centered nodes [M.F.Cohen ’97]

associated with node (a) and with alphas of 1.0 at node (a), 0.0 at node (b)and 0.5 at the fictitious node labeled (b,c). A similar pattern is used fromthe point of view of node (b). Similarly triangle II is drawn with texture (a)with alphas (1.0 , 1.0, 0.5) an with texture (b) with alphas (0.0, 0.0, 0.5)

8

Page 9: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

(Example out of [M.F.Cohen ’97]) This small amount of nodes can even beused for small motion, but if the motion gets too big, the nodes have to bechanged.

3.2.3 Nodes along a line

In some cases the user’s motion can be predicted, especially if the eyepointis moving along a line. In this case, the (s,t) nodes can be aligned along anappropriate line.As a result the (s,t) plane is not divided into triangles, but into strips, whichspan the space between adjacent nodes along the line. Here, each strip isdrawn twice, the alphas are set to 1.0 along the line at the current node an0.0 at the other ones.

3.2.4 Using Projective Textures

This method uses the current image as a texture map for nearby imagesthat have to be reconstructed. If the approximate geometry is known, thistexture is wrapped around the viewpoint.In this approach a matrix is constructed after the initial image, which isthen used to project the texture onto the next frame. This is done in ananimation, where the moving vector of the eyepoint is known, and as a resultthe appropriate transformation matrix can be computed. In this case everysecond frame of the animation might be computed with the exact image andthe transformation.Unfortunately an interpolation for an animation where the angle or the po-sition of the eyepoint differ very much is not possible because artefacts canoccur in this case.An appropriate algorithm might use this technique of interpolation whilecomputing a new exact image in the background and if this image is com-pleted, it is put into the animation and interpolated, until the next exactimage is ready.

3.2.5 Progressive Refinement

The methods above might be combined with a method of progressive refine-ment. Only the nodes of the (s,t) plane that are necessary in the currentframe are used, the rest is stored in a database or something similar. If thealgorithm detects missing and unnecessary nodes that are currently in use,it can delete the unnecessary ones and load the missing ones.In combination with the right alpha values while drawing the animation,the user should not be aware of the node streaming and the resulting imageshould look as if it was reconstructed nearly from the whole (s,t) plane.

9

Page 10: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

3.3 Conclusion

The original approach [Steveb J. Gortler ’96] where the whole (s,t) planeis used for reconstruction would be too memory consumptive, but with themodifications described above it is possible to get very good results, becausehigh resolution image can be used too.

4 The Delta Tree

The Delta Tree [Dally et al. ’96] is a data structure that represents an ob-ject using reference images. An arbitrary reprojection of an object can beachieved by traversing the Delta Tree.Each node in the tree stores an image taken from a certain point on asampling sphere. The main goal is, to store each surface pixel of the objectonly once in order to minimize storage because storage is the main bot-tleneck of all Image Based Rendering approaches. Each image is stored inthe frequency domain. The reason for storing the images in the frequencydomain is the fact, that reprojecting an image via the Delta Tree is similarto an image warp where it is also possible to handle level of detail andantialiasing in one step.

4.1 Motivation

With Plenoptic Modeling it is only possible to get an interior view of a singlescene, although it is possible to move in this scene. It is not possible to setup an exterior view of an object and move around. This can be achievedwhen using the Delta Tree.The Delta Tree is an object centered Image Based Rendering approach,where a certain amount of reference images are stored in the tree and theobject viewed from a certain viewpoint can be reconstructed during travers-ing the tree.It is possible to choose any viewpoint, as long as the viewpoint is on thehull of the sampling sphere. If the point lies the hull, the object has to bedivided into several objects.These objects can be combined again by combing the tree. The same methodis used to set up complex scenes.

4.2 Definitions

A sampling sphere is a sphere that surrounds the object completely. Thesurface is split up into a regular grid of sampling points.A pixel stores the object properties on a certain point. The information canbe stored in a disparity map. In the current version, only color values are

10

Page 11: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

stored, information about transparency, specularity and gloss may be usedin the future too.A view Vp is a photometric representation of the object stored in the DeltaTree seen from position P with a specified field of view. A subview is arectangular subset of the view, which might be reduced in height an width.[Dally et al. ’96]

4.3 Sampling

The reference images captured during sampling can be achieved from realworld objects, but it is also possible to use computer generated ones.As said before, the reference images have to be captured from a samplingsphere, where every position can be described by a pair of angles (θ, ψ). Itis not necessary to know the radius of the sphere. The surface of the sphereis split up into a regular grid of reference points, the density of these pointsis proportional to the bumpyness of the object. If it is more bumpy, thenthe density of the sampling points has to be bigger, in order to cover all ofthe surface features and to store each surface pixel of the object.

4.4 Structure

To describe the tree structure, it is necessary to describe the surface of thesampling sphere. The sampling points can be unrolled into a rectangularplane (see figure 6 ), where the height and width represent the two differentangles (θ, π) from 0 to 45 degrees. Every black point on this plane cor-responds to a camera position on the sampling sphere, where a referenceimage was captured. On the right side of figure 6 a Delta Tree which hasthe basic structure of a Quad tree is shown. Each node stores one referenceimage and a certain region of the sampling sphere is attached to it.

Figure 6: Comparison Delta Tree and viewing space [Dally et al. ’96]

E.g. the root node in figure 6 stores the reference view at (0,0) and the

11

Page 12: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

whole region from (0,0) to (45,45) is attached to the node. Each child nodestores the reference view seen from one region corner. This is done until theleafs of the tree are reached, which store the pixel. In the left square theregions stored in the leaves are shown via grey values.A complete Delta Tree has got 4P root nodes, each root node correspondingeach quadrant. Typically P = 6 according to the six cardinal directions.

4.5 Redundancy

In order to reduce the amount of pixel, every node only saves the pixelswhich are not yet stored by the ancestor node. As a result, each node apartfrom the root node only stores a certain part of the reference image. Someof the nodes are even empty, because the ancestor stores the whole referenceview. In this case only a pointer to the reference view is stored in the node(see node B0. C00, ... in figure 6 ).Sometimes more than one node needs certain pixels. In this case the pixelsare stored in the memory once, and all of the nodes point to this storagearea. This occurs in certain conditions:

• The four trees corresponding one quadrant share views along theircommon edges. The views are shared by reference in this case.

• one child of each node shares a view with its ancestor. E.g. B0 sharesa view with A. In this case B0 doesn’t need to store any information,because all is already sampled by its ancestor A. In the figure, suchviews are shown cross hatched.

• first cousins within a tree may share a view, e.g. C11 shares a view Vp1

with C23 and even four cousins shares Vp2 (C02, C12, C22.C32) in thiscase the cousins store different partial views, which means for examplethat C11 stores only those pixels that are not adequately sampled byA or B1. Any pixels that are common, are only shared by reference

• Reference views on the border of the root region are shared betweentwo Delta Trees

It can be seen in the figure, that the nodes that are down the tree store lessinformation than the nodes in the upper region

4.6 Resampling

Reconstructing a random view, located between some of the sample views onthe sampling sphere is simply done by traversing the tree. Due to the layoutof the Delta Tree this traversion is done in a spiral shape when looked at theplanar projection of the sampling sphere. For the reconstruction the leafssurrounding the viewpoint P are visited, travsersed into the viewpoint via a

12

Page 13: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

warp operation and the image is reconstructed using a z-buffer. The z-bufferis required to eliminate visible surfaces hidden in the path. In figure 7 this istraversion is illiustrated. The reason for visiting the surrounding leafs is thatthe new view is reconstructed with the reference views. In the situation that

Figure 7: Traversing the tree [Dally et al. ’96]

the view is off the sampling sphere, the reconstruction process gets a little bitmore complicated. In this situation it is necessary to recursively subdividethe generated view while traversing the Delta Tree. The subdivision isperformed by projecting the region boundary on the sampling sphere ontothe view. The projected boundary divides the pixels into two subviews whichmay then have to be divided another time, depending on it straddling someboundary edges.In figure 8 this is illustrated. Here the part V1 of the image is reconstructedusing R1 and V2 is reconstructed using R2. The computation of the final

Figure 8: Subdividing the view [Dally et al. ’96]

image is done with Plenoptic Modeling of planes.

13

Page 14: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

4.7 Level of Detail

Different levels of detail can be achieved very easily. Because all sampleimages are stored in frequency domain, it is possible to reconstruct thewhole image with low-pass filtering. To optimize level-of detail, the imagesmight be stored as mip maps during the construction of the Delta Tree. Inthis case it is possible to immediately choose the right level of detail for theresampling process.In the worst case, this approach requires reprojecting four times as manypixels as appear on the screen, but no level of detail hierarchy with multiplerepresentation is required.

4.8 Conclusion

The current version of the Delta Tree lacks information about specularityand transparency, but this can be easily implemented in future versions.One problem it shares with many Image Based Rendering approached isthe fact that an object can only be reconstructed at a certain viewpoint, ifreference views in the region of the eypoint exist.

5 IBR in Applications & Film

Image Based Rendering techniques are used in many ways when it comes todifferent applications and film. For each of these techniques the followingquestions can be asked:

• What is the model and how are the images used?

• What is the rendering method?

• What is the effect and is what we see really there?

5.1 Golden Gate Movie Map

Movie Maps [Naimark ’97] were first made in the late 1970s by the MIT.Basically a movie map is based on filmed and on panoramic images. Thenthese material is processed to a video disc where the filmed material is usedfor the motion parts of the interactive environment and the panoramic im-ages are used at certain points.In 1987 the Golden Gate Videodisc Exhibit was produced as an aerialmoviemap. A helicopter flew a 1 by 1 mile grid of the Bay Area at aconstant ground speed. The grid was determined by satellite navigation,and as a result the helicopter filmed every 30 feet (figure 9). The 35mmcamera was always pointed at the center of the Golden Gate Bridge, and asa result no turn sequences were needed in the movie map. All in all a 10 by10mile area was captured and put on a laserdisc.

14

Page 15: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

The playback system used a trackball to control speed and direction, and asa result it was possible to freely move over the Bay Area at higher speedsthan normal.

Figure 9: Schematic view of the Bay Area [Devbec ’99]

5.2 Matte Painting

Matte Painting is an old technique used since the feature film Gone Withthe Wind(1939). The principle is that instead of building a set, it can bedrawn, mostly digitally in these days. Then the actor, or the foregroundimage can be composed into the matte background. As a result it is possibleto show the actor in some large space without really building that space.Some films even used panning and zooming of the matte painting, whichgives the illusion of motion.These days the whole principle matte painting technique is used in many

films, where the scenes are composed of real footage and computer gener-ated backgrounds, or sometimes in full computer generated movies. In thelast case the technique helps to reduce rendering time. E.g. in the filmFinal Fantasy everything was done with computer graphics. In many scenesonly the actors and some items are actually moving, not the background.In such a case the background was rendered once, and then composed withthe moving characters (which were also rendered seperately). This makeschanging the scene also easier. If there are some parts of the background,that change, only these parts are rendered alone and composed into thescene (even if it is only smoke).The difference between computer generated matte painting and the realmodel can be demonstrated at the famous Death Star Scene from Star WarsEpisode IV: A new Hope. This movie was filmed in the 1970’s and here

15

Page 16: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

Figure 10: Comparison between cg an matte painting. Images courtesy ofLucas Digital, Lucasfilm, and Industrial Light and Magic [Devbec ’99]

a model of the Death Star was used for some footage. When ILM did thespecial edition of this movie in the late 1990s they replaced the footage ofthe model with computer generated material (figure 10). Figure 11 shows a

Figure 11: Matte Painting scene out of Return of the Jedi[Star Wars Directions]

scene out of the original Return of the Jedi version, where the actors wereblended into the matte background. To hide the blending artefacts, smokewas inserted afterwards. Below the picture shows the composition from theside.

16

Page 17: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

5.3 SFX in Matrix

Figure 12: Setup of the cameras used for Bullet Time [wikipedia.com]

The famous Bullet Time effect from the film Matrix was created with atechnique called Virtual Camera. This system contains a set of still cameraswhich surround a certain scene and shoot the scene at the same time. As aresult it is possible to achive an effect which looks as if the time has stopped.It is possible to chance the view by choosing another camera and warpingthe current image to the image of the next camera. In Matrix the cameraswere aligned along a line (see the black points in figure 12) and on the endof each line a conventional motion camera is placed.With this setup it is possible to fly around the object while it is moving.

5.4 IBR in Character Animation

Figure 13: Video Model [Devbec Course ’99]

Another application of Image Based Rendering is character animation

17

Page 18: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

[Devbec Course ’99]. Here, not only a large amount of images has to be cap-tured, the images have to be classified and tracked too. After this is done itis possible to estimate the motion of the mouth and eyes where, as a resultthe spoken vocal and facial expression can be detected. This informationcan be stored and in a further step a video model (figure 13) can be built outof some phonetic footage, the head pose and the mouth shape. This videomodel can be used to produce a new 2D animation of a speaking person.Another application concerning character animation is the tracking of mo-tion of a moving character and classification by the movement. This in-formation can then be used reconstruct a complete moving and speakingcharacter out of reference images.

5.5 Conclusion

Image Based Rendering in applications and in film have a long history. Somethings are not real Image Based Rendering methods (e.g. matte painting)but they are connected to this technique in some certain way. These tech-niques enable many new effects and have become standard tools in filmmak-ing.

References

[Marc Levoy ’96] Pat Hanrahan, Marc LevoyLight Field Rendering. Computer Graphics Proceedings, Annual Conference Series(Proc. SIGGRAPH ’96), pages 31-42, 1996

[Steveb J. Gortler ’96] Michael F. Cohen, Steveb J. Gortler, Radek Grzesczuk, RichardSzeliskiThe Lumigraph. Computer Graphics Proceedings, An- nual Conference Series (Proc.SIGGRAPH ’96), pages 43-54, 1996

[McMillan & Bishop ’95] Leonard McMillan, Gary Bishop.Plenoptic Modeling. Computer Graphics Proceedings, Annual Conference Series(Proc. SIGGRAPH ’95), pages 39-46, 1995

[M.F.Cohen ’97] S.J.Gortler, P.Sloan, M.F.CohenTime-Critical Lumigraph Rendering.Computer Graphics Proceedings, Annual Con-ference Series (Proc. SIGGRAPH ’97), pages 17-23, 1997

[Dally et al. ’96] William J. Dally, Leonard McMillan, Gary Bishop, Henry FuchsThe Delta Tree: An Object-Centered Approach to Image-Based Rendering. MIT AILab Technical Memo 1604, May 1996

[Devbec ’99] Paul DevbecApplications of IBMR in Art and Cinema SIGGRAPH Course, 1999.

[Naimark ’97] Michael NaimarkA 3D Moviemap and a 3D Panorama SPIE Proceedings, Vol. 3012, San Jose, 1997

[wikipedia.com] http://en.wikipedia.org/wiki/, Wikipedia, Virtual Camera 31st May2006

18

Page 19: Advanced Image Based Rendering Techniques€¦ · Advanced Image Based Rendering Techniques ... to this fact, all of the acquired images can be mapped onto a cylinder easily. 3. 2.3

[Star Wars Directions] http://www.agraphafx.com/StarNet/Newstuff/cgi.htmlStar Wars Digital Directions, 31st May 2006

[Devbec Course ’99] Paul DevbecIBMR Techniques for Animating People SIGGRAPH Course, 1999

19