Image Based Rendering With Stable Frame Rates

HuaminQu Ming Wan JiafaQin Arie Kaufman�

Centerfor VisualComputing(CVC) andDepartmentof ComputerScienceStateUniversityof New York at Stony Brook

Abstract

This paperpresentsan efficient keyframelessimage-basedrender-ing technique.An intermediateimageis usedto exploit thecoher-encesamongneighboringframes. The pixels in the intermediateimagearefirst renderedby a ray-castingmethodandthenwarpedto theintermediateimageat thecurrentviewpoint andview direc-tion. Weuseanoffsetbuffer to recordtheprecisepositionsof thesepixelsin theintermediateimage.Every frameis generatedin threesteps: warping the intermediateimageonto the frame, filling inholes,andselectively renderingagroupof ”old” pixels.By dynam-ically adjustingthenumberof those”old” pixelsin thelaststep,theworkloadat every framecanbebalanced.Thepixelsgeneratedbythelasttwostepsmakecontributionsto thenew intermediateimage.Unlike occasionalkeyframesin conventionalimage-basedrender-ing which needto be totally rerendered,intermediateimagesonlyneedto bepartiallyupdatedateveryframe.In thisway, weguaran-teemorestableframeratesandmoreuniform imagequalities.Theintermediateimagecanbewarpedefficiently by a modifiedincre-mental3D warpalgorithm. As a specificapplication,we demon-strateour techniquewith a voxel-basedterrainrenderingsystem.

Keywords: Image-basedrendering,raycasting,voxel-basedmod-eling,terrainrendering.

1 Introduction

Real time renderingin a walk-throughor fly-through systemhasbeena challengein the computergraphicsfield for decades.Thedatasetof thescenemodelis oftenso large thateventhemostad-vancedrenderinghardwarecannotprovide interactive rates. Vari-ousmethodshave beenusedto acceleratethe rendering. A greatdealof previous work hasfocusedon visibility culling and level-of-detailmanagement.

Morerecently, image-basedrendering(IBR) techniquesareusedin walk-throughor fly-through systems. IBR techniquesexploitthe frame-to-framecoherenceby reusingthe previously renderedimages. This generallyinvolves keyframes(or referenceframes)andwarpingprocesses.Key framesareusually renderedby con-ventionalrenderingtechniquesfrom the original datasetandhave�

Department of Computer Science, State University of NewYork at Stony Brook, Stony Brook, NY 11794-4400, USA. Email:�huamin�mwan� jiafa � ari� @cs.sunysb.edu

high quality. The derivedframescanbegeneratedby warpingthekeyframesandfilling in holesif needed.Key framescanbeeithergeneratedin apreprocessingphaseor renderedonthefly. However,in someapplications,keyframescanonly berenderedonthefly. Forexample,in a fly-throughsystem,usersfly over a largeterrainandthefly routeschangeinteractively. It is almostimpracticalto pre-generateall keyframesandusethembecauseof the hugenumberof keyframesneeded.Thus, for suchapplications,keyframesaregenerallyrenderedon thefly.

However, therearesomedisadvantagesto thiskeyframemethod.Generally, thetime to rendera keyframeis higherthanthetime togeneratea derivedframe. This causestheunstableframeratesbe-tweenkeyframesandderivedframes.Theusercanfeel the”pause”or dramaticchangein the framerateduring the fly-through. An-other problemof the keyframe methodis the nonuniformimagequalityamongframes.Key frameshavebetterimagequalitiesthanderivedframes.Thederivedimagesgeneratedat theviewpointnearthekeyframes’viewpointhavebetterqualitiesthantheimagesgen-eratedat theviewpoint far from thekeyframes’viewpoint.

Most of thepreviouswork usesthis keyframeIBR techniqueonsurfacemodels.Whenwe try to usethis keyframetechniquein avoxel-basedscenemodel,problemsbecomemoreserious.Thecon-ventionalrenderingtechniquein a voxel-basedscenemodelis raycasting.Theray-castingmethodcangenerateahigh quality image,but it is time-consuming.Thus,thecostto generatea keyframeonthefly is high, andtheunstableframerateproblembecomesmoreserious.

Oneadvantageof the ray-castingrenderingmethodis that it iseasyto renderany pixel in animageby castingoneray. Theholesin the derived framescan be filled in by the ray-castingmethod.Actually, the ray-castingmethodcaneasily renderany part, evenscatteredpixelsof an image.We try to takeadvantageof this fea-ture in this work. The basicideais to usean imagewarpingtech-niqueto gaina high framerate,andto usethe ray-castingmethodto selectively rerenderpartof theimageto guaranteeimagequality.

Wepresentanefficient image-basedrenderingtechniquewithoutkeyframesin thecontext of voxel-basedscenemodeling.Weuseanintermediateimageto exploit thecoherencein frames.Thepixelsin the intermediateimageareall first generatedby the ray-castingmethodin previousframesandthenwarpedto theintermediateim-ageat the currentviewpoint andview direction. We relax the re-quirementthatall pixelsin animagearelocatedin aregularrectan-gulargrid. We useanoffsetbuffer to recordtheprecisepositionofthesepixelsin theintermediateimage.Every frameis generatedinthreesteps: warpingtheintermediateimage,filling in holesandse-lectively renderingsomeotherpixels. This laststepcanguaranteethe imagequality. By dynamicallyadjustingthe numberof thosepixelsin thelaststep,we canbalancetheworkloadat every frame.Thepixelsgeneratedby thelasttwo stepsmakecontributionsto thenew intermediateimage.

McMillan andBishop’s 3D warpalgorithm[10, 11] canbesuc-cessfullyadaptedto warp the intermediateimage. We show thatthe intermediateimagecan still be warpedin McMillan’s occlu-sioncompatibleorder[9]. By quantizingtheoffsetof pixels in theintermediateimageandusinglookup tables,the advantageof the

incrementalcomputationof McMillan andBishop’s algorithmcanbepreserv� ed.

Theadvantageof our methodover thekeyframemethodis thatthe keyframeneedsto be rerenderedtotally, but the intermediateimage only needsto be updatedpartially at every frame. Bycombiningtheadvantagesof theimage-basedrenderingtechniquesandtheray-castingmethod,we cangetmorestableframeratesandmoreuniform imagequalities.Ourmethodis especiallyusefulin areal time systemwhenthekeyframerenderingis time consuming.We demonstrateour methodwith a voxel-basedterrainrenderingsystem.

This paperis organizedas follows. Section2 reviews the pre-vious work in image-basedrenderingtechniquesandvoxel-basedterrainmodeling. Our keyframelessimage-basedrenderingtech-niqueis describedin detailin Section3, which focuseson thecon-structionandwarpingof theintermediateimage.Section4 presentsresultsfrom our implementation.Conclusionsandfuturework arediscussedin Section5.

2 Previous Work

2.1 Image-based rendering

In recentyears,therehave beenmany paperson image-basedren-dering. Mark’s [8] is a goodsurvey. Someresearchershave usedthekeyframemethodto acceleraterenderingin a walk-throughorfly-throughsystem.

Mark [7] haswarpedtwo different referenceimagesandcom-positedthe resultsto avoid occlusion-relatedartifacts. The refer-enceimagesaregeneratedon thefly basedon thepredictionof thefutureviewpointandview direction.Thereferenceframesareren-deredat 5 frame/sec,and derived framescan be generatedat 30frame/sec.

Popescuetal.[12] havewarpedlayereddepthimagesin thecon-text of an architecturalwalk-throughsystem. The layereddepthimagesaresynthesizedin a preprocessingphasefor everyportal.

Chenet al.[2] useda hybrid LOD-Spritetechniqueto accelerateterrainrendering.A spriteimageis generatedfrom high-resolutionscenegeometryat every keyframe. Thenthespritesarereusedbytexturemappingin thefollowing frameusinglow-resolutionscenegeometry. The keyframesaregeneratedon the fly. However, thetime to rendera keyframecanbeashigh asthreetimesmorethanthetime to rendera derivedframe.

Insteadof usingan imageto representtheentirescene,the im-postermethods[6, 14] usesimageto representremoteobjectsin ascene.

Amongvariouswarpingalgorithms,McMillan andBishop’s 3Dwarpmethod[10, 11] is anefficient algorithm. It warpstheimagewith depthinformation,soit canprovideproperparallaxandcanbeimplementedefficiently by incrementalcomputationof the3D warpequationandby usinganocclusioncompatibleorderingalgorithmto resolve occlusionwithout z-buffering. McMillan andBishop’salgorithmrelieson the epipolargeometry. Given two images,theepipolein thesecondimageis theprojectionof thecameralocationof thefirst imageinto thesecondimage.Theepipolarlinesaretheimage-spacelinesthatpassthroughtheepipole.Therelative warporderfor differentepipolarlinesdoesnot matter, but thepointsona singleepipolarline mustbe warpedin a particularorder. Thus,theinput imageis split into sheetshorizontallyandverticallyat theepipolarpoint. Every sheetis processedin a different scanlineorder. Popescuet al.[12] pointedout thatspecialattentionsshouldbepaidto thepixelson therow andcolumnof theepipolebecausepixels in discreteimageshave a non-negligible area.They have tobewarpedeitherfirst or last,dependingon thesignof theepipole.

Ourwork is basedon McMillan andBishop’salgorithm.

2.2 Voxel-based terrain modeling

Mostof thecurrentflight simulatorsarebasedonthesurfacemodel.Howeverthevoxel-basedapproachhasmany advantagescomparedto a surface-basedone. For example,the format of the elevationmap lendsitself to generatinga very high resolution3D volumeof terrainandmulti-resolutionvolumes.Also, texturemappingforvoxelsis muchsimpler, of higherquality, andcanbepreprocessed.More importantly, thevoxel-basedmodelis somewhatscenecom-plexity independent,and it is easyto incorporateclouds, haze,flames,and other amorphousphenomenaand volumetric objects[4]. Therefore,in recentyears,the voxel-basedapproachhasbe-comea new researchissuefor height-fieldvisualizationsystems[18, 5, 3].

Most of thepublishedvoxel-basedterrainrenderingalgorithmsdirectly usethe 2D elevation mapto represent3D values,so thatbothmemorysizeandaccessingtime for theterrainmodelcanbegreatlyreduced[5, 3]. However, the quality of imagesgeneratedfrom this model is not very good. Thus, a true 3D voxel-basedterrainmodelhasbeengeneratedin our previous voxel-basedter-rain visualizationsystem[16]. The terrainmodelconsistsof a 3Dvolumetricterraindatasetandacorrespondingaerialphotographorsatelliteimageof theterrain.Therearealsosomedisadvantagesofthis 3D voxel-basedterrainmodel. The storagerequirementfor a3D voxel-basedterrainis high, but memoriesarebecominglargerandrelatively inexpensive. If thereis no hardwaresupport,it usu-ally takeslonger time to rendera 3D voxel-basedterrainby ray-castingthanto rendera surface-basedonewith the hardwaresup-port for texturemapping.

3 3D Warp Without Keyframe

3.1 Keyframeless rendering

In a walk-throughor fly-through system,the viewpoint changesgradually. Thereis substantialcoherencein adjacentframes. Astraightforwardway to take advantageof this is to rendersomekeyframesby theray-castingmethodandthenwarpthekeyframesto get thenext frames. Whenthe error is beyond a threshold,thekeyframeis rerendered.Becauseof thechangeof viewpoint,somenew part of the sceneentersthe user’s view and someoccludedpartof thescenein thekeyframecanpopup in thenew frame.Thiscausesholesin theresultingimage.Theseholescanbefilled in byraycasting.Figure1 showsthis method.

As we mentionedbefore,this keyframemethodhasthe disad-vantagesof unstableframeratesandnonuniformimagequalities.This methodonly takesadvantageof the coherencebetweenthekeyframeandthederived frame. Therearealsocoherencesin thederived frames. Let’s take a look at Figure 1. Frame1 is thekeyframe generatedby using the ray-castingmethod. Frame2to Framen arederived framesby warpingFrame1. Actually, atevery derived frame,exceptthepixels warpedfrom the keyframe,a certainamountof pixels have to be renderedby the ray-castingmethodbecauseof the holesin the warpedimages. Thesepixelshave a high imagequality andlikely will appearin thenext framesbecauseof thecoherencein theadjacentderivedframes.To makea difference,we usethe term “high quality pixels” to refer to thepixels generatedby the ray-castingmethod in a derived frameand“low quality pixels” to refer to pixels reconstructedfrom thewarpedimage. Traditionally, the keyframemethoddoesnot takeadvantageof this.

Therearetwo intuitive methodsthat cantakeadvantageof co-herencesin derivedframes:

1) Warpingframem-1 to get framem. Imagequality degrades

Keyframe Frame 1 Frame2

Frame n-2 Frame n-1 Frame n

Pixels Rendered by Ray Casting

Figure1: A straightforwardkeyframetechnique.Thekeyframeisrenderedby the ray-castingmethod. Frames1-n arerenderedbywarping the keyframe. The holes in frames1-n are filled in byray casting.Whentheerror is beyonda threshold,thekeyframeisrerendered.

Figure2: Offset buffer. The exact positionsof the warpedpixelsarerecordedby anoffsetbuffer.

quickly afterseveralsuchwarpsbecausemostpixelsin framem-1arefrom reconstructionandhave low imagequality.

2) Warpingthehighqualitypixelsfrom thepreviousm-1 framesto get framem. Then, the Z-buffer hasto be usedto resolve theocclusion. We have to warp scatteredpoints in m-1 frames. Theadvantageof theincrementalcomputationdoesnotexist.

We try to designanimage-basedrenderingtechniquewhich ex-ploits thecoherencesin themulti-frameswhich meetthefollowingrequirements:1) Only highqualitypixelscanbewarped.Wedonotwarpthepix-elsfrom reconstruction.2) No Z-buffer is neededto resolve theocclusionproblem3) Preserve theadvantageof incrementalcomputation.4) No keyframeis needed.

As shown in Figure2, whenapixel is warpedfrom thekeyframeto frame1, the centerof the pixel generallydoesnot fall exactlyon thegrid of thederived frame. Thusa reconstructionprocessisneeded.However, if we storethe exact position of the centerofthis pixel in frame 1, warping this pixel from frame1 to frame2will have the sameeffect as warping the original pixel from thekeyframeto frame2.

Intermediate Image i

Intermediate Image i+1

Intermediate Image i+2

Displayed Image i

Displayed Image i+1

Displayed Image i+2

Reconstruct Reconstruct Reconstruct

Warp Warp

UpdatePartially

Update Partially

Figure3: ThekeyframelessIBR method.The intermediateimagei+1 is renderedby warpingtheintermediateimagei. Theexactpo-sitionsof warpedpixelsin theintermediateimagei+1 arerecordedby theoffset buffer. After that, the intermediateimagei+1 is par-tially updatedby raycastingaccordingto theagesof thepixels.Thedisplayedimagei+1 is reconstructedfrom the intermediateimagei+1. Thefirst intermediateimageis renderedby raycasting.

We referto thefinal reconstructedimageasthedisplayedimage,andreferto thewarpedimagebeforereconstructionastheinterme-diateimage. The exact positionsof warpedpixels in the interme-diate imagearerecordedby an offset buffer which representstheoffset of thecenterof the pixels from their nearestneighborhood.The pixels in the intermediateimageareall first renderedby raycastingin a previous frameandthenwarpedinto the intermediateimageat thecurrentviewpointandview direction.

At thebeginning,thefirst frameis generatedby theray-castingmethod. Thus, the intermediateimage is exactly the sameasthe displayedimage. The offsetsof all pixels at the intermediateimageare0. Then,we warptheintermediateimageto get thenextintermediateimage. The offsetsof warpedpixels arerecordedbyan offset buffer. Now thereare someholes in the intermediateimage. We fill in theseholes by the ray-castingmethod. Thedisplayedimage is reconstructedfrom this intermediateimage.Then,we warpthis intermediateimageto getthenext intermediateimage and reconstructthe next displayedimage from the nextintermediateimage,andsoon.

Therefore,atany intermediateimagesomepixelsarenewly ren-deredby theray-castingmethod.Somepixelsarewarpedfrom thepreviousintermediateimage.In orderto makeadifferencebetweenthesetwo typesof pixels,we useanagebuffer to recordtheageofa pixel. If a pixel is just renderedin this imageby the ray-castingmethod,its ageis 0. If a pixel is warpedfrom theprevious inter-mediateimageandis still visibleat thecurrentintermediateimage,thenits ageis incrementedby 1. By usingtheagebuffer, we knowexactly how many timesa pixel hasbeenwarpedfrom the inter-mediateimagewherethispixel is first generatedby theray-castingmethodto thecurrentintermediateimage.Theolderthepixel isandthelongerthepixel staysin theintermediateimage,themorelikelyit will disappearin thenext frame,andmoreimportant,the morelikely the imagequality aroundthis pixel will be degraded.Thus,theageof a pixel is a nicecriteriafor theimagequalityaroundthispixel.

In order to guaranteethe imagequality, we seta thresholdforthe agesof the pixels in the intermediateimage. If the ageof apixel is older thanthis threshold,thenwe do not warp this pixel.Thisprobablycausesholesin thenext intermediateimageandtheseholesarefilled by theray-castingmethod.

Therefore,thepixelsin theintermediateimageareall highqual-

Sheet Boundary

Epipole

Figure 4: Warp order of the intermediateimage for a negativeepipole.

ity pixels. By usingtheoffset buffer to recordtheexact positionsof thesepixels, warpingthesepixels from the intermediateimagehasthesameeffect aswarpingtheoriginal pixels from the imagewherethesepixelswerefirst generatedby theray-castingmethod.By usingtheagebuffer, wecancontroltheimagequalityby settinga thresholdfor agesof pixels in the intermediateimage. Figure3showsourkeyframelessIBR method.

PleasenotethatPopescuet al. [13] have usedthe offset bufferfor a differentpurpose.They usedthe offset buffer to resolve thevisibility of thepixels.Weusetheoffsetbuffer to recordtheprecisepositionsof thepixels in the intermediateimagefor furtherwarp-ing.

3.2 Warping the intermediate image

The pixels in the intermediateimageno longer fall in the regularrectangulargrid. However, asshown in Figure4, the intermediateimage can still be warpedby McMillan and Bishop’s orderingalgorithm.As we mentionedbefore,therearetwo propertiesof theocclusion-compatibleorder[8]: (1) therelative warporderfor tworeference-imagepoints on different reference-imageepipolarlinedoesnotmatter;(2) thepointson asinglereference-imageepipolarline mustbe warpedin a particularorder. Even thoughthe scanline in theintermediateimageis no longerastraightline, thesetwopropertiesarestill preserved. The formal proof of McMillan [11]applies,sotheorderingalgorithmstill works.

Oneadvantageof McMillan andBishop’s 3D warpalgorithmisthe incrementalcomputation.We usethe notationof the pinholecameramodeladoptedby McMillan [11]. Thenthe3D warpequa-tion is [8] :

z2

S2

S1

z1u2 � P � 1

2 P1u1 � P � 12

S1

z1

C1 C2 �

S � �a �b� �a �b ��� �c � u1 �

��u1v11

��� u2 �

��u2v21

���

The vectors�a � �b � �c arethe basisvectorsfor the cameracoordinate

system.P andC representthe pinholecameraviewing parameterandcenterof projectionrespectively for theimages.Theu1 , u2 are

theimagecoordinatesof thereferenceframeandthederivedframe.z1 andz2 arethereferenceframeandderivedframedepthvalues.

To computethe warpedpositionof the next pixel alonga scanline, incrementalcomputationcanbeused:

P� 12 P1 �

��u1 � 1

v11

��� P � 1

2 P1 ���

u1v11

��� P � 1

2 P1 ���

100

��� start � uincr

For an intermediateimage, the next pixel coordinatesat theimagespacealonga scanline is

u � 1 � of f setu � v � of fsetv� 1�

insteadofu � 1 � v� 1� . The incrementalcomputationcannotbe

directlyused.

As mentionedbefore,wesetathresholdfor theagesof thepixelsin theintermediateimages.Thatmeansthenumberof timesapixelcanbewarpedis boundedby a threshold.We canusea fixedpointinsteadof a float point to improve theefficiency. We canquantizetheoffsetof pixelsinto levels.Let’ssupposeat imagespacetheonepixel distanceisquantizedintoL levels.At everyframe,thewarpedpixel positionerror introducedby thequantizationis 0 � 5

1� L � .Thus,thetotalpossibleerrorintroducedby thequantizationaftermwarpsis boundedby c m

1� L � , with c asa constant.Because the offset is quantized, we can precompute

P � 12 P1 ��� of f setu � 0 � 0 � and P � 1

2 P1 ��� 0 � of f setv� 0 � for allpossiblelevels of the offset and storethemin two lookup tablestableu and tablev. Beforerenderingeachnew image,we usethenew camerainformationto precomputevaluesfor all lookuptableindices. Then, McMillan and Bishop’s incrementalcomputationmethodcanbeadaptedby addingtwo itemsto compensatefor theoffsetof thepixelsfrom theregulargrid:

P � 12 P1 �

��u1 � 1 � of f setu

v1 � of f setv1

���

P � 12 P1 �

��u1v11

��� P � 1

2 P1 ���

100

���

P � 12 P1 �

��of f setu

00

��� P � 1

2 P1 ���

0of f setv

0

��

� start � uincr � tableu � of f setu� � tablev � of f setv�3.3 Filling in holes

Theholesin theintermediateimagearefilled in by theray-castingmethod. It can be done efficiently by exploiting the coherencebetweenrays. For example, we show how to fill in holes in avoxel-basedterrain renderingsystem. We fill in the holesin thewarpedimagecolumn by column. In eachcolumn, we castoneray througheachhole pixel bottom up, in order to speedup raytraversal,by exploiting the specificvertical ray coherence[5, 3]in terrain scenes. As shown in Figure 5, the basic idea of raycoherenceis that for two viewing raysfrom thesamecamerathatproject onto the sameline on the baseplane of the terrain, thehigherray hits the terrainsurfaceat a positionfartheraway fromthe viewpoint. By exploiting this ray coherence,ray castingcanbe dramaticallyacceleratedby skippingmostof the empty spaceabovetheterrainsurface.

Viewpoint

Image PlaneHoles

Pixels

Start Point of the Ray

Figure5: Theray to fill in a holecanemanatefrom thehit pointofthepixel just below it

During our hole filling procedure,ray castingfor eachimagecolumnis completedin thefollowing steps:first, find the locationof the lowesthole pixel andthe depthof the pixel just below thispixel from thecurrentintermediateimagebuffer. Castaraythroughthis pixel andusethecorrespondingdepthto skip theemptyspacealongtheray. In a specialcasewhenthelowestholepixel is at thebottomof thecolumn,we have to traversetheray from the imagewithout any spaceleaping. Oncethe hit position is found alongtheray, save thenew depthvalue.Second,move upwardalongthecolumnto thenext holepixel andcasta ray throughit. If its lowerpixel is a ray-castingpixel, we useits lower pixel depthfor spaceleaping;otherwise,its lower pixel is a warpedpixel, and we usethe lower pixel depthfrom warpingfor spaceleaping.Third, savethenew depthoncethehit point is found.Fourth,repeatthesecondstep,until noholepixelsareleft or thecurrentraydoesnot intersectwith the terrainsurface.Finally, searchupwardalongthecolumnfor all remainingray-castingpixels,anddirectly assigntheir colorsto bebackgroundcolor. No ray castingis performedthroughthesepixels, sincethey have no chanceof intersectingwith the terrainsurface.

3.4 Balancing the workload

At every viewpoint,we needto warptheintermediateimageat thepreviousviewpointandfill in theholes.Thetimeto warpanimageis independentof thescenecomplexity andalmosta constant.Thetime to fill in the holesby ray castingdependson the numberofholes, which can vary from frame to frame. As we mentionedbefore,theholesarecausedby :1) Someregion of terrainjust entersthecurrentview.2) Someregion of terrainoccludedin thepreviousframenow popsup.3) Theagesof somepixelsat theintermediateimageareolderthana threshold,sowe do notwarpthesepixels.

The total numberof thesepixels shouldbe controlledby ourcomputationalability. We also hope that the numbersof pixelsneededto be rerenderedby ray castingat every framearealmostthe same. We cannotcontrol the numberof holescausedby (1)(2). However, we cancontrolthenumberof holescausedby (3) byadjustingthethreshold.Theway to balancetheworkloadat everyframeis to dynamicallyadjustthenumberof pixelscausedby (3).

We seta thresholdfor the total numberof pixels we canaffordto rerenderby ray casting.Then,at every frame,afterwarpingtheintermediateimage,we countthenumberof holes. If thenumberis beyondour threshold,we cando nothingelseexceptfill in theseholes.However, if thenumberis below our threshold,we computethedifferenceof thethresholdandthenumberof ourcurrentholes.

Figure6: CameraPath.

Then we can selectthat numberof pixels from the pixels whicharetheoldestin the intermediateimageandrerenderthemby raycasting.In thiswaywecanbalancetheworkloadandguaranteetheimagequality at thesametime.

3.5 Reconstruction

Reconstructionis a difficult problem in image-basedrenderingtechniques.Changet al.[1] usea bilinear kernel. Four LDI pixelsareupdatedfor eachpixel of a referenceimage. Shadeet al.[15]usea rough approximationto the footprint evaluationoptimizedfor speed. The image size is approximatedby using a lookuptable. The four splatsizesthey usedhave 1 1,3 3, 5 5, and7 7 pixel footprints. The alphavaluesareroundedto 1, 1/2, or1/4. Therefore,thealphablendingcanbedonewith integershiftsand adds. Mark et al.[7] usea techniqueto treat the referenceframeasa mesh. The 3D warp perturbsthe verticesof the mesh.Reconstructionoccursby renderingthe perturbedmeshtrianglesinto thederivedframe.

For simplicity, we useonepixel-widereconstructionkernel.Wejust write a single pixel in the intermediateimagein the nearestneighborpositionof thederivedimage.

4 Experimental Results

Our method was implementedon a Silicon Graphics Onyx2

(1024MB RAM, four R1000 processors)with Infinite Realitygraphics. However, we did not exploit its parallel processingcapabilityin our implementation.Only oneprocessorwasused.

We demonstratedour methodwith a voxel-basedrenderingsys-tem[16]. Our terrainmodelconsistsof a 3D terrainvolumewith aresolutionof 512 512 64anda correspondingregisteredaerialphoto. Figures7 give the experimentresultsof our algorithmfora camerapath shown at Figure6 which producesmore than 300

0

0.1

0.2

0.3

0.4

0 50 100 150 200 250 300

Frame�Number

Time�(Second)

Derived�frame Keyframe

Figure7: Timeto rendereachframeby thekeyframeIBR method.

0

0.1

0.2

0.3

0.4

0 50 100 150 200 250 300

Frame�Number

Time�(Second)

Figure 8: Time to rendereachframe by our keyframelessIBRmethod.

frameimages.The imagesizeis 500 400. Theray-castingren-deringmethodusedin this paperis basedonour earlierwork[16].

Figures7 and8 show theamountof time requiredto rendereachframe. We comparedour algorithmwith the keyframealgorithm.Figure7 shows the result of the keyframemethod. We renderedthekeyframeby theray-castingmethodonthevoxel terrainmodel.Tenderivedframesweregottenby warpingonekeyframe.We canseethatthetimeto renderkeyframesis abouttwo timeslongerthanthe time to renderthederivedframes.Eventhe time to renderde-rived framescan be quite different. Figure8 shows the resultofour method.Thethresholdfor theageof pixels is 10. Thenumberof pixels neededto be rerenderedis about14 percentof the totalnumberof pixelsin theintermediateimage.ComparingFigure7 toFigure8, we canseethat our methodcangeta morestableframeratethanthekeyframemethod.

Figures11 and12 show a view of the terrain(FrameNo. 100).Figure 11 shows the terrain renderedby the ray-castingmethod.Figure 12 shows the terrain renderedby our keyframelessIBRmethod. Figure13 gives the absolutevalueof the differencebe-tweenthesetwo images.Wequantizedtheoffsetof pixelsinto1024levels. Becausethe thresholdfor theageof pixels is 10, we thinkthat thepositionerrorsof pixels in the intermediateimagecausedby quantizationandatmost10consecutivewarpingsarenegligible.The imagequality of our algorithmdependson thereconstructionmethod.We usedonepixel-widereconstructionkernelandgotsat-

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0 100 200 300 400 500 600

Frame�Number

Time�(Second)

Derived�frame Keyframe

Figure9: Time to rendereachframeby thekeyframeIBR method.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 100 200 300 400 500 600

Frame�Number

Time�(Second)

Figure 10: Time to rendereachframe by our keyframelessIBRmethod.

isfying results.If moreprecisereconstructionmethodsareusedtheimagequality canbefurtherimproved.

We alsodemonstratedour methodon anotherterrain. The ter-rain sizeis 468 693 64. Figures9 and10 show theamountoftimerequiredto rendereachframe.Figure9 showstheresultof thekeyframemethod.Figure10 shows theresultof our method.Fig-ure14 shows the terrainrenderedby the ray-castingmethod.Fig-ure15showstheterrainrenderedby ourkeyframelessIBR method.Figure16 shows the differencebetweenthesetwo images. Fromthesefigureswe got the following observations:our methodgetamorestableframeratethanthekeyframemethod. The quality ofthe imagesgeneratedby our methodis comparableto that of raycastingandcanbesustainedat thatlevel.

5 Conclusions and Future Work

We have presenteda novel image-basedrenderingtechniqueusedin voxel-basedscenemodeling.Our contributionsare:1) We designeda keyframelessimage-basedrenderingtechniquewhich combinestheadvantagesof the3D warpalgorithmandtheray-castingmethod.The3D imagewarpalgorithmis usedto gainahigh framerate,andtheray-castingmethodis usedto fill in holesandto selectively rerenderpartof theimageto guaranteetheimagequality.

2) An offsetbuffer is usedto recordtheprecisepositionof a pixel,so pixels canbe consecutively warpedfrom a singleintermediateimage without loss of accuracy. McMillan and Bishop’s warpalgorithmis adaptedto warptheintermediateimage.By quantizingtheoffset,we canpreservetheincrementalcomputation.3) An agebuffer is usedto recordtheageof a pixel. This providesa nicecriteriafor imagequality. By slightly adjustingthethresholdfor theagesof pixelsin theintermediateimage,we canbalancetheworkloadat every frame. Our methodindeedobtainsmorestableframeratesandmoreuniformimagequalities.

We demonstrateour methodwith a voxel-basedterrainrender-ing system. However, our methodcan be appliedto many otherapplications.It is especiallyuseful in theapplicationswhich havethefollowing features:1) The keyframeshave to be renderedon the fly and the time torenderonekeyframeis muchhigherthanthe time to generateonederivedframe.2) Raycastingcanbeusedasa renderingmethod.

There are also some limitations to our method. First, theirregulardistribution of pixels in theintermediateimagecancauseextradifficulty for reconstruction.If thenearestneighboris usedasthereconstructionmethod,we foundthatwarpingtheintermediateimagecancausemoreholesthanwarpingthe keyframe. Second,like other image-basedrenderingmethods the performanceofour methoddependson the frame-to-framecoherence. If thereis nice frame-to-framecoherence,suchas in the terrain dataset,our methodcangive pretty stableframeratesanduniform imagequalities. If the frame-to-framecoherenceis not that good, suchas in a datasetwith a very complicatedbackgroundscenery, thenit is possiblethat the number of holes per frame can be varysubstantially, andtheray castingmaytakelongerthanexpectedtofill in theseholes.This mayleadto inconsistentframerates.

Somework needto be donein the future. We want to find amoreaccurateandfastreconstructionmethodfor our intermediateimage. We plan to usea splattingmethodinsteadof the currentnearestneighbormethod,sothenumberof holescanbedecreasedandtheimagequalitycanbeimproved. We alsoplanto parallelizeourmethodto achievehigherframerates.

Acknowledgments

This work is partially supportedby ONR grantN000149710402.Thanksto Nan Zhang,BaoquanChen,andothermembersof thevirtual fly-throughproject.Thanksto ManuelMenezesdeOliveiraNeto andanonymousreviewersfor their valuablecomments.Wewouldalsolike to thankMarianneCatalanofor proofreading.

References

[1] Chun-Fa Chang,GaryBishop,AnselmoLastra.LDI Tree:AHierarchicalRepresentationfor Image-basedRendering.SIG-GRAPH99ConferenceProceedings, pages291-298,August1999.

[2] BauquanChen, J. EdwardSwanII, Eddy Kuo, Arie Kauf-man.A Hybrid LOD-SpriteTechniquefor AcceleratedTer-rainRendering.Proceedingsof IEEEVisualization’99 , pages291-298,October1999

[3] D. Cohen-Or, E. Rich, U. Lerner, and V. Shenkar. A Real-Time Photo-RealisticVisualFly-through.IEEE Transactionson Visualizationand ComputerGraphics, 2(3), pages255-265,1996.

[4] A. Kaufman,D. CohenandR.Yagel.VolumeGraphics.Com-puter, 26(7),pages51-64,1993.

[5] C. LeeandY. Shin.An EfficientRayTracingMethodfor Ter-rainRendering.PacificGraphics’95, pages181-193,1995.

[6] Paulo W. C. Maciel and PeterShirley. Visual navigation oflargeenvironmentsusingtexturedclusters.Proceedingsof the1995Symposiumon Interactive3D Graphics, pages95-102,April 1995.

[7] William R. Mark, LeonardMcMillan andGaryBishop.Post-Rendering3D Warping.Proceedingsof the1997Symposiumon Interactive3D Graphics, pages7-16,April 1997.

[8] William R. Mark. Post-Rendering3D ImageWarping: Vis-ibility, Reconstruction,and Performancefor Depth-ImageWarping. Ph.D. Dissertation,University of North Carolina,April 1999.(Also UNC ComputerScienceTechnicalReportTR99-022).

[9] LeonardMcMillan and Gary Bishop. Head-trackedstereo-scopicdisplayusingimagewarping.ProceedingsSPIE, vol-ume2409,pages21-30,February1995.

[10] LeonardMcMillan and Gary Bishop. PlenopticModeling.SIGGRAPH95 ConferenceProceedings, pages39-46,Au-gust1995.

[11] Leonard McMillan. An Image-BasedApproach to Three-DimensionalComputerGraphics.Ph.D. Dissertation.Tech-nical Report97-013,University of North Carolinaat ChapelHill. April 1997.

[12] Voicu S. Popescu,Anselmo A. Lastra, Daniel G. Aliaga,ManueldeOliveiraNeto.Efficient Warpingfor ArchitecturalWalkthroughsusingLayeredDepthImages.IEEE Visualiza-tion ’98 , pages211- 215,October1998.

[13] Voicu S.Popescu,andAnselmoLastra.High Quality 3D Im-ageWarping by SeparatingVisibility from Reconstruction.UNC ComputerScienceTechnicalReportTR99-017,Univer-sity of NorthCarolina,April 5, 1999.

[14] G. Schaufler. Dynamically GeneratedImpostors.In D. W.Fellner, editor,Modeling Virtual-Worlds-Distributed Graph-ics,MVD 95 Workshop, pages129-136,November1995.

[15] JonathanShade, Steven Gortler, Li-wei He and RichardSzeliski.LayeredDepthImages.SIGGRAPH98 ConferenceProceedings, pages231-242,July1998

[16] Ming Wan, Huamin Qu, and Arie Kaufman. Virtual Fly-throughoveraVoxel-BasedTerrain.Proceedingsof IEEEVir-tual Reality1999, pages53-60,1999

[17] Lee Westover. Footprint evaluation for volume rendering.ComputerGraphics (SIGGRAPH90 ConferenceProceed-ings), volume24,pages367-376,August1990.

[18] J. Wright and J. Hsieh.A Voxel-Based,ForwardProjectionAlgorithm for RenderingSurfaceandVolumetricData.IEEEVisualization’92, pages340-348,1992.

.

Figure11: A California terrain(size: 512 512 64) renderedbytheray-castingmethod.(SeealsoColorPlate.)

Figure12: A California terrain(size: 512 512 64) renderedbyourkeyframelessIBR method.(SeealsoColorPlate.)

Figure13: ThedifferencebetweenFigure11 andFigure12. (SeealsoColorPlate.)

.

Figure14: A Los Angelesterrain(size: 468 693 64)renderedby theray-castingmethod.(SeealsoColor Plate.)

Figure15: A Los Angelesterrain(size: 468 693 64)renderedby ourkeyframelessIBR method.(SeealsoColorPlate.)

Figure16: ThedifferencebetweenFigure14 andFigure15. (SeealsoColor Plate.)