cs380: introduction to computer graphics reconstruction...

18/05/15

1

MinH.Kim(KAIST) Foundationsof3DComputerGraphics,S.Gortler,MITPress,2012

CS380:IntroductiontoComputerGraphicsReconstruction&Resampling

Chapter17&18

MinH.KimKAISTSchoolofComputing


SUMMARYSampling(continuousàdiscrete)

2

18/05/15

2


Aliasing•  Scenemadeupofblackandwhitetriangles:jaggiesatboundaries–  Jaggieswillcrawlduringmotion

•  Iftrianglesaresmallenoughthenwegetrandomvaluesorweirdpatterns– Willflickerduringmotion

•  Theheartoftheproblem:toomuchinformationinonepixel

3


Anti-aliasing•  Wecanalsomodelthisasanoptimizationproblem.•  Theseapproachesleadto:

•  whereissomefunctionthattellsushowstronglythecontinuousimagevalueatshouldinfluencethepixelvalue

4

I[i][j]← I(x , y)Ω∫∫ Fi , j(x , y)dxdy

Fi , j(x , y)

[x , y]t i,j

I[i][j]

I(xw , yw )

I(x , y)←

ΩFi , jxw = i , yw = j

Fi , j(x , y)

I(x , y)

Verticalsectionview

18/05/15

3


•  Eventhatintegralisnotreallyreasonabletocompute•  Instead,itisapproximatedbysomesumoftheform:wherekindexessomesetoflocationscalledthesamplelocations,so-calledover-sampling.

•  Therendererfirstproducesa“highresolution”colorandz-buffer“image”,– wherewewillusethetermsampletorefertoeachofthesehighresolutionpixels.

Over-sampling

5

I[i][j]← 1

nI(xk , yk )

k=1

n

∑

(xk , yk )

apixel


Overoperation•  Tocompose,wecomputethecompositeimagecolors,,using

•  Thatis,theamountofobservedbackgroundcoloratapixelisproportionaltothetransparencyoftheforegroundlayeratthatpixel.

•  Likewise,alpha(coverage)forthecompositeimagecanbecomputedas:

6

If[i][j] over Ib[i][j]Ic[i][j]

Ic[i][j]← If [i][j]+Ib[i][j](1-α f [i][j])

αc[i][j]← α f [i][j]+α b[i][j](1-α f [i][j])

18/05/15

4


Alphablendingerror•  Thedifferencebetweentwo

•  Thereforetheerroristhedifferencebetweentheintegralofaproductandaproductofintegrals.

•  Itistheamountof“correlation(covariance)”betweenthedistributionofforegroundcoverageinsomepixelandthedistributionofthebackgrounddatawithinthatpixel.

7

error= Ib(x , y)Cb(x , y)C f (x , y)dxdyΩi , j∫∫ −

Ib(x , y)Cb(x , y)dxdy× C f (x , y)dxdyΩi , j∫∫Ωi , j

∫∫

cov(X ,Y )=E[XY ]−E[X ]E[Y ]

Realalphablending

Idealcontinuouscomposition


RECONSTRUCTION(DISCRETEààCONTINUOUS)

Chapter17

8

18/05/15

5


Reconstruction•  GivenadiscreteimageI[i][j],howdowecreateacontinuousimageI(x,y)?

•  Itisthekeyproblemforresizingimagesandtexturemapping.– Howtogettexturecolorsthatfallinbetweentexels.

•  Thisprocessiscalledreconstruction.

9


Constantreconstruction•  Arealvaluedimagecoordinateisassumedtohavethecoloroftheclosestdiscretepixel.Thismethodcanbedescribedbythefollowingpseudo-code:

•  The(int)typecastroundsanumberp(pixelposition)tothenearestintegernotlargerthanp.

10

colorconstantReconstruction(floatx,floaty,colorimage[][]){inti=(int)(x+.5);intj=(int)(y+.5);returnimage[i][j];}

18/05/15

6


Constantreconstruction•  Theresultingcontinuousimageismadeupoflittlesquaresofconstantcolor.

•  Eachpixelhasaninfluenceregionof1-by-1

11


Bilinearproperties•  Atintegercoordinates,wehaveI(x,y)=I[i][j];thereconstructedcontinuousimageIagreeswiththediscreteimageI.

•  Inbetweenintegercoordinates,thecolorvaluesareblendedcontinuously.

•  Eachpixelinthediscreteimageinfluences,toavaryingdegree,eachpointwithina2-by-2squareregionofthecontinuousimage.

•  Thehorizontal/verticalorderingisirrelevant.•  Coloroverasquareisbilinearfunctionof(x,y).

12

18/05/15

7


Bilinearfunction•  1by1squarewithcoordinatesforsomefixed.

•  wherearethefracxandfracyabove.

13

j < y < j +1i < x < i +1

i and j

I(i+ x f , j+ y f )←(1− y f ) (1− x f )I[i][j]+(x f )I[i+1][j]{ }+( y f ) (1− x f )I[i][j+1]+(x f )I[i+1][j+1]{ } ,

x f and yf

x f

y f


Bilinearreconstruction•  Cancreateasmootherlookingreconstructionusing

bilinearinterpolation.•  Bilinearinterpolationisobtainedbyapplyinglinearinterpolationinboththehorizontalandverticaldirections.

14

colorbilinearReconstruction(floatx,floaty,colorimage[][]){intintx=(int)x;intinty=(int)y;floatfracx=x-intx;floatfracy=y-inty;colorcolorx1=(1-fracx)*image[intx][inty]+(fracx)*image[intx+1][inty];colorcolorx2=(1-fracx)*image[intx][inty+1]+(fracx)*image[intx+1][inty+1];colorcolorxy=(1-fracy)*colorx1+(fracy)*colorx2;return(colorxy);}

18/05/15

8


Bilinearfunction•  Rearrangingtheterms,

•  Thisfunctionhastermsthatareconstant,linear,andbilineartermsinthevariables

15

I(i+ x f , j+ y f )← I[i][j]+ −I[i][j]+I[i+1][j]( )x f + −I[i][j]+I[i][j+1]( ) y f + I[i][j]-I[i][j+1]-I[i+1][j]+I[i+1][j+1]( )x f y f

(x f , yf )


Bilinearbasisfunction•  Rearrangethebilinearfunctiontoobtain:

•  Forafixedposition,thecolorofthecontinuousreconstructionislinearinthediscretepixelvaluesofI:

16

I(i+ x f , j+ y f )← 1− x f − y f + x f y f( )I[i][j]+ x f − x f y f( )I[i+1][j]+ y f − x f y f( )I[i][j+1]+ x f y f( )I[i+1][j+1]

(x f , yf )

I(x , y)← Bi , j(x , y)I[i][j]

i , j∑

18/05/15

9


Bilinearbasisfunction•  TheseBarecalledbasisfunctions(tentfunctions)

•  Theydescribehowmuchpixeli,jinfluencesthecontinuousimageat.

•  In1D,wecandefineaunivariatehatfunction.

17

[x, y]t

Hi (x)

Hi(x)=x − i+1fori−1< x < i−x+ i+1fori < x < i+1

0else

⎧

⎨⎪

⎩⎪

∧


•  In2D(bilinearfunction),letbeabivariatefunction:

•  Thisiscalledatentfunction

Bilinearbasisfunction

18

Ti, j (x, y)

Ti , j(x , y)=Hi(x)Hj( y).

18/05/15

10


Bilinearbasisfunction•  Inconstantreconstruction,isaboxfunctionthatiszeroeverywhereexceptfortheunitsquaresurroundingthecoordinates(i,j),whereithasconstantvalue1.

19

I[i][j]

I(xw , yw )

I(x , y)←ΩFi , j

xw = i , yw = j

Bi, j (x, y)

I[i][j]← I(x , y)Ω∫∫ Fi , j(x , y)dxdy


RESAMPLING(RECONSTRUCTION+SAMPLING,DISCRETEààCONTINUOUSààDISCRETE)

Chapter18

20

18/05/15

11


Resampling•  Let’srevisittexturemapping•  Westartwithadiscreteimageandendwithadiscreteimage.

•  Themappingtechnicallyinvolvesbothareconstructionandsamplingstage.

•  Inthiscontext,wewillexplainthetechniqueofmipmappingusedforanti-aliasedtexturemapping.

21


Resamplingequation•  Supposewestartwithatextureimage(discrete)T[k][l]andapplysome2DwarptothisimagetoobtainanoutputimageI[i][j].

22

T[k][l]← T(x , y)Ω∫∫ Fk ,l(x , y)dxdy

18/05/15

12


Resamplingequation•  Reconstructacontinuoustextureusingasetofbasisfunctions.

•  Applythegeometricwrap(attheviewpoint)tothecontinuousimage.

23

T(xt , yt )Bk ,l(xt , yt )

texturecoordinates(asVV): v

wn

and 1wn

T(xt , yt )← Bk ,l(xt , yt )T[k][l]

k ,l∑

T '(xw , yw )←T(M(xw , yw ))(xt , yt )=M(xw , yw )


Resamplingequation•  Integrateagainstasetoffilters(e.g.,aboxfilter)toobtainthediscreteoutputimage.

24

Fk ,l(xw , yw )

I[i][j]← T '(xw , yw )Ω∫∫ Fi , j(xw , yw )dxwdyw

18/05/15

13


Resamplingequation•  Letthegeometrictransformbedescribedbyawarpmappingfunction,whichmapsfromcontinuouswindowtotexturecoordinates.

•  Weobtain:

25

(xt , yt )=M(xw , yw )

I[i][j]← Fi, j (xw , yw )

Ω∫∫ Bk ,l M (xw , yw )( )k ,l∑ T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxwdyw

= T[k][l] Fi, j (xw , yw )

Ω∫∫ Bk ,l M (xw , yw )( )( )dxwdyw( )k ,l∑

I(x , y)← Bi , j(x , y)I[i][j]

i , j∑

(wecouldobtainanoutputpixelasalinearcombinationoftheinputtexturepixels.)


Resamplingequation

•  whereistheinversemappingistheJacobianmatrixofN(changesofvariables)(continuityofcompositionofFandN)

26

I[i][j]← det(D

M −1 ) Fi, j M −1(xt , yt )( )( )M (Ωi , j )∫∫ Bk ,lk ,l∑ (xt , yt )T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxtdyt

DN

ʹF =F !N

N =M−1

= det(DN ) ʹFi, j (xt , yt )( )M (Ωi , j )∫∫ Bk ,l

k ,l∑ (xt , yt )T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxtdyt

1( , ) ( , )w w t tx y M x y−=

•  Instead,wecanrewritetheintegrationoverthetexturedomain,insteadofthewindowdomain.


Ω∫∫ Bk ,l M (xw , yw )( )k ,l∑ T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxwdyw


18/05/15

14


Resamplingequation

•  Insummary,whenFisaboxfilter

•  whereishowmuchtexelk,lblendstoobtainthevalueatcontinuoustexturelocationp,fromreconstructionstep.

27

= det(DN )

M (Ωi , j )∫∫ Bk ,lk ,l∑ (xt , yt )T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxtdyt

I[i][j]← Bk ,l M (xw , yw )( )

k ,l∑ T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxw dywΩ∫∫

Bk ,l (p)



Ω∫∫ Bk ,l M (xw , yw )( )k ,l∑ T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxwdyw

= det(DN ) ʹFi, j (xt , yt )( )M (Ωi , j )∫∫ Bk ,l

k ,l∑ (xt , yt )T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxtdyt


Summary•  Thatis,weneedtointegrateovertheregiononthetexturedomain,andblendthatdatatogether.

•  WhenourtransformationMeffectivelyshrinksthetexture,thenhasalargefootprintover

•  IfMisblowingupthetexture,thenhasaverynarrowfootprintover

•  Duringtexturemapping,Mcanalsodofunnierthings,likeshrinkinonedirectiononly.

28

M(Ωi , j )

M(Ωi , j )T(xt , yt )

M (Ωi, j )T (xt , yt )


18/05/15

15


Blowuptextureàsmallfootprint•  Inthecasethatweareblowingupthetexture(narrowfootprintinwindows),thefilteringcomponenthasminimalimpactontheoutput.

•  Inparticular,thefootprintofmaybesmallerthanapixelunitintexturespace,andthusthereisnotmuchdetailthatneedsblurring/averaging.

•  Assuch,theintegrationstepcanbedropped,andtheresamplingcanbeimplementedas

29

M (Ωi, j )

I[i][j]← Bk ,l(xt , yt )

k ,l∑ T[k][l]⇐ det(DN )M(Ωi , j )

∫∫ Bk ,l(xt , yt )k ,l∑ T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxtdyt

(xt , yt )=M(i , j)


Blowuptextureàsmallfootprint•  WetellOpenGLtodothisusingthecallglTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_LINEAR).

•  Forasingletexturelookupinafragmentshader,thehardwareneedstofetch4texturepixelsandblendthemappropriately.

30

18/05/15

16


Shrinkingtextureàlargefootprint•  Inthecasethatatextureisgettingshrunkdown,then,toavoidaliasing,thefiltercomponentshouldnotbeignored.

•  Unfortunately,theremaybenumeroustexturepixelsunderthefootprintof,andwemaynotbeabletodoourtexturelookupinconstanttime.

31

M (Ωi, j )


Recap:MIPmapping•  AnacronymoftheLatinphrase,multuminparvo(muchinlittle).

•  Mipmapsareprecalculatedscaledversionsofanoriginaltexture.

32

18/05/15

17


Recap:MIPmapping•  Observethepatternofeachmipmapasitdecreasesinsize;itishalfthedimensionofthepreviousone.

•  Thispatternisrepeateduntilthelastmipmap’sdimensionis1×1.

•  Interpolatemulti-resolutiontexturesviatrilinearinterpolation

33


Mipmappingindetail•  Inmipmapping,onestartswithanoriginaltextureandthencreatesaseriesoflowerandlowerresolution(blurrier)texture.

•  Eachsuccessivetextureistwiceasblurry.Andbecausetheyhavesuccessivelylessdetail,theycanberepresentedwith½thenumberofpixelsinboththehorizontalandverticaldirections.

34

T 0

T i

18/05/15

18


Mipmappingindetail•  Thiscollection,calledamipmap,isbuiltbeforetheanytrianglerenderingisdone.

•  Thus,thesimplestwaytoconstructamipmapistoaveragetwo-by-twopixelblockstoproduceeachpixelinalowerresolutionimage.

•  Duringtexturemapping,foreachtexturecoordinate,thehardwareestimateshowmuchshrinkingisgoingon.– Howbigisthepixelfootprintonthegeometry.

35

(xt , yt )


Mipmappingindetail•  Thisshrinkingfactoristhenusedtoselectfromanappropriateresolutiontexturefromthemipmap.Sincewepickasuitablylowresolutiontexture,additionalfilteringisnotneeded,andagain,wecanjustusereconstruction.

•  Toavoidspatialortemporaldiscontinuitieswhere/wherethetexturemipmapswitchesbetweenlevels,wecanso-calledtrilinearinterpolation.Weusebilinearinterpolationtoreconstructonecolorfromandanotherreconstructionfrom.Thesetwocolorsarethenlinearlyinterpolated.Thisthirdinterpolationfactorisbasedonhowclosewearetochoosingleveliori+1

36

T i

T i T i+1

18/05/15

19


Mipmappingindetail•  MipmappingwithtrilinearinterpolationisspecifiedwiththecallglTexParameteri(GLTEXTURE2D,GLTEXTUREMINFILTER,GLLINEARMIPMAPLINEAR)

•  TrilinearinterpolationrequiresOpenGLtofetch8texturepixelsandblendthemappropriatelyforeveryrequestedtextureaccess.

37


Trilinearinterpolation•  Trilinearinterpolation(3D):

38

p(x, y, z) = c0 + c1Δx + c2Δy + c3Δz + c4ΔxΔy + c5ΔxΔzc6ΔyΔz + c7ΔxΔyΔz

p0 + [0,1,n,n +1,n2,

n2 +1,n2 + n,n2 + n +1]ThedimensionofCLUTis(2n+1)x(2n+1)x(2n+1)

18/05/15

20


Trilinearinterpolation•  where

39

c1 = (p100 − p000 ) / (x1 − x0 )c2 = (p010 − p000 ) / (y1 − y0 )c3 = (p001 − p000 ) / (z1 − z0 )c4 = (p110 − p010 − p100 + p000 ) / [(x1 − x0 )(y1 − y0 )]c5 = (p101 − p001 − p100 + p000 ) / [(x1 − x0 )(z1 − z0 )]c6 = (p011 − p001 − p010 + p000 ) / [(y1 − y0 )(z1 − z0 )]c7 = (p111 − p011 − p101 − p110 + p100 + p001 + p010 − p000 ) /

[(x1 − x0 )(y1 − y0 )(z1 − z0 )].

Δx = x − x0Δy = y − y0Δz = z − z0c0 = p000


Properties•  Itiseasytoseethatmipmappingdoesnotdotheexactlycorrectcomputation.

•  Firstofall,eachlowerresolutionimageinthemipmapisobtainedbyisotropicshrinking,equallyineverydirection.But,duringtexturemapping,someregionoftexturespacemaygetshrunkinonlyonedirection.

40

Nomipmapping

Mipmapping

Anisotropicmipmapping

18/05/15

21


Properties

41

M (Ωi, j )

•  Evenforisotropicshrinking,thedatainthelowresolutionimageonlyrepresentsaveryspecific,dyadic,patternofpixelaveragesfromtheoriginalimage.

•  Filteringcanbebetterapproximatedattheexpenseofmorefetchesfromvariouslevelsofthemipmaptoapproximatelycovertheareaonthetexture.

•  ThisapproachisoftencalledansiotropicfilteringandcanbeabledinanAPIorusingthedrivercontrolpanel.

cs380: introduction to computer graphics reconstruction...

Documents