![Page 1: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/1.jpg)
Image-Based Rendering using Hardware Accelerated
Dynamic Textures
Keith Yerex
Dana Cobzas Martin Jagersand
![Page 2: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/2.jpg)
Motivation
Rendering
Traditional geometry based techniques:
detailed 3D model + texture
hard to achieve photorealism
Image-based models
non-geometric model from images
practically hard to apply
![Page 3: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/3.jpg)
Challenges
Hard to generate detailed 3D models Texturing from images require very precise
alignment with the model
Rendering arbitrary views using IBM requires dense sample of the plenoptic function
IBR techniques don’t deal with dynamic scenes
![Page 4: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/4.jpg)
Overview I1 It
…
= =
+ +
Warped texture
Motion params
(R1 a1 b1) …(Rt at bt)
Structure P
Texture basis
y1 … yt
Texture coeff
New pose
(R a b)
Training Model New view
uv
ô õ= RP + a
b
ô õ
I wt = Byt+ IöI wqt = I qt(W(ut;vt))
I wt =P
q=1Q I wqt
I t =P
q=1Q I qt
![Page 5: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/5.jpg)
Structure from motion
Input – Structure from set of corresponding points tracked in a set of images
Assumptions Static scene Camera model
injective perspective weak perspective orthographic
Estimated modelprojective affine metric euclidean
![Page 6: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/6.jpg)
Structure from motionTracked features
Structure from motion
algorithm
poses
structure
![Page 7: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/7.jpg)
SFM algorithms
Few images, perspective camera, precise calibration
epipolar constraint
trilinear tensor
Long motion, affine or perspective structure
factorization methods
![Page 8: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/8.jpg)
Metric structure
Weak perspective camera
Extension of Tomasi Kanade factorisation
algorithm
Extract affine structure Relation between the sffine structure and camera
coordinate frame Transform the structure into metric (unit pixel size)
![Page 9: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/9.jpg)
Weak perspective projection
N points
Normalized with respect to centroid
Rank theorem
Factorization
uti = stiTtP i +atvti = stjTtP i +bt
W= RP + t1
Wê = RP where Wê =Wà t1
Wê = O1Î O2
Rê = O01
Pê= Î 0O02
![Page 10: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/10.jpg)
Metric constraints
Extract motion parameters Eliminate scale Compute direction of camera axis k = i x j parameterize rotation with Euler angles
Model P ReprojectionPose x = (r,s,a,b)
R =RêQP =Qà 1Pê
iêTtQQ
Tiêt =jêTtQQ
Tjêt ( = s2t)
iêTtQQ
Tjêt =0 t 2 f1::Mg
[u;v]= sR(r)P + ab
ô õ
![Page 11: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/11.jpg)
Dynamic Textures
Purpose
Model image intensity variations due to
1. Small geometric errors due to tracking
2. Non planarity of real surface
3. Non-rigidity of real object
4. Pose varying lighting effects
Non-geometric, mixing of spatial basis
![Page 12: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/12.jpg)
Spatial Basis Intro
1. Moving sine wave can be modeled:
2. Small image motion
I = sin(u+at) = sin(u) cos(at) + cos(u) sin(at) = sin(u)y1+cos(u)y2
I = I 0+@u@IÉ u+@v
@IÉ v
Spatially fixed basis
2 basis vectors 6 basis vectors
![Page 13: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/13.jpg)
Image Variability
Formally consider residual variation in an image stabilization problem
Optic flow type constraint
É I = I q(wê) à I (t)
É I = I (w+É w) à I w = I (f ) +@w@I (f )É wà I w = @w
@I (w)É w
![Page 14: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/14.jpg)
Structural Image Variability
Affine warp function
Corresponding image variability
Discretized for images
uwvw
ô õ
= W(p;a) =a3 a4a5 a6
ô õ
p +a1a2
ô õ
É I a = @u@I ;@v
@Iâ ã 1 0 ãu 0 ãv 00 1 0 ãu 0 ãv
ô õ y1...y6
2
4
3
5
= [B 1. . .B 6][y1; . . .;y6]T = Baya
É I a =P
i=16
@ai@I wÉ ai = @u
@I ;@v@Iâ ã @a1
@u ááá @a6@u
@a1@v ááá @a6
@v
" # É a1...É a6
2
4
3
5
![Page 15: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/15.jpg)
Composite Image variability
Similarily can show that composite image variability
Can be modeled as sum of basis
É I = É I s+É I d+É I n+É I l +É I e
É I = B sys+B dyd+B nyn+B lyl +É I e = By +É I e
Struct Depth Non-plan Light Res Err
![Page 16: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/16.jpg)
Example Lighting variation
![Page 17: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/17.jpg)
Statistical Image Variability
In practice image variability hard to compute from one image
Instead we use PCA to estimate image variability from a large sequence of images
This yields a transformed basis
Can estimate linear model J
In practice Delaunay triang & bi-linear model
É I = Bêyê
É yê = J É x
![Page 18: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/18.jpg)
Image variability comparison
Derivatives from one picture
Statistically estimated variability
![Page 19: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/19.jpg)
Implementation
Matlab for geometric modeling and prototyping
mexVision for tracking (30Hz frame rate)Hardware accelerated OpenGL for rendering
(2.8Hz in SW, 18Hz on GeForce 2)pthreads and pvm for parallel processing
MATLAB
meXVision OpenGL
![Page 20: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/20.jpg)
Hardware rendering
Unsigned basis
Scaling to 8 bit
Where
I w(t) = B+y(t) à Bày(t) + Iö
Bà+ =255B+ðà 1
Bàà =255Bàðà 1
yà =255à 1ðy
ð= diag(maxjBj)
![Page 21: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/21.jpg)
OpenGL==draw the meanBi ndTexture(Iö);DrawTr i angl es();==add basi s t exturesf or (each i)fSetBl endCoef f i ci ent (jyài(t)j);Bi ndTexture(Bà+i );i f (yài(t) > 0)SetBl endEquat i on(ADD);el se SetBl endEquat i on(SUBTRACT);DrawTr i angl es();Bi ndTexture(Bàài );i f (yài(t) > 0)SetBl endEquat i on(SUBTRACT);el se SetBl endEquat i on(ADD);DrawTr i angl es();
g
![Page 22: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/22.jpg)
Example Renderings
![Page 23: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/23.jpg)
Kinematic arm
![Page 24: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/24.jpg)
Geometric errors
static dynamic
![Page 25: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/25.jpg)
Geometric errors
static dynamic
![Page 26: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/26.jpg)
Geometric errors
DynamicStaticStatic texturing Dynamic
![Page 27: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/27.jpg)
Pixel error
Vertical jitter Horizontal jitter
Static texture 1.15 0.98
Dynamic texture 0.52 0.71
![Page 28: Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d5a5503460f94a3acb2/html5/thumbnails/28.jpg)
Conclusions
Coarse geometry tractable to estimateErrors from small geometric misalignments
compensated for using dynamic textureSystem runs on consumer PC with web cam
and game graphics cardApplications
Insert characters/objects into gamesVideo phone