image-based rendering using hardware accelerated dynamic textures keith yerex dana cobzas martin...
Post on 21-Dec-2015
216 views
TRANSCRIPT
Image-Based Rendering using Hardware Accelerated
Dynamic Textures
Keith Yerex
Dana Cobzas Martin Jagersand
Motivation
Rendering
Traditional geometry based techniques:
detailed 3D model + texture
hard to achieve photorealism
Image-based models
non-geometric model from images
practically hard to apply
Challenges
Hard to generate detailed 3D models Texturing from images require very precise
alignment with the model
Rendering arbitrary views using IBM requires dense sample of the plenoptic function
IBR techniques don’t deal with dynamic scenes
Overview I1 It
…
= =
+ +
Warped texture
Motion params
(R1 a1 b1) …(Rt at bt)
Structure P
Texture basis
y1 … yt
Texture coeff
New pose
(R a b)
Training Model New view
uv
ô õ= RP + a
b
ô õ
I wt = Byt+ IöI wqt = I qt(W(ut;vt))
I wt =P
q=1Q I wqt
I t =P
q=1Q I qt
Structure from motion
Input – Structure from set of corresponding points tracked in a set of images
Assumptions Static scene Camera model
injective perspective weak perspective orthographic
Estimated modelprojective affine metric euclidean
Structure from motionTracked features
Structure from motion
algorithm
poses
structure
SFM algorithms
Few images, perspective camera, precise calibration
epipolar constraint
trilinear tensor
Long motion, affine or perspective structure
factorization methods
Metric structure
Weak perspective camera
Extension of Tomasi Kanade factorisation
algorithm
Extract affine structure Relation between the sffine structure and camera
coordinate frame Transform the structure into metric (unit pixel size)
Weak perspective projection
N points
Normalized with respect to centroid
Rank theorem
Factorization
uti = stiTtP i +atvti = stjTtP i +bt
W= RP + t1
Wê = RP where Wê =Wà t1
Wê = O1Î O2
Rê = O01
Pê= Î 0O02
Metric constraints
Extract motion parameters Eliminate scale Compute direction of camera axis k = i x j parameterize rotation with Euler angles
Model P ReprojectionPose x = (r,s,a,b)
R =RêQP =Qà 1Pê
iêTtQQ
Tiêt =jêTtQQ
Tjêt ( = s2t)
iêTtQQ
Tjêt =0 t 2 f1::Mg
[u;v]= sR(r)P + ab
ô õ
Dynamic Textures
Purpose
Model image intensity variations due to
1. Small geometric errors due to tracking
2. Non planarity of real surface
3. Non-rigidity of real object
4. Pose varying lighting effects
Non-geometric, mixing of spatial basis
Spatial Basis Intro
1. Moving sine wave can be modeled:
2. Small image motion
I = sin(u+at) = sin(u) cos(at) + cos(u) sin(at) = sin(u)y1+cos(u)y2
I = I 0+@u@IÉ u+@v
@IÉ v
Spatially fixed basis
2 basis vectors 6 basis vectors
Image Variability
Formally consider residual variation in an image stabilization problem
Optic flow type constraint
É I = I q(wê) à I (t)
É I = I (w+É w) à I w = I (f ) +@w@I (f )É wà I w = @w
@I (w)É w
Structural Image Variability
Affine warp function
Corresponding image variability
Discretized for images
uwvw
ô õ
= W(p;a) =a3 a4a5 a6
ô õ
p +a1a2
ô õ
É I a = @u@I ;@v
@Iâ ã 1 0 ãu 0 ãv 00 1 0 ãu 0 ãv
ô õ y1...y6
2
4
3
5
= [B 1. . .B 6][y1; . . .;y6]T = Baya
É I a =P
i=16
@ai@I wÉ ai = @u
@I ;@v@Iâ ã @a1
@u ááá @a6@u
@a1@v ááá @a6
@v
" # É a1...É a6
2
4
3
5
Composite Image variability
Similarily can show that composite image variability
Can be modeled as sum of basis
É I = É I s+É I d+É I n+É I l +É I e
É I = B sys+B dyd+B nyn+B lyl +É I e = By +É I e
Struct Depth Non-plan Light Res Err
Example Lighting variation
Statistical Image Variability
In practice image variability hard to compute from one image
Instead we use PCA to estimate image variability from a large sequence of images
This yields a transformed basis
Can estimate linear model J
In practice Delaunay triang & bi-linear model
É I = Bêyê
É yê = J É x
Image variability comparison
Derivatives from one picture
Statistically estimated variability
Implementation
Matlab for geometric modeling and prototyping
mexVision for tracking (30Hz frame rate)Hardware accelerated OpenGL for rendering
(2.8Hz in SW, 18Hz on GeForce 2)pthreads and pvm for parallel processing
MATLAB
meXVision OpenGL
Hardware rendering
Unsigned basis
Scaling to 8 bit
Where
I w(t) = B+y(t) à Bày(t) + Iö
Bà+ =255B+ðà 1
Bàà =255Bàðà 1
yà =255à 1ðy
ð= diag(maxjBj)
OpenGL==draw the meanBi ndTexture(Iö);DrawTr i angl es();==add basi s t exturesf or (each i)fSetBl endCoef f i ci ent (jyài(t)j);Bi ndTexture(Bà+i );i f (yài(t) > 0)SetBl endEquat i on(ADD);el se SetBl endEquat i on(SUBTRACT);DrawTr i angl es();Bi ndTexture(Bàài );i f (yài(t) > 0)SetBl endEquat i on(SUBTRACT);el se SetBl endEquat i on(ADD);DrawTr i angl es();
g
Example Renderings
Kinematic arm
Geometric errors
static dynamic
Geometric errors
static dynamic
Geometric errors
DynamicStaticStatic texturing Dynamic
Pixel error
Vertical jitter Horizontal jitter
Static texture 1.15 0.98
Dynamic texture 0.52 0.71
Conclusions
Coarse geometry tractable to estimateErrors from small geometric misalignments
compensated for using dynamic textureSystem runs on consumer PC with web cam
and game graphics cardApplications
Insert characters/objects into gamesVideo phone