csce 641 computer graphics: image-based modeling jinxiang chai
Post on 21-Dec-2015
221 views
TRANSCRIPT
![Page 1: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/1.jpg)
CSCE 641 Computer Graphics: Image-based Modeling
Jinxiang Chai
![Page 2: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/2.jpg)
Image-based modeling
Estimating 3D structure
Estimating motion, e.g., camera motion
Estimating lighting
Estimating surface model
![Page 3: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/3.jpg)
Traditional modeling and rendering
User input Texture map survey data
Geometry Reflectance Light source
Camera model
Images modeling rendering
For photorealism:
- Modeling is hard
- Rendering is slow
![Page 4: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/4.jpg)
Can we model and render this?
What do we want to do for this model?
![Page 5: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/5.jpg)
Image based modeling and rendering
Images user input range
scansModel Images
Image-based modeling
Image-based rendering
![Page 6: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/6.jpg)
Spectrum of IBMR
Images user input range
scans
Model
Images
Image based modeling
Image-based renderingGeometry+ Images
Geometry+ Materials
Images + Depth
Light field
Panoroma
Kinematics
Dynamics
Etc.
Camera + geometry
![Page 7: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/7.jpg)
Spectrum of IBMR
Images user input range
scans
Model
Images
Image based modeling
Image-based renderingGeometry+ Images
Geometry+ Materials
Images + Depth
Light field
Panoroma
Kinematics
Dynamics
Etc.
Camera + geometry
![Page 8: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/8.jpg)
Spectrum of IBMR
Images user input range
scans
Model
Images
Image based modeling
Image-based renderingGeometry+ Images
Geometry+ Materials
Images + Depth
Light field
Panoroma
Kinematics
Dynamics
Etc.
Camera + geometry
![Page 9: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/9.jpg)
Stereo reconstruction
Given two or more images of the same scene or object, compute a representation of its shape
How can we estimate camera parameters?
knownknowncameracamera
viewpointsviewpoints
![Page 10: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/10.jpg)
How can we estimate the camera parameters?
![Page 11: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/11.jpg)
Camera calibration
Augmented pin-hole camera - focal point, orientation
- focal length, aspect ratio, center, lens distortion
Known 3DKnown 3D
Classical calibration - 3D 2D
- correspondence
Camera calibration online resources
![Page 12: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/12.jpg)
Camera and calibration target
![Page 13: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/13.jpg)
Classical camera calibration
Known 3D coordinates and 2D coordinates - known 3D points on calibration targets
- find corresponding 2D points in image using feature detection
algorithm
![Page 14: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/14.jpg)
Camera parameters
u0
v0
100-sy0
sx аuv1
Perspective proj. View trans.Viewport proj.
Known 3D coords and 2D coordsKnown 3D coords and 2D coords
![Page 15: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/15.jpg)
Camera parameters
u0
v0
100-sy0
sx аuv1
Perspective proj. View trans.Viewport proj.
Known 3D coords and 2D coordsKnown 3D coords and 2D coords
Intrinsic camera parameters (5 parameters)
extrinsic camera parameters (6 parameters)
![Page 16: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/16.jpg)
Camera matrix
Fold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into acamera matrix
M = K [R | t ]
(put 1 in lower r.h. corner for 11 d.o.f.)
![Page 17: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/17.jpg)
Camera matrix calibration
Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)
![Page 18: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/18.jpg)
Camera matrix calibration
Linear regression:• Bring denominator over, solve set of (over-determined) linear
equations. How?
![Page 19: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/19.jpg)
Camera matrix calibration
Linear regression:• Bring denominator over, solve set of (over-determined) linear
equations. How?
• Least squares (pseudo-inverse) - 11 unknowns (up to scale) - 2 equations per point (homogeneous coordinates) - 6 points are sufficient
![Page 20: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/20.jpg)
Nonlinear camera calibration
Perspective projection:
1100
0
1 3
2
1
3
2
1
0
0
i
i
i
T
T
T
y
x
i
i
z
y
x
t
t
t
r
r
r
vf
uf
v
u
![Page 21: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/21.jpg)
Nonlinear camera calibration
Perspective projection:
1100
0
1 3
2
1
3
2
1
0
0
i
i
i
T
T
T
y
x
i
i
z
y
x
t
t
t
r
r
r
vf
uf
v
u
K R T P
![Page 22: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/22.jpg)
Nonlinear camera calibration
Perspective projection:
2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters:
1100
0
1 3
2
1
3
2
1
0
0
i
i
i
T
T
T
y
x
i
i
z
y
x
t
t
t
r
r
r
vf
uf
v
u
K R T P
33
32302
33
30213021
)(
)(
tPr
ttfPrvrfv
tPr
tuttfPrurrfu
T
yTT
yi
Tx
TTTx
i
![Page 23: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/23.jpg)
Nonlinear camera calibration
Perspective projection:
2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters:
1100
0
1 3
2
1
3
2
1
0
0
i
i
i
T
T
T
y
x
i
i
z
y
x
t
t
t
r
r
r
vf
uf
v
u
33
32302
33
30213021
)(
)(
tPr
ttfPrvrfv
tPr
tuttfPrurrfu
T
yTT
yi
Tx
TTTx
i
K
);,,( iPTRKf
);,,( iPTRKg
R T P
![Page 24: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/24.jpg)
Multiple calibration images
Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M
for i=1,…,N
This can be formulated as a nonlinear optimization problem:
);,,(
);,,(
ijjji
ijjji
PTRKgv
PTRKfu
M
j
N
iijj
jiijj
ji PTRKgvPTRKfu
1 1
22 ));,,(());,,((
![Page 25: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/25.jpg)
Multiple calibration images
Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N
This can be formulated as a nonlinear optimization problem:
);,,(
);,,(
ijjji
ijjji
PTRKgv
PTRKfu
M
j
N
iijj
jiijj
ji PTRKgvPTRKfu
1 1
22 ));,,(());,,((
Solve the optimization using nonlinear optimization techniques:
- Gauss-newton
- Levenberg-Marquardt
![Page 26: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/26.jpg)
Nonlinear approach
Advantages:• can solve for more than one camera pose at a time
• fewer degrees of freedom than linear approach
• Standard technique in photogrammetry, computer vision, computer graphics
- [Tsai 87] also estimates lens distortions (freeware @ CMU)http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/v-source.html
Disadvantages:• more complex update rules
• need a good initialization (recover K [R | t] from M)
![Page 27: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/27.jpg)
Camera Calibration
Public calibration toolbox- http://research.microsoft.com/en-us/um/people/zhang/Calib/
- http://www.vision.caltech.edu/bouguetj/calib_doc/
![Page 28: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/28.jpg)
How can we estimate the camera parameters?
![Page 29: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/29.jpg)
Application: camera calibration for sports video
[Farin et. Al]
images Court model
![Page 30: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/30.jpg)
Stereo matching
Given two or more images of the same scene or object as well as their camera parameters, how to compute a representation of its shape?
What are some possible representations for shapes?• depth maps
• volumetric models
• 3D surface models
• planar (or offset) layers
![Page 31: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/31.jpg)
Outline
Stereo matching - Traditional stereo
- Active stereo
Volumetric stereo - Visual hull
- Voxel coloring
- Space carving
![Page 32: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/32.jpg)
Stereo matching• 11.1, 11.2,.11.3,11.5 in Sezliski book
• D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.International Journal of Computer Vision, 47(1/2/3):7-42, April-June 2002.
Readings
![Page 33: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/33.jpg)
Stereo
scene pointscene point
optical centeroptical center
image planeimage plane
![Page 34: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/34.jpg)
Stereo
Basic Principle: Triangulation• Gives reconstruction as intersection of two rays• Requires
> calibration
> point correspondence
![Page 35: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/35.jpg)
Stereo correspondence
Determine Pixel Correspondence• Pairs of points that correspond to same scene point
Epipolar Constraint• Reduces correspondence problem to 1D search along conjugate
epipolar lines• Java demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html
epipolar lineepipolar lineepipolar lineepipolar lineepipolar plane
![Page 36: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/36.jpg)
Stereo image rectification
![Page 37: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/37.jpg)
Stereo image rectification
• reproject image planes onto a commonplane parallel to the line between optical centers
• pixel motion is horizontal after this transformation• two homographies (3x3 transform), one for each
input image reprojection C. Loop and Z. Zhang. Computing Rectifying Homographies
for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
![Page 38: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/38.jpg)
Rectification
Original image pairs
Rectified image pairs
![Page 39: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/39.jpg)
Stereo matching algorithms
Match Pixels in Conjugate Epipolar Lines• Assume brightness constancy
• This is a tough problem
• Numerous approaches> A good survey and evaluation: http://www.middlebury.edu/stereo/
![Page 40: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/40.jpg)
Your basic stereo algorithm
For each epipolar line
For each pixel in the left image• compare with every pixel on same epipolar line in right image
• pick pixel with minimum matching cost
Improvement: match windows• This should look familiar.. (normalized cross correlation or SSD)• Can use Lukas-Kanade or discrete search (latter more common)
![Page 41: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/41.jpg)
Window size
• Smaller window+
-
• Larger window+
-
W = 3 W = 20
Effect of window size
![Page 42: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/42.jpg)
More constraints?
We can enforce more constraints to reduce matching ambiguity - smoothness constraints: computed disparity at a pixel
should be consistent with neighbors in a surrounding window.
- uniqueness constraints: the matching needs to be bijective
- ordering constraints: e.g., computed disparity at a pixel
should not be larger than the disparity of its right neighbor pixel by
more than one pixel.
![Page 43: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/43.jpg)
Stereo results
Ground truthScene
• Data from University of Tsukuba
• Similar results on other images without ground truth
![Page 44: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/44.jpg)
Results with window search
Window-based matching(best window size)
Ground truth
![Page 45: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/45.jpg)
Better methods exist...
A better methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
![Page 46: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/46.jpg)
More recent development
High-Quality Single-Shot Capture of Facial Geometry [siggraph 2010, project website] - capture high-fidelity facial geometry from multiple cameras
- pairwise stereo reconstruction between neighboring cameras
- hallucinate facial details
![Page 47: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/47.jpg)
More recent development
High Resolution Passive Facial Performance Capture [siggraph 2010, project website] - capture dynamic facial geometry from multiple video cameras
- spatial stereo reconstruction for every frame
- building temporal correspondences across the entire sequence
![Page 48: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/48.jpg)
Stereo reconstruction pipeline
Steps• Calibrate cameras
• Rectify images
• Compute disparity
• Estimate depth
![Page 49: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/49.jpg)
• Camera calibration errors
• Poor image resolution
• Occlusions
• Violations of brightness constancy (specular reflections)
• Large motions
• Low-contrast image regions
Stereo reconstruction pipeline
Steps• Calibrate cameras
• Rectify images
• Compute disparity
• Estimate depth
What will cause errors?
![Page 50: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/50.jpg)
Outline
Stereo matching - Traditional stereo
- Active stereo
Volumetric stereo - Visual hull
- Voxel coloring
- Space carving
![Page 51: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/51.jpg)
Active stereo with structured light
Project “structured” light patterns onto the object• simplifies the correspondence problem
camera 2
camera 1
projector
camera 1
projector
Li Zhang’s one-shot stereo
![Page 52: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/52.jpg)
Active stereo with structured light
![Page 53: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/53.jpg)
Laser scanning
Optical triangulation• Project a single stripe of laser light• Scan it across the surface of the object• This is a very precise version of structured light scanning
Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/
![Page 54: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/54.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
![Page 55: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/55.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
![Page 56: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/56.jpg)
RGBD Sensors
2008 2010 2013
Lower size/cost with better accuracy
![Page 57: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/57.jpg)
Kinect Sensor
![Page 58: CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai](https://reader035.vdocument.in/reader035/viewer/2022062714/56649d595503460f94a39973/html5/thumbnails/58.jpg)
Kinect Sensor