Efficient High-Resolution Stereo Matching using Local Plane Sweeps

Sudipta N. Sinha, Daniel Scharstein, Richard Szeliski @ CVPR 2014

Yongho Shin

2

High-resolution images require long time for computing a disparity map– Complexity for general local methods for 2x size images

Problems

𝑂 (𝑊𝐻𝑁𝐷 )

x4

𝑂 (210𝑊𝐻𝑁𝐷)

3

Semi-global matching– Optimize following energy function

– NP-hard problem!!• Approximate methods operate in adequate computing time, but still

slow• Dynamic programming gives faster way, but erroneous result

– Instead do dynamic programming along many directions

– It cannot model second-order smoothness

Related works

𝑂 (𝑊𝐻𝐷 )

4

Efficient large-scale stereo matching– Stereo matching based on search space reduction

• Computation GCPs• Delaunay triangulation on GCPs• Matching on triangles with restricted range

Related works

5

VERY RELATED WORKSegment-Based Stereo Matching Using Belief Propagation

Initial matching– Any matching method can be used

Matching with a segmenta-tion

Initial matching

Extraction of reliable pixels

Extraction of model parameter

from each segment

Assignment of optimal parameter

for each segment by BP

Noisy result

Extraction of reliable pixels– Simple cross checking method is used– Occlusion region can be detected

Matching with a segmenta-tion

Initial matching

Extraction of reliable pixels

Extraction of model parameter

from each segment

Assignment of optimal parameter

for each segment by BP

Left image Right image

Left result Right result

Extraction of model param-eter from each segment– At each segment, a model param-

eter is extracted using reliable pixels and robust statistical tech-nique

– Add the parameter to a parameter set

Matching with a segmenta-tion

Initial matching

Extraction of reliable pixels

Extraction of model parameter

from each segment

Assignment of optimal parameter

for each segment by BP

Reliable pixels Segments

Extraction of model parame-ter from each segment– At each segment, a model parame-

ter is extracted using reliable pixels and robust statistical technique

– Add the parameter to a parameter set

Matching with a segmenta-tion

Initial matching

Extraction of reliable pixels

Extraction of model parameter

from each segment

Assignment of optimal parameter

for each segment by BP

Parameter Set

Parameter

Assignment of optimal pa-rameter for each segment by BP– Assign an optimal parameter for

each segment as total energy can be minimized

Matching with a segmenta-tion

Initial matching

Extraction of reliable pixels

Extraction of model parameter

from each segment

Assignment of optimal parameter

for each segment by BP

Parameter #29Parameter #29

Parameter Set

Matching with a segmentation

a : Initial disparity mapb : Interpolated resultc : Reliable pixel mapd : Result from a segmentation

a bc d

12

What they did– Make plane parameter by segment and initial disparity

map

– Find optimal plane parameters for each segment of the image

– Select optimal parameters by BP

Matching with a segmenta-tion

13

PROPOSED METHOD

14

What they do– Make plane parameter by feature points

– Find optimal plane parameters for each tiles of the im-age• Allowing objects having curved surface

– Select optimal parameters by SGM

Information for understand-ing

15

PROPOSED METHODHypothesis generation

16

Feature matching– By Harris corner keypoints and upright DAISY descriptors

– Matching only points along near epipolar line• Due to stereo matching• But, they allow small vertical misalignments

– First round• Initial set of matches are selected using the ratio test heuristic

– Second round• For obtaining more matched features• Horizontal search range is reduced using local estimates

Hypothesis generation

17

Vertical alignment– Correct for small vertical misalignments from errors in

rectification

– By fitting a global linear model using RANSAC with matched features

Hypothesis generation

𝑑𝑦=𝑎𝑦+𝑏

18

Disparity plane estimation– Cluster matched points and find plane parameters

• Find k number of planes

– Using variational approach used for mesh simplification• Graph based approach with priority queue

Hypothesis generation

19

PROPOSED METHODLocal plane sweeps

20

Plane for tiles having parallax– Because there are curved objects in the world– Hence, gives range of ±T pixels of parallax from plane– For each plane, investigate similarity among range 2T

• Optimize by SGM

Local plane sweep

21

Identifying in-range disparities– By disparity map, they give cost U

Local plane sweep

AD NCC JUMP

22

PROPOSED METHODProposal generation

23

Initial proposals– Find the planes with associated points within each tile

Online proposals– Find frequent plane parameter for each tile– Propagate the parameter to neighbors

Proposal generation

24

PROPOSED METHODGlobal optimization

We have– Plane parameters for each tile– Cost U– Energy function

–Power SGM!!

Global optimization

26

EXPERIMENTS

27

Quantitative results

28

Qualitative results