hongik university school of radio science & communication engineering visual information...
TRANSCRIPT
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
A Fast Cooperative Stereo AlgorithmA Fast Cooperative Stereo Algorithmusing 3D Moving Window andusing 3D Moving Window andParallel Processing TechniqueParallel Processing Technique
2003. 1. 92003. 1. 9
Young Ki Baik, Hyun Mok ChoYoung Ki Baik, Hyun Mok Cho° ° , Kyoung Mu Lee, Kyoung Mu LeeVisual Information Processing Lab. Hong-Ik Univ.Visual Information Processing Lab. Hong-Ik Univ.
Visual Information Processing Lab.Visual Information Processing Lab.School of Radio Science & Communication Engineering. Hong-Ik UniversitySchool of Radio Science & Communication Engineering. Hong-Ik University
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
ContentsContents
IntroductionIntroduction Stereo visionStereo vision Cooperative stereo algorithmCooperative stereo algorithm
Fast CS algorithmFast CS algorithm Moving window techniqueMoving window technique Parallel processing and optimization techniqueParallel processing and optimization technique
ResultResult
Conclusion & Future worksConclusion & Future works
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Stereo visionStereo vision
Extraction of the depth information from Extraction of the depth information from two or more imagestwo or more images
f
(x, y, z)M N
L R
xl
CrCl
xr
Focal length
Scene object point
b
P
Pl Pr
rx
lx
bfz
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Cooperative Stereo algorithmCooperative Stereo algorithm
C.S AlgorithmC.S Algorithm Idea (Marr & Poggio, 1976)Idea (Marr & Poggio, 1976)
• UniquenessUniqueness
• ContinuityContinuity
Embodiment (Zitnick and Kanade, 1999)Embodiment (Zitnick and Kanade, 1999)• Local support AreaLocal support Area
• Inhibition AreaInhibition Area
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Cooperative Stereo algorithmCooperative Stereo algorithm
Illustration of 3D disparity space Illustration of 3D disparity space
Disparity
Column
Current Element
Inhibition Element
Local support Area
Left Camera Right Camera
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Cooperative Stereo algorithmCooperative Stereo algorithm
Summary of algorithmSummary of algorithm Prepare a 3D array ( memory allocation )Prepare a 3D array ( memory allocation ) Set initial match values CSet initial match values C00
Iteratively update match values CIteratively update match values Cnn
Search maximum match value dSearch maximum match value d
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Cooperative Stereo algorithmCooperative Stereo algorithm
Initial ValueInitial Value 0C
),,,,(),,(0 dyxIIdyxC RightLeft
) ... SSD (SAD,Function Similarity Corr
imagesright andLeft , RightLeft II
2/1tanh1),,( ,, dyxII RightLeft
std
meandyxCorr RightLeft II
),,,,(
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Cooperative Stereo algorithmCooperative Stereo algorithm
3D local support area3D local support area
Inhibition FunctionInhibition Function
Update FunctionUpdate Function
),,(),,(),,(
dyxCdyxSdyx
nn
),,(
),,(),,(
),,(
dyxS
dyxSdyxR
dyxn
nn
AreaSupport Local
Element InhibitionConstant Inhibition
),,(*),,(),,( dyxRdyxCdyxC n01n
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Fast CS AlgorithmFast CS Algorithm
Moving Window Technique (2D)Moving Window Technique (2D)
y
x
Total sum of 2D window value
Initial value
Addition
Addition and sutraction
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Fast CS AlgorithmFast CS Algorithm
Moving Window Technique (2D)Moving Window Technique (2D)
y
x
Total sum of 2D window value
Initial value
Addition
Addition and sutraction
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Fast CS AlgorithmFast CS Algorithm
Moving Window Technique (2D)Moving Window Technique (2D)
y
x
Total sum of 2D window value
Initial value
Addition
Addition and sutraction
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Fast CS AlgorithmFast CS Algorithm
Moving Window Technique (2D)Moving Window Technique (2D)
y
x
Total sum of 2D window value
Initial value
Addition
Addition and sutraction
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Fast CS AlgorithmFast CS Algorithm
Moving Window Technique (3D)Moving Window Technique (3D)
xy
d
Total sum of 3D window value
Total sum of 2D window value
Addition
Addition andsubtraction
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Fast CS AlgorithmFast CS Algorithm
Moving Window Technique (3D)Moving Window Technique (3D)
xy
d
Total sum of 3D window value
Total sum of 2D window value
Addition
Addition andsubtraction
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Parallel processing TechniqueParallel processing Technique
SIMD (Single Instruction Multiple Data)SIMD (Single Instruction Multiple Data) SSE2 (Streaming SIMD Extensions2)SSE2 (Streaming SIMD Extensions2)
• 128-bit SIMD integer arithmetic operations.128-bit SIMD integer arithmetic operations.
• 128-bit SIMD double precision floating point operations128-bit SIMD double precision floating point operations
• Cache and memory management operationsCache and memory management operations
XMM0
XMM1
XMM2
XMM3
XMM4
XMM5
XMM6
XMM72 x Double
16 x BYTE
8 x WORD
4 x DWORD
2 x QWROD
Data type and XMM Register
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Parallel processing TechniqueParallel processing Technique
d direction redundancyd direction redundancy Multiple calculationMultiple calculation
• 4 operation at the same time4 operation at the same time
(floating point operations)(floating point operations)
OperationSource and Result data
d
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Fast CS AlgorithmFast CS Algorithm
EnvironmentEnvironment System : Pentium IV 1.4GhzSystem : Pentium IV 1.4Ghz Cache Memory : 256KbyteCache Memory : 256Kbyte
ConditionCondition Image : 384x288 TsukubaImage : 384x288 Tsukuba Disparity searching range : 16Disparity searching range : 16 Initial window size : 3x3Initial window size : 3x3 Iteration : 1Iteration : 1
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Fast CS AlgorithmFast CS Algorithm
Time checkTime check
30
ST
iteration one of timeTaking :T
iteration 30 of timeTotal :S
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
ResultResult
Local support window : Local support window : 3x3x3 3x3x3
Local support window : Local support window : 5x5x35x5x3
OriginalCS Algorithm
Moving Window
technique
Parallel Processing
Support region
2.100 (sec) 0.392 (sec) 0.063 (sec)
Inhibition region
And update
0.641 (sec) 0.133 (sec) 0.052 (sec)
Sum 2.741 (sec) 0.525 (sec) 0.115 (sec)
OriginalCS Algorithm
Moving Window
technique
Parallel Processing
Support region
5.582 (sec) 0.395 (sec) 0.064 (sec)
Inhibition region
And update
0.643 (sec) 0.134 (sec) 0.052 (sec)
Sum 6.225 (sec) 0.529 (sec) 0.116 (sec)
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
ResultResult
Local support window : Local support window : 7x7x3 7x7x3 Original
CS AlgorithmMoving
Window technique
Parallel Processing
Support region
10.623 (sec) 0.395 (sec) 0.064 (sec)
Inhibition region
And update
0.643 (sec) 0.134 (sec) 0.052 (sec)
Sum 11.266 (sec) 0.529 (sec) 0.116 (sec)
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
ResultResult
Time comparisonTime comparison
0
2
4
6
8
10
12
3x3x3 5x5x3 7x7x3Local support area size
Tim
e(S
ec)
Original CS Moving Window Parallel Processing
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
ResultResult
Tsukuba Image True disparity map
Initial disparity map
100 iteration result
HONGIK UNIVERSITY School of Radio Science & Communication Engineering
Visual Information Processing Lab
Hon
g-I
k U
niv
ers
ity
Hon
g-I
k U
niv
ers
ity
Sch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
S
ch
ool of
Rad
io S
cie
nce &
Com
mu
nic
ati
on
En
gin
eeri
ng
En
gin
eeri
ng
Conclusion & Future worksConclusion & Future works
ConclusionConclusion Fast Cooperative Stereo AlgorithmFast Cooperative Stereo Algorithm
• 3D Moving Window Technique3D Moving Window Technique
• Parallel Processing Technique Parallel Processing Technique
Future worksFuture works Removing redundancy for iterationRemoving redundancy for iteration Calculating OptimizationCalculating Optimization Usage of SIMD techniqueUsage of SIMD technique Application to other stereo algorithmApplication to other stereo algorithm