photorealistic scene reconstruction by voxel , 1999, pp...
TRANSCRIPT
Mul
tivie
wst
ereo C
MU
’s 3
D R
oom
Rea
ding
s (O
ptio
nal)
•S
. M. S
eitz
and
C. R
. Dye
r, P
hoto
real
istic
Sce
ne R
econ
stru
ctio
n by
Vox
el
Col
orin
g, In
tern
atio
nal J
ourn
al o
f Com
pute
r Vis
ion,
35(
2), 1
999,
pp.
151
-173
.
Cho
osin
g th
e B
asel
ine
wid
th o
f a
pixe
l
all o
f the
sepo
ints
pro
ject
to th
e sa
me
pair
of p
ixel
s
Wha
t’s th
e op
timal
bas
elin
e?•
Too
smal
l: la
rge
dept
h er
ror
•To
o la
rge:
diff
icul
t sea
rch
prob
lem
Larg
e B
asel
ine
Larg
e B
asel
ine
Smal
l Bas
elin
eSm
all B
asel
ine
The
Effe
ct o
f Bas
elin
e on
Dep
th E
stim
atio
n
Mul
tibas
elin
eS
tere
o
Bas
ic A
ppro
ach
•C
hoos
e a
refe
renc
e vi
ew•
Use
you
r fav
orite
ste
reo
algo
rithm
BU
T>
repl
ace
two-
view
SS
D w
ith S
SD
ove
r all
base
lines
Lim
itatio
ns•
Mus
t cho
ose
a re
fere
nce
view
(bad
)•
Vis
ibili
ty!
The
glob
al v
isib
ility
prob
lem
Whi
ch p
oint
s ar
e vi
sibl
e in
whi
ch im
ages
?
Know
n Sc
ene
Know
n Sc
ene
Forw
ard
Vis
ibili
tykn
own
scen
eIn
vers
e V
isib
ility
know
n im
ages
Unk
now
n Sc
ene
Unk
now
n Sc
ene
Vol
umet
ric s
tere
o
Scen
e Vo
lum
eSc
ene
Volu
me
VV Inpu
t Im
ages
Inpu
t Im
ages
(Cal
ibra
ted)
(Cal
ibra
ted)
Goa
l:
Goa
l: D
eter
min
e oc
cupa
ncy,
“co
lor”
of p
oint
s in
VD
eter
min
e oc
cupa
ncy,
“co
lor”
of p
oint
s in
V
Dis
cret
e fo
rmul
atio
n: V
oxel
Col
orin
g
Dis
cret
ized
D
iscr
etiz
ed
Scen
e Vo
lum
eSc
ene
Volu
me
Inpu
t Im
ages
Inpu
t Im
ages
(Cal
ibra
ted)
(Cal
ibra
ted)
Goa
l: A
ssig
n R
GB
A v
alue
s to
vox
els
in V
phot
o-co
nsis
tent
with
imag
es
Com
plex
ity a
nd c
ompu
tabi
lity
Dis
cret
ized
D
iscr
etiz
ed
Scen
e Vo
lum
eSc
ene
Volu
me
N
voxe
lsN
vo
xels
C
colo
rsC
co
lors
33
All S
cene
s (C
N3 )
Phot
o-Co
nsis
tent
Scen
es
True
Scen
e
Issu
es
Theo
retic
al Q
uest
ions
•Id
entif
y cl
ass
of a
llph
oto-
cons
iste
nt s
cene
s
Pra
ctic
al Q
uest
ions
•H
ow d
o w
e co
mpu
te p
hoto
-con
sist
ent m
odel
s?
Vox
el c
olor
ing
solu
tions
1. C
=2 (s
hape
from
silh
ouet
tes)
•V
olum
e in
ters
ectio
n [B
aum
gart
1974
]>
For m
ore
info
: R
apid
octre
eco
nstru
ctio
n fro
m im
age
sequ
ence
s.R
.Sze
liski
, C
VGIP
: Im
age
Und
erst
andi
ng, 5
8(1)
:23-
32, J
uly
1993
. (th
is p
aper
is a
ppar
ently
no
t ava
ilabl
e on
line)
2. C
unc
onst
rain
ed, v
iew
poin
t con
stra
ints
•V
oxel
col
orin
g al
gorit
hm [S
eitz
& D
yer 9
7]
3. G
ener
al C
ase
•S
pace
car
ving
[Kut
ulak
os &
Sei
tz 9
8]
Rec
onst
ruct
ion
from
Silh
ouet
tes
(C =
2)
Bina
ry I
mag
esBi
nary
Im
ages
App
roac
h:
•P
roje
ctea
ch s
ilhou
ette
•In
ters
ect p
roje
cted
vol
umes
Vol
ume
inte
rsec
tion
Rec
onst
ruct
ion
Con
tain
s th
e Tr
ue S
cene
•In
the
limit
(all
view
s) g
et v
isua
l hul
l>
Com
plem
ent o
f all
lines
that
don
’t in
ters
ect S
Vox
el a
lgor
ithm
for v
olum
e in
ters
ectio
n
Col
or v
oxel
bla
ck if
on
silh
ouet
te in
eve
ry im
age
•fo
r M im
ages
, N3
voxe
ls•
Don
’t ha
ve to
sea
rch
2N3
poss
ible
sce
nes!
O(M
N3 )
,
Pro
perti
es o
f Vol
ume
Inte
rsec
tion
Pro
s •E
asy
to im
plem
ent,
fast
•A
ccel
erat
ed v
ia o
ctre
es[S
zelis
ki19
93]
Con
s •N
o co
ncav
ities
•R
econ
stru
ctio
n is
not
pho
to-c
onsi
sten
t•
Req
uire
s id
entif
icat
ion
of s
ilhou
ette
s
Vox
el C
olor
ing
Sol
utio
ns
1. C
=2 (s
ilhou
ette
s)•
Vol
ume
inte
rsec
tion
[Bau
mga
rt19
74]
2. C
unc
onst
rain
ed, v
iew
poin
t con
stra
ints
•V
oxel
col
orin
g al
gorit
hm [S
eitz
& D
yer 9
7]>
For m
ore
info
: ht
tp://
ww
w.c
s.w
ashi
ngto
n.ed
u/ho
mes
/sei
tz/p
aper
s/ijc
v99.
3. G
ener
al C
ase
•S
pace
car
ving
[Kut
ulak
os &
Sei
tz 9
8]
Vox
el C
olor
ing
App
roac
h
1.
Ch
oose
vox
el1
. C
hoo
se v
oxel
2.
Pro
ject
an
d co
rrel
ate
2.
Pro
ject
an
d co
rrel
ate
3.
3.
Col
or if
con
sist
ent
Col
or if
con
sist
ent
(sta
ndar
d de
viat
ion
of p
ixel
co
lors
bel
ow th
resh
old)
Vis
ibili
ty P
robl
em:
Vis
ibili
ty P
robl
em:
in w
hich
imag
es is
eac
h vo
xel v
isib
le?
in w
hich
imag
es is
eac
h vo
xel v
isib
le?
Dep
th O
rder
ing:
vis
it oc
clud
ers
first
!
Laye
rsLa
yers
Scen
eSc
ene
Trav
ersa
lTr
aver
sal
Con
ditio
n:
Con
ditio
n: d
epth
ord
er is
the
dept
h or
der i
s th
e sa
me
for a
ll in
put v
iew
ssa
me
for a
ll in
put v
iew
s
Pan
oram
ic D
epth
Ord
erin
g
•C
amer
as o
rient
ed in
man
y di
ffere
nt d
irect
ions
•P
lana
r dep
th o
rder
ing
does
not
app
ly
Pan
oram
ic D
epth
Ord
erin
g
Lay
ers r
adia
te o
utw
ards
from
cam
eras
Lay
ers r
adia
te o
utw
ards
from
cam
eras
Pan
oram
ic L
ayer
ing
Lay
ers r
adia
te o
utw
ards
from
cam
eras
Lay
ers r
adia
te o
utw
ards
from
cam
eras
Pan
oram
ic L
ayer
ing
Lay
ers r
adia
te o
utw
ards
from
cam
eras
Lay
ers r
adia
te o
utw
ards
from
cam
eras
Com
patib
le C
amer
a C
onfig
urat
ions
Dep
th-O
rder
Con
stra
int
•S
cene
out
side
con
vex
hull
of c
amer
a ce
nter
s
Out
war
d-Lo
okin
gC
amer
a in
side
sce
neIn
war
d-Lo
okin
gC
amer
a ab
ove
scen
eca
mer
as a
bove
sce
neca
mer
as in
side
sce
ne
Cal
ibra
ted
Imag
e A
cqui
sitio
n
Sele
cted
Din
osau
r Im
ages
Sele
cted
Din
osau
r Im
ages
Cal
ibra
ted
Turn
tabl
e36
0° ro
tatio
n (2
1 im
ages
)
Sele
cted
Flo
wer
Imag
esSe
lect
ed F
low
er Im
ages
Vox
el C
olor
ing
Res
ults
(Vid
eo)
Din
osau
r R
econ
stru
ctio
nD
inos
aur
Rec
onst
ruct
ion
72
K v
oxel
s co
lore
d7
2 K
vox
els
colo
red
7.6
M v
oxel
s te
sted
7.6
M v
oxel
s te
sted
7 m
in. t
o co
mpu
te
7 m
in. t
o co
mpu
te
on a
25
0M
Hz
SGI
Flow
er R
econ
stru
ctio
nFl
ower
Rec
onst
ruct
ion
70
K v
oxel
s co
lore
d7
0 K
vox
els
colo
red
7.6
M v
oxel
s te
sted
7.6
M v
oxel
s te
sted
7 m
in. t
o co
mpu
te
7 m
in. t
o co
mpu
te
on a
25
0M
Hz
SGI
on a
25
0M
Hz
SGI
on a
25
0M
Hz
SGI
Lim
itatio
ns o
f Dep
th O
rder
ing
A v
iew
-inde
pend
ent d
epth
ord
er m
ay n
ot e
xist
pq
Nee
d m
ore
pow
erfu
l gen
eral
-cas
e al
gorit
hms
•U
ncon
stra
ined
cam
era
posi
tions
•U
ncon
stra
ined
sce
ne g
eom
etry
/topo
logy
Vox
el C
olor
ing
Sol
utio
ns
1. C
=2 (s
ilhou
ette
s)•
Vol
ume
inte
rsec
tion
[Bau
mga
rt19
74]
2. C
unc
onst
rain
ed, v
iew
poin
t con
stra
ints
•V
oxel
col
orin
g al
gorit
hm [S
eitz
& D
yer 9
7]
3. G
ener
al C
ase
•S
pace
car
ving
[Kut
ulak
os &
Sei
tz 9
8]>
For m
ore
info
: ht
tp://
ww
w.c
s.w
ashi
ngto
n.ed
u/ho
mes
/sei
tz/p
aper
s/ku
tu-ij
cv00
Spa
ce C
arvi
ng A
lgor
ithm
Spa
ce C
arvi
ng A
lgor
ithm
Imag
e 1
Imag
e N
…...
•In
itial
ize
to a
vol
ume
V c
onta
inin
g th
e tru
e sc
ene
•R
epea
t unt
il co
nver
genc
e
•C
hoos
e a
voxe
l on
the
curre
nt s
urfa
ce
•C
arve
if n
ot p
hoto
-con
sist
ent
•P
roje
ct to
vis
ible
inpu
t im
ages
Con
verg
ence
Con
sist
ency
Pro
perty
•Th
e re
sulti
ng s
hape
is p
hoto
-con
sist
ent
>al
l inc
onsi
sten
t poi
nts
are
rem
oved
Con
verg
ence
Pro
perty
•C
arvi
ng c
onve
rges
to a
non
-em
pty
shap
e>
a po
int o
n th
e tru
e sc
ene
is n
ever
rem
oved
p
Vis
ibili
ty le
mm
a
Let p
be
a po
int o
n V
’s s
urfa
ce, S
urf(V
), an
d le
t Vis
v(p) b
e th
e co
llect
ion
of in
put i
mag
es in
whi
ch V
doe
s no
t occ
lude
p. I
f V’,
a su
bset
of V
, is
a sh
ape
that
als
o ha
s p
on it
s su
rface
, Vis
v(p) i
s a
subs
et o
f Vis
v’(p
).
pV
’
Vc 2
c 3
c 1
c 4
Non
-pho
to-c
onsi
sten
cy le
mm
a
Let p
, whi
ch is
in S
urf(V
), is
not
pho
to-c
onsi
sten
t with
a s
ubse
t of
Vis
v(p),
it is
not
pho
to-c
onsi
sten
t with
the
entir
e V
isv(p
).
pV
c 2
c 3
c 1
c 4
Whi
ch s
hape
do
you
get?
Tru
e Sc
ene
Tru
e Sc
ene
VV
Phot
o H
ull
Phot
o H
ull
VV
The
Pho
to H
ulli
s th
e U
NIO
N o
f all
phot
o-co
nsis
tent
sce
nes
in V
•It
is a
pho
to-c
onsi
sten
t sce
ne re
cons
truct
ion
•Ti
ghte
st p
ossi
ble
boun
d on
the
true
scen
e
Spa
ce C
arvi
ng A
lgor
ithm
The
Bas
ic A
lgor
ithm
is U
nwie
ldy
•C
ompl
ex u
pdat
e pr
oced
ure
Alte
rnat
ive:
Mul
ti-P
ass
Pla
ne S
wee
p•
Effi
cien
t, ca
n us
e te
xtur
e-m
appi
ng h
ardw
are
•C
onve
rges
qui
ckly
in p
ract
ice
•E
asy
to im
plem
ent
Res
ults
Alg
orith
m
Mul
ti-P
ass
Pla
ne S
wee
p•
Swee
p pl
ane
in e
ach
of 6
prin
cipl
e di
rect
ions
•C
onsi
der c
amer
as o
n on
ly o
ne s
ide
of p
lane
•R
epea
t unt
il co
nver
genc
e
True
Sce
neR
econ
stru
ctio
n
Mul
ti-P
ass
Pla
ne S
wee
p•
Swee
p pl
ane
in e
ach
of 6
prin
cipl
e di
rect
ions
•C
onsi
der c
amer
as o
n on
ly o
ne s
ide
of p
lane
•R
epea
t unt
il co
nver
genc
e
Mul
ti-P
ass
Pla
ne S
wee
p•
Swee
p pl
ane
in e
ach
of 6
prin
cipl
e di
rect
ions
•C
onsi
der c
amer
as o
n on
ly o
ne s
ide
of p
lane
•R
epea
t unt
il co
nver
genc
e
Mul
ti-P
ass
Pla
ne S
wee
p•
Swee
p pl
ane
in e
ach
of 6
prin
cipl
e di
rect
ions
•C
onsi
der c
amer
as o
n on
ly o
ne s
ide
of p
lane
•R
epea
t unt
il co
nver
genc
e
Mul
ti-P
ass
Pla
ne S
wee
p•
Swee
p pl
ane
in e
ach
of 6
prin
cipl
e di
rect
ions
•C
onsi
der c
amer
as o
n on
ly o
ne s
ide
of p
lane
•R
epea
t unt
il co
nver
genc
e
Mul
ti-P
ass
Pla
ne S
wee
p•
Swee
p pl
ane
in e
ach
of 6
prin
cipl
e di
rect
ions
•C
onsi
der c
amer
as o
n on
ly o
ne s
ide
of p
lane
•R
epea
t unt
il co
nver
genc
e
Spa
ce C
arvi
ng R
esul
ts:
Afri
can
Vio
let
Inpu
t Im
age
(1 o
f 4
5)
Rec
onst
ruct
ion
Rec
onst
ruct
ion
Rec
onst
ruct
ion
Spa
ce C
arvi
ng R
esul
ts:
Han
d
Inpu
t Im
age
(1 o
f 1
00
)
Vie
ws
of R
econ
stru
ctio
n
Hou
se W
alkt
hrou
gh
24 re
nder
ed in
put v
iew
s fro
m in
side
and
outs
ide
Spa
ce C
arvi
ng R
esul
ts:
Hou
se
Inpu
t Im
age
Inpu
t Im
age
(tru
e sc
ene)
Rec
onst
ruct
ion
Rec
onst
ruct
ion
37
0,0
00
vox
els
(tru
e sc
ene)
37
0,0
00
vox
els
Spa
ce C
arvi
ng R
esul
ts:
Hou
se
Inpu
t Im
age
Inpu
t Im
age
(tru
e sc
ene)
Rec
onst
ruct
ion
Rec
onst
ruct
ion
37
0,0
00
vox
els
(tru
e sc
ene)
37
0,0
00
vox
els
Spa
ce C
arvi
ng R
esul
ts:
Hou
se
New
Vie
w (
tru
e sc
ene)
New
Vie
w (
tru
e sc
ene)
Rec
onst
ruct
ion
Rec
onst
ruct
ion
New
Vie
wN
ew V
iew
(tru
e sc
ene)
(tru
e sc
ene)
Rec
onst
ruct
ion
Rec
onst
ruct
ion
(wit
h n
ew in
put
view
)R
econ
stru
ctio
nR
econ
stru
ctio
n(w
ith
new
inpu
t vi
ew)
Oth
er F
eatu
res
Coa
rse-
to-fi
ne R
econ
stru
ctio
n•
Rep
rese
nt s
cene
as
octre
e•
Rec
onst
ruct
low
-res
mod
el fi
rst,
then
refin
e
Har
dwar
e-A
ccel
erat
ion
•U
se te
xtur
e-m
appi
ng to
com
pute
vox
el p
roje
ctio
ns•
Pro
cess
vox
els
an e
ntire
pla
ne a
t a ti
me
Lim
itatio
ns•
Nee
d to
acq
uire
cal
ibra
ted
imag
es•
Res
trict
ion
to s
impl
e ra
dian
ce m
odel
s•
Bia
s to
war
d m
axim
al (f
at) r
econ
stru
ctio
ns•
Tran
spar
ency
not
sup
porte
d
Oth
er A
ppro
ache
sLe
vel-S
et M
etho
ds [
Faug
eras
& K
eriv
en19
98]
•E
volv
e im
plic
it fu
nctio
n by
sol
ving
PD
E’s
Pro
babi
listic
Vox
el R
econ
stru
ctio
n [D
eBon
et&
Vio
la 1
999]
, [B
road
hurs
tet a
l. 20
01]
•S
olve
for v
oxel
unc
erta
inty
(als
o tra
nspa
renc
y)
Tran
spar
ency
and
Mat
ting
[Sze
liski
& G
olla
nd19
98]
•C
ompu
te v
oxel
s w
ith a
lpha
-cha
nnel
Max
Flo
w/M
in C
ut
[Roy
& C
ox 1
998]
•G
raph
theo
retic
form
ulat
ion
Mes
h-B
ased
Ste
reo
[Fua
& L
ecle
rc19
95],
[Zha
ng &
Sei
tz 2
001]
•M
esh-
base
d bu
t sim
ilar c
onsi
sten
cy fo
rmul
atio
n
Virt
ualiz
ed R
ealit
y [N
aray
an, R
ande
r, K
anad
e 19
98]
•P
erfo
rm s
tere
o 3
imag
es a
t a ti
me,
mer
ge re
sults
Bib
liogr
aphy
Vol
ume
Inte
rsec
tion
•M
artin
& A
ggar
wal
, “V
olum
etric
des
crip
tion
of o
bjec
ts fr
om m
ultip
le v
iew
s”, T
rans
. Pat
tern
A
naly
sis
and
Mac
hine
Inte
lligen
ce,
5(2)
, 199
1, p
p. 1
50-1
58.
•Sz
elis
ki, “
Rap
id O
ctre
eC
onst
ruct
ion
from
Imag
e S
eque
nces
”, C
ompu
ter V
isio
n, G
raph
ics,
an
d Im
age
Pro
cess
ing:
Imag
e U
nder
stan
ding
, 58(
1), 1
993,
pp.
23-
32.
Vox
el C
olor
ing
and
Spa
ce C
arvi
ng•
Seitz
& D
yer,
“Pho
tore
alis
tic S
cene
Rec
onst
ruct
ion
by V
oxel
Col
orin
g”, P
roc.
Com
pute
r Vis
ion
and
Pat
tern
Rec
ogni
tion
(CV
PR
), 19
97, p
p. 1
067-
1073
.•
Seitz
& K
utul
akos
, “P
leno
ptic
Imag
e E
ditin
g”,
Pro
c. In
t. C
onf.
on C
ompu
ter V
isio
n (IC
CV
), 19
98, p
p. 1
7-24
.•
Kutu
lako
s &
Sei
tz, “
A T
heor
y of
Sha
pe b
y S
pace
Car
ving
”, P
roc.
ICC
V, 1
998,
pp.
307
-314
.
Bib
liogr
aphy
Rel
ated
Ref
eren
ces
•Bo
lles,
Bak
er, a
nd M
arim
ont,
“Epi
pola
r-P
lane
Imag
e A
naly
sis:
An
App
roac
h to
Det
erm
inin
g S
truct
ure
from
Mot
ion”
, Int
erna
tiona
l Jou
rnal
of C
ompu
ter V
isio
n, v
ol1,
no
1, 1
987,
pp.
7-5
5.•
DeB
onet
& V
iola
, “P
oxel
s: P
roba
bilis
tic V
oxel
ized
Volu
me
Rec
onst
ruct
ion ”
, Pro
c. In
t. C
onf.
on
Com
pute
r Vis
ion
(ICC
V) 1
999.
•Br
oadh
urst
, Dru
mm
ond,
and
Cip
olla
, "A
Pro
babi
listic
Fra
mew
ork
for S
pace
Car
ving
“, In
tern
atio
nal C
onfe
renc
e of
Com
pute
r Vis
ion
(ICC
V), 2
001,
pp.
388
-393
.•
Faug
eras
& K
eriv
en, “
Var
iatio
nalp
rinci
ples
, sur
face
evo
lutio
n, P
DE
's, l
evel
set
met
hods
and
the
ster
eo p
robl
em",
IEE
E T
rans
. on
Imag
e P
roce
ssin
g, 7
(3),
1998
, pp.
336
-344
.•
Szel
iski
& G
olla
nd, “
Ste
reo
Mat
chin
g w
ith T
rans
pare
ncy
and
Mat
ting”
, Pro
c. In
t. C
onf.
on
Com
pute
r Vis
ion
(ICC
V), 1
998,
517
-524
.•
Roy
& C
ox, “
A M
axim
um-F
low
For
mul
atio
n of
the
N-c
amer
a S
tere
o C
orre
spon
denc
e P
robl
em”,
Pro
c. IC
CV
, 199
8, p
p. 4
92-4
99.
•Fu
a &
Lec
lerc
, “O
bjec
t-cen
tere
d su
rface
reco
nstru
ctio
n: C
ombi
ning
mul
ti-im
age
ster
eo a
nd
shad
ing"
, Int
erna
tiona
l Jou
rnal
of C
ompu
ter V
isio
n, 1
6, 1
995,
pp.
35-
56.
•N
aray
anan
, Ran
der,
& K
anad
e, “C
onst
ruct
ing
Virt
ual W
orld
s U
sing
Den
se S
tere
o”, P
roc.
ICC
V,
1998
, pp.
3-1
0.
Sum
mar
y
Thin
gs to
take
aw
ay fr
om th
is le
ctur
e•
Bas
elin
e tra
deof
f•
Mul
tibas
elin
est
ereo
app
roac
h•
Vox
el c
olor
ing
prob
lem
•V
olum
e in
ters
ectio
n al
gorit
hm•
Vox
el c
olor
ing
algo
rithm
•S
pace
car
ving
alg
orith
m