single depth imaging - mit media labweb.media.mit.edu/~achoo/iccvtoftutorial/prelimslides/...single...

Ayush Bhandari [email protected]

Massachusetts Institute of Technology 77 Massachusetts Avenue, Building E14–474A Cambridge, Massachusetts 02139–4307

Phone 617–715–4420 Web www.mit.edu/~ayush

Ayush Bhandari [email protected]

Massachusetts Institute of Technology 77 Massachusetts Avenue, Building E14–474A Cambridge, Massachusetts 02139–4307

Phone 617–715–4420 Web www.mit.edu/~ayush

www.mit.edu/~ayushAyush Bhandari

Signal Processing for ToF Sensors Single Depth Imaging

1 Single Depth Imaging - December 10, 2015

3D Imaging

a

b c

d

x

y

z


Single Depth Imaging in a Nutshell

� ToF probes the scene with a continuous wave signal.

� Reflected signal is cross—correlated at the lock-in sensor.

� Estimated phase and amplitude encode scene intensity and distance.


Distance

Phase Difference / Time DelayReceived SignalEmitted Signal

Light SourceCamera

d1d3

d2

Light SourceCamera

d1

Emitted Light Direct Reflection Second BounceFirst Bounce

(a) (b)

Light SourceCamera

d1d3

d2

(a) (b)

Light SourceCamera

d1

Emitted Light Direct Reflection First Bounce Second Bounced

Depth Information

p (t) = 1 + s0 cos (!t)

r (t) = � (1 + s0 cos (!t � �))

Single Depth Imaging

� = 2d!c

� = 2d!c� = 2d!c

� = 2d!c

� = 2d!c� = 2d!c


Single Depth Imaging — Lock-in Sensor Measurements

Light source

Pixel

Distance � = 2d!c

� = 2d!c� = 2d!c

m (t) =

Zp (z + t) r (z) dz

= �

✓1 +

s202

cos (!t + �)

◆

� = 2d!c

� = 2d!c� = 2d!c

��

z!0 = �e�|�!0z!0 = �e�|�!0z!0 = �e�|�!0

⌅ Goal: Estimate �� and ��.

Cross Correlation


Single Depth Imaging — Lock-in Sensor Measurements

� = 2d!c

� = 2d!c� = 2d!c

2 p 4 p

1

2

Ê

ÊÊ

Ê

2 p 4 p

1

�

✓1 +

s202

cos (!t+ �)

◆�

✓1 +

s202

cos (!t+ �)

◆�

✓1 +

s202

cos (!t+ �)

◆

m (t) =

Zp (z + t) r (z) dz

= �

✓1 +

s202

cos (!t + �)

◆

m0 m1

m2 m3

�= �

s202

+cos (�) � sin (�)� cos (�) + sin (�)

�m0 m1

m2 m3

�= �

s202

+cos (�) � sin (�)� cos (�) + sin (�)

�m0 m1

m2 m3

�= �

s202

+cos (�) � sin (�)� cos (�) + sin (�)

�

m0

m1 m2

m3mk = m

✓⇡k

2!

◆mk = m

✓⇡k

2!

◆mk = m

✓⇡k

2!

◆


Single Depth Imaging — 4 Bucket Trick

m0 m1

m2 m3

�= �

s202

+cos (�) � sin (�)� cos (�) + sin (�)

�m0 m1

m2 m3

�= �

s202

+cos (�) � sin (�)� cos (�) + sin (�)

�m0 m1

m2 m3

�= �

s202

+cos (�) � sin (�)� cos (�) + sin (�)

� Ê

ÊÊ

Ê

2 p 4 p

1

�

✓1 +

s202

cos (!t+ �)

◆�

✓1 +

s202

cos (!t+ �)

◆�

✓1 +

s202

cos (!t+ �)

◆

m0

m1 m2

m3

mk = m

✓⇡k

2!

◆mk = m

✓⇡k

2!

◆mk = m

✓⇡k

2!

◆

m3 �m1

m0 �m2

�= � s20

sin (�)cos (�)

�m3 �m1

m0 �m2

�= � s20

sin (�)cos (�)

�m3 �m1

m0 �m2

�= � s20

sin (�)cos (�)

�

m3 �m1

m0 �m2

�=

�I�R

�m3 �m1

m0 �m2

�=

�I�R

�m3 �m1

m0 �m2

�=

�I�R

�

z = �R + |�I

z = �R + |�I

m0 �m2m0 �m2m0 �m2

m3 �m1m3 �m1m3 �m1

e� = \z , tan�1

✓�I�R

◆e� = |z| /s20


Single Depth Imaging — Summary

Distance


Light SourceCamera

d1d3

d2

Light SourceCamera

d1


(a) (b)

Light SourceCamera

d1d3

d2

(a) (b)

Light SourceCamera

d1

Emitted Light Direct Reflection First Bounce Second Bounced

Depth Information

p (t) = 1 + s0 cos (!t)

r (t) = � (1 + s0 cos (!t � �))

� = 2d!c

� = 2d!c� = 2d!c

� = 2d!c

� = 2d!c� = 2d!c

m (t) =

Zp (z + t) r (z) dz

= �

✓1 +

s202

cos (!t + �)

◆

II

1

III e�2 = s�40

⇣(m3 �m1)

2 + (m0 �m2)2⌘

e� = tan�1

✓m3 �m1

m0 �m2

◆


Single Depth Imaging — Summary

� = 2d!c

� = 2d!c� = 2d!c

��

z!0 = �e�|�!0z!0 = �e�|�!0z!0 = �e�|�!0

⌅ Goal: Estimate �� and ��.

!

z = �e|!2dc

t

Time Domain

!

Frequency Domain

z = �e|!2dc

��

✓t� 2d

c

◆��

✓t� 2d

c

◆��

✓t� 2d

c

◆

t0 = 2d/c


Single Depth Imaging — Summary (In Phase Space)

z!0 = �e�|�!0z!0 = �e�|�!0z!0 = �e�|�!0

Ê

ÊÊ

Ê

2 p 4 p

1

�

✓1 +

s202

cos (!t+ �)

◆�

✓1 +

s202

cos (!t+ �)

◆�

✓1 +

s202

cos (!t+ �)

◆

m0

m1 m2

m3

m0

m1

m2

m3

mk = m

✓⇡k

2!

◆mk = m

✓⇡k

2!

◆mk = m

✓⇡k

2!

◆



Ê

ÊÊ

Ê

2 p 4 p

1

�

✓1 +

s202

cos (!t+ �)

◆�

✓1 +

s202

cos (!t+ �)

◆�

✓1 +

s202

cos (!t+ �)

◆

m0

m1 m2

m3 mk = m

✓⇡k

2!

◆mk = m

✓⇡k

2!

◆mk = m

✓⇡k

2!

◆

Real Measurements

m0

m1

m2

m3

z!0 = �e�|�!0z!0 = �e�|�!0z!0 = �e�|�!0



m3 �m1

m0 �m2

�=

�I�R

�m3 �m1

m0 �m2

�=

�I�R

�m3 �m1

m0 �m2

�=

�I�R

�

z = �R + |�I

e� = \z , tan�1

✓�I�R

◆e� = |z| /s20

z = �R + |�I

m0 �m2m0 �m2m0 �m2

m3�

m1

m3�

m1

m3�m

1

m0

m1

m2

m3

z!0 = �e�|�!0z!0 = �e�|�!0z!0 = �e�|�!0

m0 �m2m0 �m2m0 �m2

m3 �m1m3 �m1m3 �m1

Ê

ÊÊ

Ê

2 p 4 p

1

�

✓1 +

s202

cos (!t+ �)

◆�

✓1 +

s202

cos (!t+ �)

◆�

✓1 +

s202

cos (!t+ �)

◆

m0

m1 m2

m3 mk = m

✓⇡k

2!

◆mk = m

✓⇡k

2!

◆mk = m

✓⇡k

2!

◆


Single Depth Imaging in a Nutshell

Distance


Light SourceCamera

d1d3

d2

Light SourceCamera

d1


(a) (b)

Light SourceCamera

d1d3

d2

(a) (b)

Light SourceCamera

d1

Emitted Light Direct Reflection First Bounce Second BouncedDepth Information

p (t) = 1 + s0 cos (!t)

r (t) = � (1 + s0 cos (!t � �))

� = 2d!c

� = 2d!c� = 2d!c

� = 2d!c

� = 2d!c� = 2d!c

� ToF probes the scene with a continuous wave signal.

� Reflected signal is cross—correlated at the lock-in sensor.

� Estimated phase and amplitude encode scene intensity and distance.


single depth imaging - mit media labweb.media.mit.edu/~achoo/iccvtoftutorial/prelimslides/...single...

Documents