temporal frequency probing for 5d transient analysis of global light transport
DESCRIPTION
Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport Matthew O'Toole, Felix Heide, Lei Xiao, Matthias B. Hullin, Wolfgang Heidrich, and Kiriakos N. Kutulakos. ACM SIGGRAPH, 2014. Abstract: We analyze light propagation in an unknown scene using projectors and cameras that operate at transient timescales. In this new photography regime, the projector emits a spatio-temporal 3D signal and the camera receives a transformed version of it, determined by the set of all light transport paths through the scene and the time delays they induce. The underlying 3D-to-3D transformation encodes scene geometry and global transport in great detail, but individual transport components (e.g., direct reflections, inter-reflections, caustics, etc.) are coupled nontrivially in both space and time. To overcome this complexity, we observe that transient light transport is always separable in the temporal frequency domain. This makes it possible to analyze transient transport one temporal frequency at a time by trivially adapting techniques from conventional projector-to-camera transport. We use this idea in a prototype that offers three never-seen-before abilities: (1) acquiring time-of-flight depth images that are robust to general indirect transport, such as inter-reflections and caustics; (2) distinguishing between direct views of objects and their mirror reflection; and (3) using a photonic mixer device to capture sharp, evolving wavefronts of "light-in-flight".TRANSCRIPT
![Page 1: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/1.jpg)
Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport
http://www.dgp.toronto.edu/~motoole/temporalprobing.html
Matt O’Toole1 Felix Heide2 Lei Xiao2 Matthias Hullin3 Wolfgang Heidrich2,4 Kyros Kutulakos1
1University of Toronto 2University of British Columbia 3University of Bonn 4KAUST
![Page 2: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/2.jpg)
the time-of-flight (ToF) sensing revolution
estimating poses[Schwarz et al. 11]
Kinect for Xbox Onefemto-photography[Velten et al. 2013]
looking-around-corners[Velten et al. 2012]
gesture recognition[Droeschel et al. 11]
Google’s self-driving car
![Page 3: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/3.jpg)
temporal probing for scene analysis
capture depthrobust to indirect light
reconstruct “light-in-flight” videowith high temporal resolution
key insight: complex-valued image formation model
![Page 4: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/4.jpg)
contributions
the transient-frequency transport matrix
unified theory for light transport analysis analyze ToF images using classical techniques
ToF imaging using projectors & maskscapture depth robust to indirect light
reconstruct “light-in-flight” video with high temporal resolution
![Page 5: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/5.jpg)
what is a ToF photo?
ToF camera
![Page 6: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/6.jpg)
what is a ToF photo?
point lightToF camera
![Page 7: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/7.jpg)
what is a ToF photo?
ToF camera ToF projector
![Page 8: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/8.jpg)
what is a ToF photo?
ToF camera ToF projector
![Page 9: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/9.jpg)
what is a ToF photo?
ToF camera ToF projector
![Page 10: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/10.jpg)
what is a ToF photo?
ToF camera ToF projector
![Page 11: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/11.jpg)
what is a ToF photo?
ToF camera ToF projector
![Page 12: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/12.jpg)
what is a ToF photo?
ToF camera ToF projector
![Page 13: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/13.jpg)
what is a ToF photo?
ToF projectorToF camera
![Page 14: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/14.jpg)
what is a ToF photo?
ToF projectorToF camera
![Page 15: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/15.jpg)
what is a ToF photo?
ToF projectorToF camera
intensity
![Page 16: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/16.jpg)
what is a ToF photo?
phase delay
ToF projectorToF camera
![Page 17: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/17.jpg)
what is a ToF photo?
transport coefficient
ToF camera ToF projector
![Page 18: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/18.jpg)
what is a ToF photo?
transport coefficient
ToF camera ToF projector
![Page 19: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/19.jpg)
what is a ToF photo?
transport coefficient
ToF camera ToF projector
![Page 20: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/20.jpg)
mirror
what is a ToF photo?
transport coefficient
ToF camera ToF projector
![Page 21: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/21.jpg)
mirror
camera projector
it is a vector of complex-valued pixels…
camera
photo
![Page 22: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/22.jpg)
camera
it is a vector of complex-valued pixels…
camera
photo pattern
![Page 23: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/23.jpg)
camera
… produced by a complex-valued matrix
camera
photo patterntransport matrix for
![Page 24: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/24.jpg)
conventional image under ambient lighting
![Page 25: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/25.jpg)
brightness
hue
visualizing a complex ToF image
ph
ase
![Page 26: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/26.jpg)
camera
transient-frequency light transport equation
camera
photo patterntransport matrix for
![Page 27: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/27.jpg)
transient-frequency light transport equation
conventional transport matrix [Ng et al. 03; …]
photo transport matrix for pattern
![Page 28: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/28.jpg)
Techniques Reference(s)
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
![Page 29: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/29.jpg)
Techniques Reference(s) Conventional Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
![Page 30: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/30.jpg)
Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
![Page 31: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/31.jpg)
Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
![Page 32: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/32.jpg)
Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
![Page 33: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/33.jpg)
Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
![Page 34: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/34.jpg)
structured light transport [O’Toole et al. CVPR 2014]
acquire direct & indirect ToF images using epipolar constraints
fast direct/global separation [Nayar et al. SIGGRAPH 2006]
acquire caustic indirect & diffuse indirect ToF images using radiometric constraints
![Page 35: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/35.jpg)
structured light transport [O’Toole et al. CVPR 2014]
acquire direct & indirect ToF images using epipolar constraints
fast direct/global separation [Nayar et al. SIGGRAPH 2006]
acquire caustic indirect & diffuse indirect ToF images using radiometric constraints
![Page 36: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/36.jpg)
mirror
ToF camera ToF projector
structured light transport for freq.
![Page 37: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/37.jpg)
mirror
ToF camera ToF projector
structured light transport for freq.
![Page 38: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/38.jpg)
mirror
direct paths obey epipolar geometry
ToF camera ToF projector
structured light transport for freq.
![Page 39: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/39.jpg)
mirror
indirect paths DONTobey epipolar geometry
ToF camera ToF projector
structured light transport for freq.
![Page 40: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/40.jpg)
mirror
projection pattern
mask pattern
ToF camera ToF projector
structured light transport for freq.
![Page 41: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/41.jpg)
mirror
mask pattern
ToF camera ToF projector
structured light transport for freq.
projection pattern
![Page 42: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/42.jpg)
mirror
projection pattern
mask pattern
1. open electronic shutter2. for i = 1 to N
use random epipolar mask &project complementary pattern
3. close electronic shutter
ToF camera ToF projector
structured light transport for freq.
![Page 43: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/43.jpg)
brightness
hue
conventional ToF imaging
![Page 44: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/44.jpg)
brightness
hue
indirect ToF
structured light transport conventional ToF imaging
![Page 45: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/45.jpg)
direct ToF
brightness
hue
indirect ToF
structured light transport conventional ToF imaging
![Page 46: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/46.jpg)
application: 3D from ToF
![Page 47: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/47.jpg)
ToF depth maps & indirect illumination
![Page 48: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/48.jpg)
direct-only
conventional vs direct-only ToF
conventional
![Page 49: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/49.jpg)
conventional direct-only
3D from conventional vs direct-only ToF
![Page 50: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/50.jpg)
conventional direct-only
conventional vs direct-only ToF
![Page 51: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/51.jpg)
conventional direct-only
conventional vs direct-only ToF
![Page 52: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/52.jpg)
conventional direct-only
3D from conventional vs direct-only ToF
![Page 53: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/53.jpg)
structured light transport [O’Toole et al. CVPR 2014]
acquire direct & indirect ToF images using epipolar constraints
fast direct/global separation [Nayar et al. SIGGRAPH 2006]
acquire caustic indirect & diffuse indirect ToF images using radiometric constraints
![Page 54: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/54.jpg)
mirror
effect of indirect transport on spatial frequencies
ToF camera ToF projector
![Page 55: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/55.jpg)
mirror
effect of indirect transport on spatial frequencies
caustic high frequencynon-caustic low frequencyToF camera ToF projector
![Page 56: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/56.jpg)
mirror
effect of indirect transport on spatial frequencies
ToF camera ToF projector
caustic high frequencynon-caustic low frequency
![Page 57: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/57.jpg)
mirror
effect of indirect transport on spatial frequencies
1. for i = 1 to Nproject high spatial-freq. patterncapture image
2. non-caustic = min of images
ToF camera ToF projector
caustic high frequencynon-caustic low frequency
![Page 58: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/58.jpg)
direct ToF
indirect ToF
brightness
hue
structured light transport conventional ToF imaging
![Page 59: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/59.jpg)
direct ToF non-caustic ToF
indirect ToF
structured light transport fast caustic/non-caustic separation
brightness
hue
conventional ToF imaging
![Page 60: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/60.jpg)
direct ToF non-caustic ToF
caustic ToFindirect ToF
structured light transport fast caustic/non-caustic separation
brightness
hue
conventional ToF imaging
![Page 61: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/61.jpg)
direct ToF non-caustic ToF
caustic ToF
![Page 62: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/62.jpg)
application: light-in-flight imaging
![Page 63: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/63.jpg)
mirror
“light-in-flight” imaging for time instant t
goal: reconstruct image for time t
ToF camera ToF projector
![Page 64: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/64.jpg)
“light-in-flight” imaging: prior work
femto-photography[Velten et al. 2013]
very high costwell-resolved wavefronts
PMD-based system[Heide et al. 2013]
many modulation frequenciesstrong scene priors
optical wavefronts not resolvable
our work
many modulation frequenciesweaker, transport-specific priors
resolvable optical wavefronts
![Page 65: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/65.jpg)
conventional ToF for = 100 MHz conventional image
piecewise “light-in-flight” video construction
![Page 66: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/66.jpg)
direct ToF for = 100 MHz light-in-flight video
piecewise “light-in-flight” video construction
direct (no regularization)
![Page 67: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/67.jpg)
caustic ToF for = 100 MHz light-in-flight video
piecewise “light-in-flight” video construction
direct (no regularization)
caustic indirect (no regularization)
![Page 68: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/68.jpg)
light-in-flight videonon-caustic ToF for = 12 to 140 MHz
piecewise “light-in-flight” video construction
direct (no regularization)
non-caustic indirect (regularizedusing [Heide et al. 2013])
caustic indirect (no regularization)
![Page 69: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/69.jpg)
light-in-flight video (ours)light-in-flight video [Heide et al. 13]
comparison to [Heide et al. 2013]
![Page 70: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/70.jpg)
light-in-flight video (ours)light-in-flight video [Heide et al. 13]
comparison to [Heide et al. 2013]
![Page 71: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/71.jpg)
conclusion
• a new paradigm for ToF sensing
• combines ToF sensors with projectors & masks
• can now leverage from decade of light transport research
• widespread implications for spatio-temporal transport analysis
![Page 72: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/72.jpg)
come visit us at our E-Tech booth
visualizing light transport phenomena
![Page 73: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/73.jpg)
Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport
http://www.dgp.toronto.edu/~motoole/temporalprobing.html
Matt O’Toole1 Felix Heide2 Lei Xiao2 Matthias Hullin3 Wolfgang Heidrich2,4 Kyros Kutulakos1
1University of Toronto 2University of British Columbia 3University of Bonn 4KAUST
![Page 74: Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport](https://reader034.vdocument.in/reader034/viewer/2022052601/559839ae1a28ab2d628b4708/html5/thumbnails/74.jpg)
temporal probing setup
max. exposure time of 8 ms, max. modulation frequency of 150 MHz