![Page 1: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/1.jpg)
Raskar, Camera Culture, MIT Media Lab
Computational Photography d
Camera Culture
and Light TransportCamera Cultureg a po
Ramesh Raskar
C C ltRamesh Raskar http://raskar.infoCamera CultureMIT Media Lab
![Page 2: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/2.jpg)
LCD = CameraLooking around corners
Phone = Scientific InstrumentTheory of Rays / Waves
![Page 3: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/3.jpg)
The goal is to create an entirely The goal is to create an entirely new class of imaging platforms
that have an
understanding of the world that far exceeds human abilityfar exceeds human ability
to produce meaningful abstractions that are p gwell within human comprehensibility
![Page 4: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/4.jpg)
OpticsComputational Li h T
acking
Optics
Displays
Light Transport
ton Ha Sensors Computational Photography
Phot
Signal ProcessingComputer Vision
Signal Processing
Machine Learning
Bit Hacking
![Page 5: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/5.jpg)
Computational Illumination
Office of the Future Multi‐Projector Spatial Augmented Pocket Projector Optical
Computational IlluminationPhD + MERL 1998‐2004
Office of the Future Multi‐ProjectorDisplay
Spatial AugmentedReality (SAR)
Pocket Projector OpticalCommunication
3Dvideoconferencing
Quadric Transfer Shader Lamps iLamps Location Tracking
Siggraph 19981999
Siggraph 20031998 2000
Siggraph 2003 Siggraph 2004
•Technology Review TR100, 2004•Book: Spatial Augmented Reality•Mitsubishi Electric Invention Awards (4), 2003, 2004, 2006( )•Over 20 patents•Planar and Curved Display Screen Product (6 products)
![Page 6: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/6.jpg)
C t ti l Ph t hComputational PhotographyMERL 2002‐2008
Coding in Time Coding in Space (Optical Path) Coded Illumination
Coded Wavelength
Coded Sensing
C d d E C d d A f M k b d O i l M l i fl h A il S G diCoded Exposure for Motion Deblurring
Coded Aperture for Extended Depth of
Field
Mask‐based Optical Heterodyning for Light Field Capture
Multi‐flash Imaging for Depth Edge Detection
Agile Spectrum Imaging
Gradient Encoding
Sensor for HDR
Siggraph 2006 Siggraph 2007 Siggraph 2007 Siggraph 2004 EG 2007 CVPR 2006
![Page 7: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/7.jpg)
Motion Blur in Low Light
![Page 8: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/8.jpg)
Traditional
Blurred Photo
Deblurred Image
![Page 9: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/9.jpg)
Fluttered Shutter CameraR k A l T bli Si h2006Raskar, Agrawal, Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
![Page 10: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/10.jpg)
Sh Bl d
Fourier Transform
Sharp Photo
Blurred PhotoPSF == Broadband Function
Preserves High Spatial Frequencies
Flutter Shutter: Shutter is OPEN and CLOSED
![Page 11: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/11.jpg)
Traditional Coded Exposure
Deblurred I
Deblurred I ImageImage
Image of Static Object
![Page 12: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/12.jpg)
Motion Blur in Low Light
New CollaborationsBill Freeman, EECSDavid Brady, Duke U./MosaicDan Schuette, Lincoln L
![Page 13: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/13.jpg)
C t ti l Ph t hComputational PhotographyMERL 2002‐2008
Coding in Time Coding in Space (Optical Path) Coded Illumination
Coded Wavelength
Coded Sensing
C d d E C d d A f M k b d O i l M l i fl h A il S G diCoded Exposure for Motion Deblurring
Coded Aperture for Extended Depth of
Field
Mask‐based Optical Heterodyning for Light Field Capture
Multi‐flash Imaging for Depth Edge Detection
Agile Spectrum Imaging
Gradient Encoding
Sensor for HDR
Siggraph 2006 Siggraph 2007 Siggraph 2007 Siggraph 2004 EG 2007 CVPR 2006
•Alfred P Sloan Research Fellowship, 2009•Over 15 patents•Coded Exposure and Multi‐flash camera now multi‐million dollar projects•Mitsubishi Electric invention award•Upcoming book on Computational Photography
![Page 14: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/14.jpg)
OpticsComputational Li h T
nsOptics
Displays
Light Transport
Photon Sensors Computational Photography
P
Signal ProcessingComputer Vision
Signal Processing
Machine Learning
Bits
![Page 15: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/15.jpg)
Computational Light TransportMedia Lab 2008‐
1. Time resolved
Descattering Spatial Augmented Rank‐constraint of Computational Wavefront Compressive
2. Angle resolved
Analysis Heterodyning Light Field 3D Displays Probes Sensing Sensing
CAT‐Scan withouti
BiDi Screen Geometric+ Wave optics
Glasses Free 3D Bokode NETRA SparsityAnalysis
moving parts
2009 ‐ 2007‐ 2008‐ 2009 ‐ 2008 ‐ 2010 2009 ‐
![Page 16: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/16.jpg)
Can you look around the corner ?around the corner ?
![Page 17: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/17.jpg)
Can you look around the corner ?around the corner ?
![Page 18: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/18.jpg)
Can you look around the corner ?around the corner ?
Multi‐bounce reflections
![Page 19: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/19.jpg)
Multi‐path Analysis
2nd Bounce
Analysis
1st Bounce
3rd Bounce
![Page 20: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/20.jpg)
Femto‐PhotographyFemtoFlash
UltraFast DetectorUltraFast Detector
Computational Optics
Serious Sync
•Darpa Young Faculty Award, 2010•Lincoln Labs Campus Collaboration Award 2010•Lincoln Labs, Campus Collaboration Award, 2010•Kirmani, Hutchinson, Davis, Raskar, ICCV’2009, Marr Prize Honorable Mention•Pandharkar, Velten, Bardagjy, Bawendi, Raskar, CVPR 2011
![Page 21: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/21.jpg)
Femto‐Photography (Transient Imaging)FemtoFlash
UltraFast DetectorUltraFast Detector
Computational Optics
Serious Sync
•Darpa Young Faculty Award, 2010•Lincoln Labs Campus Collaboration Award 2010•Lincoln Labs, Campus Collaboration Award, 2010•Kirmani, Hutchinson, Davis, Raskar, ICCV’2009, Marr Prize Honorable Mention•Pandharkar, Velten, Bardagjy, Bawendi, Raskar, CVPR 2011
![Page 22: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/22.jpg)
Steady State 4D
Impulse Response, 5D
![Page 23: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/23.jpg)
Ti:Sapph LaserTi:Sapph Laser
Ti:Sapph LaserTi:Sapph Laser
Streak Camera
With M Bawendi, MIT Chemistry
![Page 24: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/24.jpg)
With M Bawendi, MIT Chemistry
![Page 25: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/25.jpg)
t
rc2rc2/c
t
rc1 rc1/csc1/
![Page 26: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/26.jpg)
xz
S
L
sOccluder
Streak‐camera
Laser CbeamB
Echoes of Light
![Page 27: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/27.jpg)
Streak Image
xz
IRSStreak Image
L
sOccluder
Hyperbolic Kernel
Streak‐camera
Laser CbeamB
Echoes of Light
![Page 28: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/28.jpg)
Streak Image
xz
IRSStreak Image
L
sOccluder
Hyperbolic Kernel
Streak‐camera
Laser C3rd bounce
beamB
Echoes of Light
![Page 29: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/29.jpg)
Trillion FPS
ToF Streak Camera = Inverse of CRO
![Page 30: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/30.jpg)
![Page 31: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/31.jpg)
Tim
e
Space
Third Bounce (First bounce not shown)
Space
![Page 32: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/32.jpg)
Tim
e
Space
Third Bounce (First bounce not shown)
Space
![Page 33: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/33.jpg)
Tim
e
Space
Third Bounce (First bounce not shown)
Space
![Page 34: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/34.jpg)
Tim
e
Space
Third Bounce (First bounce not shown)
Space
![Page 35: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/35.jpg)
Tim
e
Space
Third Bounce (First bounce not shown)
Space
![Page 36: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/36.jpg)
![Page 37: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/37.jpg)
Forward Reconstruction Invertibility Analysis
Modified Fresnel Approximation
Backpropagation+ Carving
Scene Priors, L1 reconstruction
Resolution and dimensionsApproximation + Carving L1 reconstruction
via COSaMPdimensions
![Page 38: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/38.jpg)
Inverting Light Transport
Multiple Scattering Direct/Global
[Seitz , Kutulakos, Matsushita 2005] [Nayar, Raskar et al 2006]
Dual Photography LIDARDual Photography
[Sen et al 2005]
LIDAR
![Page 39: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/39.jpg)
Rescue and Planningg
![Page 40: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/40.jpg)
Robot, Car Path Planning, g
![Page 41: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/41.jpg)
Endoscopypy
![Page 42: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/42.jpg)
Time Resolved Multi‐path Imaging: Plans
Scene with hidden elements
Capture
Lasers, scene, sensors, ICCD
Capture time
profiles Raw Data
Signal Proc
Photo, geometry
t
Proc.
Novel light transport models and inference
algorithms
around the corner
3D Time images3D Time images
Scenes: NLoS, Motion, BRDF in single shot, Volumetric (tissue)Scale: Endscopes table top room sized outdoors underwaterScale: Endscopes, table‐top, room‐sized, outdoors, underwaterInversion: Sparsity, Rank, Bounded Approx, Scene Priors, TransformsSignalProc: Compressive, SNR, Bandwidth, Noise modelsCapture: Coding in space/time/wavelgth Solid state non‐linear opticsCapture: Coding in space/time/wavelgth, Solid state, non linear opticsSpectrum: Radar, Sonar/Ultrasound
![Page 43: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/43.jpg)
Computational Light TransportMIT 2008‐
1. Time resolved
Descattering Spatial Augmented Rank‐constraint of Computational Wavefront Compressive
2. Angle resolved
Analysis Heterodyning Light Field 3D Displays Probes Sensing Sensing
CAT‐Scan withouti
BiDi Screen Geometric+ Wave optics
Glasses Free 3D Dual Layer LCD
Bokode NETRA SparsityAnalysis
moving parts
2009 ‐ 2008‐ 2008‐ 2009 ‐ 2008 ‐ 2010 2009 ‐
![Page 44: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/44.jpg)
LCD = a big flat camera?g
![Page 45: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/45.jpg)
Two Layer Displays19081903
Angular Information
19081903
Parallax Barrier = dim displays Lenslets = fixed low space/angle resolution
Pin hole Lensletarray
Lensletarray
sensor/display sensor/display
Spatial Heterodyning
New Solution:
using a patterned mask
![Page 46: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/46.jpg)
Beyond Multi-touch: Thin LCD for touch+hover
Mobile
Laptops
![Page 47: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/47.jpg)
BiDi Screen: Multi-touch + Hover 3D interface
![Page 48: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/48.jpg)
Sensing Depth from Array of Virtual Cameras in thin LCDy
Funding : Samsung SAITHirsch, Holtzman, Lanman, Raskar, SiggraphAsia 2009
![Page 49: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/49.jpg)
CAT Scan without moving parts
With , Berthold Horn, EECS,Dick Lanza, Nuclear Engg.
![Page 50: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/50.jpg)
Rank Analysis of 3D Pbarriers Displays
kL[i k]
i
L[i,k]
`
g[k]k
i L[i k]`f[i] L[i,k]
light box
][][],[ kgifkiL gfL
![Page 51: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/51.jpg)
Glasses Free 3D using High Rank Displays
G
FL̀~ =
0,for ,21 min arg 2
GF, GFFGL
W Content-Adaptive Parallax Barriers
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
E l it t t d ti ll b iExploit content-adaptive parallax barriers
Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010
![Page 52: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/52.jpg)
Light Rays vs WavesLight Rays vs Waves
WDFWigner Distribution Function Augmented
Li ht Fi ld
WDF
Function
Light Field LF
Light Field
Rays WavesAugmented Light Field
Supports diffraction/interferenceSupports diffraction/interferenceRadiance = Positive/Negative
![Page 53: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/53.jpg)
Computational Light TransportMIT 2008‐
1. Time resolved
Descattering Spatial Augmented Rank‐constraint of Computational Wavefront Compressive
2. Angle resolved
Analysis Heterodyning Light Field 3D Displays Probes Sensing Sensing
CAT‐Scan withouti
BiDi Screen Geometric+ Wave optics
Glasses Free 3D Dual Layer LCD
Bokode NETRA SparsityAnalysis
moving parts
2009 ‐ 2008‐ 2008‐ 2009 ‐ 2008 ‐ 2010 2009 ‐
![Page 54: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/54.jpg)
NETRA: Interactive Display for Estimating Refractive Errors and Focal RangeRefractive Errors and Focal Range
Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar
54
![Page 55: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/55.jpg)
NETRA: Near Eye Tool for Refractive Assessment
Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar
55
![Page 56: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/56.jpg)
2Brefractive errorsrefractive errors
0.6B t d
4.5B withMobile phone
uncorrected refractive errors
6.5 Billion people NETRA at LVP Eye Institute
![Page 57: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/57.jpg)
Needs expert, Moving parts, Shining lasers
Retinoscope w/ Lenses
Auto‐refracto‐meter
Chart with Lenses
In‐Focus: Focometer Optiopia
Solo‐health: EyeSite
NETRALenses meter Lenses EyeSite
Technology Shining Light plus lenses
FundusCamera
Moving lenses + target
Moving lenses + target
Reading chart on monitor
Cellphone+ eyepiece
Cost to buy $2,000* ~$10,000 ~$100 ~$495 ~$200 ‐‐ $30
C $36 $36 $ $1Cost per test ~$36 ~$36 ~$5 ‐‐ ‐‐ ‐‐ ~$1
Data capture No Comp. No No No Comp. Phone
Mobility <500g >10Kg 2kg 1kg <5kg >10Kg <100g
S d F t F t M di M di F t F tSpeed Fast Fast Medium Medium ‐‐ Fast Fast
Scalability No No No Yes Probably No Yes
Accuracy 0.15 0.15 0.5 0.75 ‐‐ ‐‐ <0.5
Self evaluation No No Yes Yes Yes Yes YesSelf evaluation No No Yes Yes Yes Yes Yes
Electricity Req No Yes No No ‐‐ Yes No
Astigmatism Yes Yes Yes/No No ‐‐ Yes Yes
Network No Yes No No No Yes YesNetwork No Yes No No No Yes Yes
Training High High High Medium Medium Low Low
* Phoropter‐based: $5,000.00
![Page 58: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/58.jpg)
Shack‐Hartmann Wavefront SensorShack‐Hartmann Wavefront Sensor
Wavefront aberrometer
Expensive; Bulky, Requires trained professionals58
![Page 59: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/59.jpg)
Shack‐Hartmann Wavefront Sensor
Laser
Shack & Platt 1971Liang et al 1994
David Williams et al, Rochester
Spot Diagram
Sensor Microlens Planar
WavefrontArray
59
![Page 60: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/60.jpg)
Shack‐Hartmann Wavefront Sensor
LaserSpot Diagram
Sensor
Displacement = Local Slope
of the Wavefront Shack‐Hartmann ~ Lightfields
Oh, Raskar, Barbastathis 2009: Augmented Light Field
![Page 61: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/61.jpg)
NETRA = Inverse of Shack‐Hartmann
Spot Diagram on LCD
Cell Eye Phone Display
yPiece
61
![Page 62: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/62.jpg)
NETRA = Inverse of Shack‐Hartmann
Spot Diagram on LCD
Cell Eye Phone Display
yPiece
62
![Page 63: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/63.jpg)
Inverse of Shack‐HartmannUser interactively creates the Spot Diagram
Spot Diagram on LCD
Displace 25
63
ppoints with smart UI
![Page 64: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/64.jpg)
Inverse of Shack‐HartmannUser interactively creates the Spot Diagram
Spot Diagram on LCD
Displace 25
64
ppoints but 3 parameters
![Page 65: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/65.jpg)
Cataract screening using inverse Shack Hartmanninverse Shack‐Hartmann
Under review 2011
![Page 66: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/66.jpg)
Limitations
• Ability to align linesChildren– Children
– Retinal conditions
– Accomodation cues
• Resolution is a function of the display DPI– Samsung Behold II – 160 DPI – 0.35D
– Google Nexus One – 250 DPI – 0.2D
– Apple iPhone 4G – 326 DPI – 0.14D
![Page 67: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/67.jpg)
NETRA: Refraction + Cataract Tests
• Inverse of Shack‐Hartmann wavefront aberrometerHi h l ti di l d i t ti– High‐resolution displays and user interaction
– No lasers, moving parts
– Trials in progress
H d t– Hardware app store
• Parameters– Myopia, Hyperopia, Astigmatism
– Cataract, Lazy eye
• Impact in Developing Countries– 600 Million without corrective glasses
– $1 cost, easy to deploy, free s/w, see EyeNetra.comy p y y
67
![Page 68: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/68.jpg)
Clinical Testing Partners
![Page 69: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/69.jpg)
NETRA in a dozen+ countriesNETRA in a dozen+ countries
69
![Page 70: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/70.jpg)
Kenya India
Awards Selection
• MIT IDEAS (#2 award)
• Deshpande Ignition Grant
• MIT 100K (dev finalist)
• NASA/USAID Innovators
• Worldbank Social Health Inventions
• International Space Station evaluationMIT 100K (dev finalist)
• Google + (L Page)
• Vodafone Foundation (finalist)
International Space Station evaluation
Validation• 0.09 D : objective precision
Papers
• SIGGRAPH
70• ~ 0.5 D: subjective trials
• 0.3 D: IRB approved wet‐studies (Prelim data)
• Frontiers in Optics
• Am Academy of Optometry (AAO)
![Page 71: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/71.jpg)
Chicken Eye Inducing Cataract
‘Lens’ Imaging CCD
![Page 72: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/72.jpg)
Optics: Bokodes = Human EyeBokode Capture = Retinal ImagingBokode Capture = Retinal Imaging
cell-phone cameraclose to the Bokodeclose to the Bokode
(10,000+ bytes of data)
Like a Retinal Image
![Page 73: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/73.jpg)
Eye = Mirror of Health
73Yoon Lab, U of Rochester
![Page 74: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/74.jpg)
Slit Lamp Exam Retinal Scanp
Ocular Manifestation: Leading Indicator?
74
![Page 75: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/75.jpg)
T di i l U D iTraditional User Driven
Mass‐use Devices ‐> Scientific Instruments
![Page 76: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/76.jpg)
Current and Former Members• Post-docs
• Ankit Mohan• Andreas Velten• Douglas Lanman• Yunhee Kim
• RAs• Andy Bardagjy• Kevin Chiu
Matthew Hirsch (w Holtzman)• Matthew Hirsch (w Holtzman)• Roarke Horstmeyer• Otkrist Gupta• Ahmed Kirmani• Jaewon Kim• Nikhil Naik• Rohit Pandharkar
• MEng• Tyler Hutchison
Sh B k• Sharmeen Browarek• Dennis Miaw
• Visitors• Vitor Pamplona• Abhijit Bendale
• Daniel Saakes• Grace Woo
T Cb j da
• Erick Passos• Behzad Sajjadi• Gordon Wetzstein• Matthias Hullin
• Tom Cuypers• Manuel Oliviera• Shinsaku Hiura• Yasuhiro Mukaigawa
![Page 77: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/77.jpg)
Codesigning Optical and Digital Processing
Computational
s
Optics
Displays
Computational Light Transport
hotons
Sensors Computational Photography
Ph
g p y
Computer VisionSignal Processing
Machine Learning
Bits
Machine Learning
![Page 78: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/78.jpg)
Computational Light Transport
Ramesh Raskar http://raskar.info
OpticsComputational Light TransportComputational Light Transport
– Super‐human visual abilities
– Empirical, Multi‐directional rather than one narrow field
F i f di i ilBits
Photon
s
Computer Vision
Sensors
Signal Processing
Displays
Machine Learning
Computational Photography
– Fusion of dissimilar
• New Fields– Femto‐photography
Dream Augmentation Image IP– Dream Augmentation, Image IP
• New Insights– Challenge the status quo
– BiDi Screen, CAT‐scan, Augmented LFBiDi Screen, CAT scan, Augmented LF
– Sparsity, rank, priors
• New Purpose– Disruptive mass‐use tech for social impact F
G
L̀~
– Netra, Cataract, Retinal scans
– $1M ‐> $1 0,for ,21 min arg 2
GF, GFFGL
W
=
Cameras, Displays, Medical Tools, Future Devices– Theory, Modeling, Optical+Mathematical insight
![Page 79: New Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/11Sig/Movies/Old/RaskarFaculty... · 2011. 2. 14. · Computational Light Transport MIT 2008‐ 1. Time resolved Descattering](https://reader033.vdocument.in/reader033/viewer/2022052105/604004407e5b7c183c055aab/html5/thumbnails/79.jpg)
Computational Light TransportMedia Lab 2008‐
1. Time resolved
Descattering Spatial Augmented Rank‐constraint of Computational Wavefront Compressive
2. Angle resolved
Analysis Heterodyning Light Field 3D Displays Probes Sensing Sensing
CAT‐Scan withouti
BiDi Screen Geometric+ Wave optics
Glasses Free 3D Bokode NETRA SparsityAnalysis
moving parts
2009 ‐ 2007‐ 2008‐ 2009 ‐ 2008 ‐ 2010 2009 ‐