ramesh raskar - mit media labraskar/11sig/movies/old/raskarfacultyp… · bit hacking....
TRANSCRIPT
![Page 1: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/1.jpg)
Raskar, Camera Culture, MIT Media Lab
Computational Photography d
Camera Culture
and Light TransportCamera Cultureg a po
Ramesh Raskar
C C ltRamesh Raskar http://raskar.infoCamera CultureMIT Media Lab
![Page 2: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/2.jpg)
LCD = CameraLooking around corners
Phone = Scientific InstrumentTheory of Rays / Waves
![Page 3: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/3.jpg)
The goal is to create an entirely The goal is to create an entirely new class of imaging platforms
that have an
understanding of the world that far exceeds human abilityfar exceeds human ability
to produce meaningful abstractions that are p gwell within human comprehensibility
![Page 4: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/4.jpg)
OpticsComputational Li h T
acking
Optics
Displays
Light Transport
ton Ha Sensors Computational Photography
Phot
Signal ProcessingComputer Vision
Signal Processing
Machine Learning
Bit Hacking
![Page 5: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/5.jpg)
Computational Illumination
Office of the Future Multi‐Projector Spatial Augmented Pocket Projector Optical
Computational IlluminationPhD + MERL 1998‐2004
Office of the Future Multi‐ProjectorDisplay
Spatial AugmentedReality (SAR)
Pocket Projector OpticalCommunication
3Dvideoconferencing
Quadric Transfer Shader Lamps iLamps Location Tracking
Siggraph 19981999
Siggraph 20031998 2000
Siggraph 2003 Siggraph 2004
•Technology Review TR100, 2004•Book: Spatial Augmented Reality•Mitsubishi Electric Invention Awards (4), 2003, 2004, 2006( )•Over 20 patents•Planar and Curved Display Screen Product (6 products)
![Page 6: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/6.jpg)
C t ti l Ph t hComputational PhotographyMERL 2002‐2008
Coding in Time Coding in Space (Optical Path) Coded Illumination
Coded Wavelength
Coded Sensing
C d d E C d d A f M k b d O i l M l i fl h A il S G diCoded Exposure for Motion Deblurring
Coded Aperture for Extended Depth of
Field
Mask‐based Optical Heterodyning for Light Field Capture
Multi‐flash Imaging for Depth Edge Detection
Agile Spectrum Imaging
Gradient Encoding
Sensor for HDR
Siggraph 2006 Siggraph 2007 Siggraph 2007 Siggraph 2004 EG 2007 CVPR 2006
![Page 7: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/7.jpg)
Motion Blur in Low Light
![Page 8: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/8.jpg)
Traditional
Blurred Photo
Deblurred Image
![Page 9: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/9.jpg)
Fluttered Shutter CameraR k A l T bli Si h2006Raskar, Agrawal, Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
![Page 10: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/10.jpg)
Sh Bl d
Fourier Transform
Sharp Photo
Blurred PhotoPSF == Broadband Function
Preserves High Spatial Frequencies
Flutter Shutter: Shutter is OPEN and CLOSED
![Page 11: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/11.jpg)
Traditional Coded Exposure
Deblurred I
Deblurred I ImageImage
Image of Static Object
![Page 12: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/12.jpg)
Motion Blur in Low Light
New CollaborationsBill Freeman, EECSDavid Brady, Duke U./MosaicDan Schuette, Lincoln L
![Page 13: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/13.jpg)
C t ti l Ph t hComputational PhotographyMERL 2002‐2008
Coding in Time Coding in Space (Optical Path) Coded Illumination
Coded Wavelength
Coded Sensing
C d d E C d d A f M k b d O i l M l i fl h A il S G diCoded Exposure for Motion Deblurring
Coded Aperture for Extended Depth of
Field
Mask‐based Optical Heterodyning for Light Field Capture
Multi‐flash Imaging for Depth Edge Detection
Agile Spectrum Imaging
Gradient Encoding
Sensor for HDR
Siggraph 2006 Siggraph 2007 Siggraph 2007 Siggraph 2004 EG 2007 CVPR 2006
•Alfred P Sloan Research Fellowship, 2009•Over 15 patents•Coded Exposure and Multi‐flash camera now multi‐million dollar projects•Mitsubishi Electric invention award•Upcoming book on Computational Photography
![Page 14: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/14.jpg)
OpticsComputational Li h T
nsOptics
Displays
Light Transport
Photon Sensors Computational Photography
P
Signal ProcessingComputer Vision
Signal Processing
Machine Learning
Bits
![Page 15: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/15.jpg)
Computational Light TransportMedia Lab 2008‐
1. Time resolved
Descattering Spatial Augmented Rank‐constraint of Computational Wavefront Compressive
2. Angle resolved
Analysis Heterodyning Light Field 3D Displays Probes Sensing Sensing
CAT‐Scan withouti
BiDi Screen Geometric+ Wave optics
Glasses Free 3D Bokode NETRA SparsityAnalysis
moving parts
2009 ‐ 2007‐ 2008‐ 2009 ‐ 2008 ‐ 2010 2009 ‐
![Page 16: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/16.jpg)
Can you look around the corner ?around the corner ?
![Page 17: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/17.jpg)
Can you look around the corner ?around the corner ?
![Page 18: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/18.jpg)
Can you look around the corner ?around the corner ?
Multi‐bounce reflections
![Page 19: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/19.jpg)
Multi‐path Analysis
2nd Bounce
Analysis
1st Bounce
3rd Bounce
![Page 20: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/20.jpg)
Femto‐PhotographyFemtoFlash
UltraFast DetectorUltraFast Detector
Computational Optics
Serious Sync
•Darpa Young Faculty Award, 2010•Lincoln Labs Campus Collaboration Award 2010•Lincoln Labs, Campus Collaboration Award, 2010•Kirmani, Hutchinson, Davis, Raskar, ICCV’2009, Marr Prize Honorable Mention•Pandharkar, Velten, Bardagjy, Bawendi, Raskar, CVPR 2011
![Page 21: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/21.jpg)
Femto‐Photography (Transient Imaging)FemtoFlash
UltraFast DetectorUltraFast Detector
Computational Optics
Serious Sync
•Darpa Young Faculty Award, 2010•Lincoln Labs Campus Collaboration Award 2010•Lincoln Labs, Campus Collaboration Award, 2010•Kirmani, Hutchinson, Davis, Raskar, ICCV’2009, Marr Prize Honorable Mention•Pandharkar, Velten, Bardagjy, Bawendi, Raskar, CVPR 2011
![Page 22: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/22.jpg)
Steady State 4D
Impulse Response, 5D
![Page 23: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/23.jpg)
Ti:Sapph LaserTi:Sapph Laser
Ti:Sapph LaserTi:Sapph Laser
Streak Camera
With M Bawendi, MIT Chemistry
![Page 24: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/24.jpg)
With M Bawendi, MIT Chemistry
![Page 25: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/25.jpg)
t
rc2rc2/c
t
rc1 rc1/csc1/
![Page 26: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/26.jpg)
xz
S
L
sOccluder
Streak‐camera
Laser CbeamB
Echoes of Light
![Page 27: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/27.jpg)
Streak Image
xz
IRSStreak Image
L
sOccluder
Hyperbolic Kernel
Streak‐camera
Laser CbeamB
Echoes of Light
![Page 28: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/28.jpg)
Streak Image
xz
IRSStreak Image
L
sOccluder
Hyperbolic Kernel
Streak‐camera
Laser C3rd bounce
beamB
Echoes of Light
![Page 29: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/29.jpg)
Trillion FPS
ToF Streak Camera = Inverse of CRO
![Page 30: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/30.jpg)
![Page 31: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/31.jpg)
Tim
e
Space
Third Bounce (First bounce not shown)
Space
![Page 32: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/32.jpg)
Tim
e
Space
Third Bounce (First bounce not shown)
Space
![Page 33: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/33.jpg)
Tim
e
Space
Third Bounce (First bounce not shown)
Space
![Page 34: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/34.jpg)
Tim
e
Space
Third Bounce (First bounce not shown)
Space
![Page 35: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/35.jpg)
Tim
e
Space
Third Bounce (First bounce not shown)
Space
![Page 36: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/36.jpg)
![Page 37: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/37.jpg)
Forward Reconstruction Invertibility Analysis
Modified Fresnel Approximation
Backpropagation+ Carving
Scene Priors, L1 reconstruction
Resolution and dimensionsApproximation + Carving L1 reconstruction
via COSaMPdimensions
![Page 38: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/38.jpg)
Inverting Light Transport
Multiple Scattering Direct/Global
[Seitz , Kutulakos, Matsushita 2005] [Nayar, Raskar et al 2006]
Dual Photography LIDARDual Photography
[Sen et al 2005]
LIDAR
![Page 39: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/39.jpg)
Rescue and Planningg
![Page 40: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/40.jpg)
Robot, Car Path Planning, g
![Page 41: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/41.jpg)
Endoscopypy
![Page 42: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/42.jpg)
Time Resolved Multi‐path Imaging: Plans
Scene with hidden elements
Capture
Lasers, scene, sensors, ICCD
Capture time
profiles Raw Data
Signal Proc
Photo, geometry
t
Proc.
Novel light transport models and inference
algorithms
around the corner
3D Time images3D Time images
Scenes: NLoS, Motion, BRDF in single shot, Volumetric (tissue)Scale: Endscopes table top room sized outdoors underwaterScale: Endscopes, table‐top, room‐sized, outdoors, underwaterInversion: Sparsity, Rank, Bounded Approx, Scene Priors, TransformsSignalProc: Compressive, SNR, Bandwidth, Noise modelsCapture: Coding in space/time/wavelgth Solid state non‐linear opticsCapture: Coding in space/time/wavelgth, Solid state, non linear opticsSpectrum: Radar, Sonar/Ultrasound
![Page 43: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/43.jpg)
Computational Light TransportMIT 2008‐
1. Time resolved
Descattering Spatial Augmented Rank‐constraint of Computational Wavefront Compressive
2. Angle resolved
Analysis Heterodyning Light Field 3D Displays Probes Sensing Sensing
CAT‐Scan withouti
BiDi Screen Geometric+ Wave optics
Glasses Free 3D Dual Layer LCD
Bokode NETRA SparsityAnalysis
moving parts
2009 ‐ 2008‐ 2008‐ 2009 ‐ 2008 ‐ 2010 2009 ‐
![Page 44: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/44.jpg)
LCD = a big flat camera?g
![Page 45: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/45.jpg)
Two Layer Displays19081903
Angular Information
19081903
Parallax Barrier = dim displays Lenslets = fixed low space/angle resolution
Pin hole Lensletarray
Lensletarray
sensor/display sensor/display
Spatial Heterodyning
New Solution:
using a patterned mask
![Page 46: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/46.jpg)
Beyond Multi-touch: Thin LCD for touch+hover
Mobile
Laptops
![Page 47: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/47.jpg)
BiDi Screen: Multi-touch + Hover 3D interface
![Page 48: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/48.jpg)
Sensing Depth from Array of Virtual Cameras in thin LCDy
Funding : Samsung SAITHirsch, Holtzman, Lanman, Raskar, SiggraphAsia 2009
![Page 49: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/49.jpg)
CAT Scan without moving parts
With , Berthold Horn, EECS,Dick Lanza, Nuclear Engg.
![Page 50: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/50.jpg)
Rank Analysis of 3D Pbarriers Displays
kL[i k]
i
L[i,k]
`
g[k]k
i L[i k]`f[i] L[i,k]
light box
][][],[ kgifkiL gfL
![Page 51: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/51.jpg)
Glasses Free 3D using High Rank Displays
G
FL̀~ =
0,for ,21 min arg 2
GF, GFFGL
W Content-Adaptive Parallax Barriers
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
E l it t t d ti ll b iExploit content-adaptive parallax barriers
Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010
![Page 52: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/52.jpg)
Light Rays vs WavesLight Rays vs Waves
WDFWigner Distribution Function Augmented
Li ht Fi ld
WDF
Function
Light Field LF
Light Field
Rays WavesAugmented Light Field
Supports diffraction/interferenceSupports diffraction/interferenceRadiance = Positive/Negative
![Page 53: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/53.jpg)
Computational Light TransportMIT 2008‐
1. Time resolved
Descattering Spatial Augmented Rank‐constraint of Computational Wavefront Compressive
2. Angle resolved
Analysis Heterodyning Light Field 3D Displays Probes Sensing Sensing
CAT‐Scan withouti
BiDi Screen Geometric+ Wave optics
Glasses Free 3D Dual Layer LCD
Bokode NETRA SparsityAnalysis
moving parts
2009 ‐ 2008‐ 2008‐ 2009 ‐ 2008 ‐ 2010 2009 ‐
![Page 54: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/54.jpg)
NETRA: Interactive Display for Estimating Refractive Errors and Focal RangeRefractive Errors and Focal Range
Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar
54
![Page 55: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/55.jpg)
NETRA: Near Eye Tool for Refractive Assessment
Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar
55
![Page 56: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/56.jpg)
2Brefractive errorsrefractive errors
0.6B t d
4.5B withMobile phone
uncorrected refractive errors
6.5 Billion people NETRA at LVP Eye Institute
![Page 57: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/57.jpg)
Needs expert, Moving parts, Shining lasers
Retinoscope w/ Lenses
Auto‐refracto‐meter
Chart with Lenses
In‐Focus: Focometer Optiopia
Solo‐health: EyeSite
NETRALenses meter Lenses EyeSite
Technology Shining Light plus lenses
FundusCamera
Moving lenses + target
Moving lenses + target
Reading chart on monitor
Cellphone+ eyepiece
Cost to buy $2,000* ~$10,000 ~$100 ~$495 ~$200 ‐‐ $30
C $36 $36 $ $1Cost per test ~$36 ~$36 ~$5 ‐‐ ‐‐ ‐‐ ~$1
Data capture No Comp. No No No Comp. Phone
Mobility <500g >10Kg 2kg 1kg <5kg >10Kg <100g
S d F t F t M di M di F t F tSpeed Fast Fast Medium Medium ‐‐ Fast Fast
Scalability No No No Yes Probably No Yes
Accuracy 0.15 0.15 0.5 0.75 ‐‐ ‐‐ <0.5
Self evaluation No No Yes Yes Yes Yes YesSelf evaluation No No Yes Yes Yes Yes Yes
Electricity Req No Yes No No ‐‐ Yes No
Astigmatism Yes Yes Yes/No No ‐‐ Yes Yes
Network No Yes No No No Yes YesNetwork No Yes No No No Yes Yes
Training High High High Medium Medium Low Low
* Phoropter‐based: $5,000.00
![Page 58: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/58.jpg)
Shack‐Hartmann Wavefront SensorShack‐Hartmann Wavefront Sensor
Wavefront aberrometer
Expensive; Bulky, Requires trained professionals58
![Page 59: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/59.jpg)
Shack‐Hartmann Wavefront Sensor
Laser
Shack & Platt 1971Liang et al 1994
David Williams et al, Rochester
Spot Diagram
Sensor Microlens Planar
WavefrontArray
59
![Page 60: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/60.jpg)
Shack‐Hartmann Wavefront Sensor
LaserSpot Diagram
Sensor
Displacement = Local Slope
of the Wavefront Shack‐Hartmann ~ Lightfields
Oh, Raskar, Barbastathis 2009: Augmented Light Field
![Page 61: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/61.jpg)
NETRA = Inverse of Shack‐Hartmann
Spot Diagram on LCD
Cell Eye Phone Display
yPiece
61
![Page 62: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/62.jpg)
NETRA = Inverse of Shack‐Hartmann
Spot Diagram on LCD
Cell Eye Phone Display
yPiece
62
![Page 63: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/63.jpg)
Inverse of Shack‐HartmannUser interactively creates the Spot Diagram
Spot Diagram on LCD
Displace 25
63
ppoints with smart UI
![Page 64: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/64.jpg)
Inverse of Shack‐HartmannUser interactively creates the Spot Diagram
Spot Diagram on LCD
Displace 25
64
ppoints but 3 parameters
![Page 65: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/65.jpg)
Cataract screening using inverse Shack Hartmanninverse Shack‐Hartmann
Under review 2011
![Page 66: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/66.jpg)
Limitations
• Ability to align linesChildren– Children
– Retinal conditions
– Accomodation cues
• Resolution is a function of the display DPI– Samsung Behold II – 160 DPI – 0.35D
– Google Nexus One – 250 DPI – 0.2D
– Apple iPhone 4G – 326 DPI – 0.14D
![Page 67: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/67.jpg)
NETRA: Refraction + Cataract Tests
• Inverse of Shack‐Hartmann wavefront aberrometerHi h l ti di l d i t ti– High‐resolution displays and user interaction
– No lasers, moving parts
– Trials in progress
H d t– Hardware app store
• Parameters– Myopia, Hyperopia, Astigmatism
– Cataract, Lazy eye
• Impact in Developing Countries– 600 Million without corrective glasses
– $1 cost, easy to deploy, free s/w, see EyeNetra.comy p y y
67
![Page 68: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/68.jpg)
Clinical Testing Partners
![Page 69: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/69.jpg)
NETRA in a dozen+ countriesNETRA in a dozen+ countries
69
![Page 70: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/70.jpg)
Kenya India
Awards Selection
• MIT IDEAS (#2 award)
• Deshpande Ignition Grant
• MIT 100K (dev finalist)
• NASA/USAID Innovators
• Worldbank Social Health Inventions
• International Space Station evaluationMIT 100K (dev finalist)
• Google + (L Page)
• Vodafone Foundation (finalist)
International Space Station evaluation
Validation• 0.09 D : objective precision
Papers
• SIGGRAPH
70• ~ 0.5 D: subjective trials
• 0.3 D: IRB approved wet‐studies (Prelim data)
• Frontiers in Optics
• Am Academy of Optometry (AAO)
![Page 71: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/71.jpg)
Chicken Eye Inducing Cataract
‘Lens’ Imaging CCD
![Page 72: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/72.jpg)
Optics: Bokodes = Human EyeBokode Capture = Retinal ImagingBokode Capture = Retinal Imaging
cell-phone cameraclose to the Bokodeclose to the Bokode
(10,000+ bytes of data)
Like a Retinal Image
![Page 73: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/73.jpg)
Eye = Mirror of Health
73Yoon Lab, U of Rochester
![Page 74: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/74.jpg)
Slit Lamp Exam Retinal Scanp
Ocular Manifestation: Leading Indicator?
74
![Page 75: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/75.jpg)
T di i l U D iTraditional User Driven
Mass‐use Devices ‐> Scientific Instruments
![Page 76: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/76.jpg)
Current and Former Members• Post-docs
• Ankit Mohan• Andreas Velten• Douglas Lanman• Yunhee Kim
• RAs• Andy Bardagjy• Kevin Chiu
Matthew Hirsch (w Holtzman)• Matthew Hirsch (w Holtzman)• Roarke Horstmeyer• Otkrist Gupta• Ahmed Kirmani• Jaewon Kim• Nikhil Naik• Rohit Pandharkar
• MEng• Tyler Hutchison
Sh B k• Sharmeen Browarek• Dennis Miaw
• Visitors• Vitor Pamplona• Abhijit Bendale
• Daniel Saakes• Grace Woo
T Cb j da
• Erick Passos• Behzad Sajjadi• Gordon Wetzstein• Matthias Hullin
• Tom Cuypers• Manuel Oliviera• Shinsaku Hiura• Yasuhiro Mukaigawa
![Page 77: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/77.jpg)
Codesigning Optical and Digital Processing
Computational
s
Optics
Displays
Computational Light Transport
hotons
Sensors Computational Photography
Ph
g p y
Computer VisionSignal Processing
Machine Learning
Bits
Machine Learning
![Page 78: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/78.jpg)
Computational Light Transport
Ramesh Raskar http://raskar.info
OpticsComputational Light TransportComputational Light Transport
– Super‐human visual abilities
– Empirical, Multi‐directional rather than one narrow field
F i f di i ilBits
Photon
s
Computer Vision
Sensors
Signal Processing
Displays
Machine Learning
Computational Photography
– Fusion of dissimilar
• New Fields– Femto‐photography
Dream Augmentation Image IP– Dream Augmentation, Image IP
• New Insights– Challenge the status quo
– BiDi Screen, CAT‐scan, Augmented LFBiDi Screen, CAT scan, Augmented LF
– Sparsity, rank, priors
• New Purpose– Disruptive mass‐use tech for social impact F
G
L̀~
– Netra, Cataract, Retinal scans
– $1M ‐> $1 0,for ,21 min arg 2
GF, GFFGL
W
=
Cameras, Displays, Medical Tools, Future Devices– Theory, Modeling, Optical+Mathematical insight
![Page 79: Ramesh Raskar - MIT Media Labraskar/11Sig/Movies/Old/RaskarFacultyP… · Bit Hacking. Computational Illumination ... UltraFast Detector Computational Optics Serious Sync •DarpaYoung](https://reader036.vdocument.in/reader036/viewer/2022071408/60ffc74b26db15424d47ab3c/html5/thumbnails/79.jpg)
Computational Light TransportMedia Lab 2008‐
1. Time resolved
Descattering Spatial Augmented Rank‐constraint of Computational Wavefront Compressive
2. Angle resolved
Analysis Heterodyning Light Field 3D Displays Probes Sensing Sensing
CAT‐Scan withouti
BiDi Screen Geometric+ Wave optics
Glasses Free 3D Bokode NETRA SparsityAnalysis
moving parts
2009 ‐ 2007‐ 2008‐ 2009 ‐ 2008 ‐ 2010 2009 ‐