diffuse reflection imaging: earthshine and other faint signals sam hasinoff mit csail, ttic,...
TRANSCRIPT
Diffuse Reflection Imaging:Earthshine and other Faint Signals
Sam HasinoffMIT CSAIL, TTIC, Google[x]
Samuel W. Hasinoff, Anat Levin, Philip R. Goode, and William T. Freeman, Diffuse Reflectance Imaging with Astronomical Applications,Proc. 13th IEEE International Conference on Computer Vision, ICCV 2011http://people.csail.mit.edu/hasinoff/diffuse/
2
Bill FreemanMIT CSAIL
Joint work with:
Anat Levin Weizmann Institute
Philip GoodeBig Bear Solar Observatory
Special thanks to: Bernhard Schölkopf, Frédo Durand, Livia Illie, David Chen
3
Indirect Imaging with Reflective Objects
specular reflectance
virtual camera
glossyreflectance
bad camera?
diffuse reflectance
really bad camera?
-180° 0° +180°0
1
4
Resolution of a Diffuse Reflector
Diffuse surfaces blur together incident lighting from many directions
How much detail can we resolve?
Lambertian reflectance
5
Single-Bounce Light Transport
reflectedradiance
surfacealbedo
incomingradiance
BRDF clipped cosinevisibility
reflectance = (albedo lighting) BRDF for convex objects, distant illumination
*.-180° 0° +180°0
1
6
Linear Light Transport
Usual matrix formulation
observedpixels
transfermatrix
distantlighting
noise
transfer matrix encodes geometry, BRDF, surface albedo
7
Resolution of a Diffuse Reflector
about 9 pixels?images of a convex Lambertian objects
with distant lighting lie in a 9D subspace (<2% average error)
[Basri and Jacobs, 2001][Ramamoorthi and Hanrahan, 2001]
e.g. lighting-insensitive recognition with 9 basis images
…
8
Key: Occlusion Geometry Codes Illumination
convexabout 9 pixels?
self-occlusionpotentially much better
9
Key: Occlusion Geometry Codes Illumination
convexabout 9 pixels?
self-occlusionpotentially much better
What 3D shape lets us best resolve the lighting?
10
Sculpture Design for Reflectance Imaging
double-pinholes coded reflectance natural complexity
• self-occlusion preserves high frequencies• extreme: isolate rays by building pinhole cameras
using the scene• arbitrary fidelity (in theory), limited only by diffraction
and geometric calibration
Standard MAP estimation• Gaussian prior [Wiener 1949]
• Sparse derivative prior [Levin et al. 2007]
11
Bayesian Reconstruction Method
For time-varying lighting, reconstruct stable part• marginalize out temporal variations
12
Reconstructing a Changing Target
temporal binning withGaussian prior
temporal variations as correlated noise
Approach #1: Rim Reflectance Imaging13
Exploit visibility changes near occlusion boundary• works for compact lighting, e.g.
object lit head on• multiple orientations = tomography
+ single shot enough- need high resolution- need accurate geometry
not previously used in astronomyor computer vision
Approach #2: Time Varying Imaging14
Exploit variation over time in the occlusion geometry• classical astronomy (“light curves”)
+ single pixel enough- need natural variation- need many shots- target might change
e.g. surface of Pluto from Charon’s 1985-1990 transits [Young et al. 1999]
Earth from Space from Earth - Feasibility Study
Earth from Apollo 17, 1972 (NASA)
15
Low-Resolution Expectations16
NASA’s “Blue Planet”
reconstruction we’d be happy with
Moon as Earth Reflector (“Earthshine”)
Leonardo Da Vinci’s Codex Leicester (ca. 1510)
17
18
sunlight
Earth north pole
(Exaggerated) Geometry of Earthshine
Moon north pole Moon from observer
observer
19
sunlight
Earth north pole
Moon north pole Moon from observer
observer
Earthshine
Moonshine
(Exaggerated) Geometry of Earthshine
Integrating over Earth from the Moon20
Sun and
Moon tovisible EarthEarth ),,Sun(BRDF)ˆˆ()ˆunS( dRIRIRRII aaR
albedo of “unknown” Earth patch
Earth phase (≈ 180° - Moon phase)
cos(incoming)cos(outgoing)
Aug 8, 2010, 5am Feb 24, 2010, 1am
Earth integrand, from point on the Moon (simulation)
Earthshine as Linear Transfer Matrix21
y = T x + n∙observations of
earthshine Earthimage
Earth-Moontransfer matrix
noise
22
Recipe for Earthshine Imaging
feasible observing times: • moon >2° above horizon (air
mass)• moon <85% full
(moonshine)• sun >8° below horizon
(twilight)• clear skies
track moon - multi-minute exposures> -2.5 app. mag. (V)
block moonshine glare with occluder
BBSO [Qiu et al. 2003]
Approach #1: Visibility at Moon Rim
Earthrise, as seen from Apollo 8 (NASA)
23
• From rim of Moon, different Earth regions visible
• Special challenges:• need very high resolution, SNR• need detailed moon topography• BRDF at grazing angles?
24
Best Case Earth-Bound Moon Imaging
8m telescopeESO Very Large Telescope (Chile)0.07” resolution26,000 pixels across lunar disk
by comparison, HST is 0.05” (i.e. near diffraction limit for it 2.4m mirror)
http://optics.org/article/9654/
25
ground truth earth model (75x30 pixels)
relative contribution,Lambertian Earth
moon simulation, 2475 samples (55 per radial line)
Sampling the Earth and Moon Surface
26
Simulated Reconstruction for Single Shot sp
arse
prio
r rec
onst
ructi
on
std. optics (0.4”) adaptive optics (0.07”)
80 d
B
60
dB
40 d
B
Approach #2: Scanning over Time27
• Over time, Earth region visible to Moon & Sun changes• Single pixel observations is fine
• Special challenges:• 1D set of viewpoints per day• time varying cloud cover, snow, etc.
region of Earth visible from Moon’s disk, i.e. contributing to earthshine
1am, Feb 24, 2010 5am, Feb 24, 2010 2am, Aug 14, 2010
28
Simulated Reconstruction over Time
1 nightAug 12-13, 2010
107 obs. (@3 min)
1 monthAug, 2010
412 obs. (@10 min)
1 yearJan-Dec, 2010
963 obs. (@60 min)
reco
n.co
ntrib
.gr
d. tr
uth
29
Simulated Reconstruction over Time
1 nightAug 12-13, 2010
107 obs. (@3 min)
1 monthAug, 2010
412 obs. (@10 min)
1 yearJan-Dec, 2010
963 obs. (@60 min)
reco
n.co
ntrib
.gr
d. tr
uth
30
Time-Varying Clouds
• 50-75% of Earth covered by clouds• large-scale systems last about 3 days
const. mean cloudreconstruction
earth with ISCCP clouds next day clouds
simulated reconstruction (1 year @3min, 60 dB)
mean clouds
naïve time-varyingreconstruction
covariantreconstruction
31
Reconstructing Mars from Light Curves
• direct observation• outer planet, so less phase change (>85% disk lit)• much less active weather than earth• axial tilt of 25° (2D directions)• more Lambertian?
Martian surface
32
Martian Reconstruction from Historical Spectra
ground truth + visibility relative contribution
234 spectra (single-pixel images) observedat 2 sites, from 1963-1965 [Irivine et al. 1968ab]
recon. from simulation (scaled) recon. from real data (hVB), 47 dB
33
Simulation: Limits of Single-Pixel Mars Reconstruction
200 samples 400 samples 800 samples
1600 samples 3200 samples 23731 samples
1 Martian year (= 2 years)single-pixel images, 60 dB, no temporal variation
10 Years of Moon Photos34
earthshine
moonshine filter ∙
Ia
Ib
• 10 years• 40k+ photos• photos every 5 min• Big Bear Solar Observatory (BBSO), near LA• single CCD (grayscale)
Moon from BBSO [Qiu et al. 2003]
The Earthshine-Moonshine Ratio35
earthshine
moonshine filter ∙
Ia
Ib
)obs,,Sun(BRDF
)obs,,Earth(BRDF
Moon
Moon
Sun
Earth
b
a
b
a
I
I
I
I
I
I
irradiance from “unknown” Earth < 1°
lunar phase 0° = full, 180° = new
Earth
atmosphere
eII 0
moonshine reference: factors out viewing effects
36
• imaging tradeoffs, e.g. blur vs. noise• sensor characteristics
2.8 dB
14.6 dB
Coda: Optimizing Capture Strategies
vs.
http://people.csail.mit.edu/hasinoff/