practical scene illuminant estimation via flash/no-flash pairs
DESCRIPTION
Practical Scene Illuminant Estimation via Flash/No-Flash Pairs. Cheng Lu and Mark S. Drew Simon Fraser University {clu, mark}@cs.sfu.ca. Flash/No-flash Imagery – a Brief History. diCarlo, Xiao, & Wandell, CIC 2001. Combine flash/no-flash images to produce a pure-flash image. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/1.jpg)
1
Practical Scene Practical Scene Illuminant Estimation via Illuminant Estimation via
Flash/No-Flash PairsFlash/No-Flash Pairs
Cheng Lu and Mark S. Drew
Simon Fraser University{clu, mark}@cs.sfu.ca
![Page 2: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/2.jpg)
2
Flash/No-flash Imagery – a Brief History
diCarlo, Xiao, & Wandell, CIC 2001Combine flash/no-flash images to produce a pure-flash image.Use dim=3 FDM + knowledge of flash SPD and sensor curves to estimate surface reflectance most likely ambient illuminant
Raskar et al., Non-Realistic Rendering 2004
Filling in night-time imagery with daytime image info.
Copy edges from cloned image region into edge-map of target background; re-integrate.
Blake et al., Poisson Image Editing, Siggraph 2004
Szeliski et al., Siggraph 2004
Transfer lower-noise information from flash image to higher-noise ambient-light image.
Find shadow-mask, copy edges inside shadow from flash image into ambient image, re-integrate.
Drew,Lu,Finlayson, Removing Shadows using Flash/Noflash Image Edges , ICME 2006
![Page 3: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/3.jpg)
3
This paper:
Estimate Ambient Illuminant, usingFlash/No-flash Pairs
Like diCarlo&Wandell approach, but replace knowledge of camera sensor curves with a camera RGB-based calibration using difference of with-flash and no-flash images. How?
- Spectral sharpening- Subtract “both” – “no-flash” pure-flash image- Log’s- Project difference of flash minus ambient into geometric-mean chromaticity color space
Calibrate such to get illuminant chromaticity.
![Page 4: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/4.jpg)
4
What’s the point?:
Can estimate scene (ambient) illuminant without knowing:
- Flash SPD- Camera sensors- Surface reflectance
![Page 5: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/5.jpg)
5
Why estimate the illuminant?
White balance, plus many computer vision applications == intrinsic imageswithout illumination.
- Simple- Fast
What’s good about this method?
![Page 6: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/6.jpg)
6
The set-up:
2 images , one under ambient lighting, & another under flash.
Under Ambient: Image “A”. Under Both: Image “B”.
+
![Page 7: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/7.jpg)
7
The Key: Pure-Flash Image The ambient light from “A” is also in “B”. Therefore if we subtract the two, we have “F”: the pure-flash image.
Under Flash: Image “F”:
+ - =)(
![Page 8: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/8.jpg)
8
Incidentally, note that there are now extra shadows, from the flash(since it’s offset from the lens).
Image “F”: the scene as imaged under Flash light only.
![Page 9: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/9.jpg)
9
1. Lambertian surface:
RGB =
Shading = normal effective light-direction
Illum
inant
Surfa
ceSen
sors
Simple Image Formation Model
will guide us.
Assumptions: 1., 2., 3.
![Page 10: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/10.jpg)
10
)(kQ
2. Narrow-band sensors:
so then
is exactly a single-spike sensor:
![Page 11: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/11.jpg)
11
3. Planckian light:
But, can violate 1., 2., 3. and still succeed.
(in Wien’s approximation)
Gives
![Page 12: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/12.jpg)
12
-Now take Log’s, to pull apart multiplications:
Camera-dep’t vector
Camera-dep’t vector
Intensity and shading
Surface
Color-temperature of light
where
![Page 13: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/13.jpg)
13
Camera-dep’t vector
Surface
Color-temperature of light
So form geometric-mean chromaticity:
-We’d like to remove intensity/shading term:
In logs:
where
![Page 14: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/14.jpg)
14
-The point:
As temp (light color) changes, move along straight line.
-But, we have “A” and “F” images: Subract them, and use same chromaticity trick Only illumination is left!
![Page 15: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/15.jpg)
15
Log-difference Geometric-Mean Chromaticity
So log-log delivers inverse-temperature difference:
-Calibrate for 1/TA-1/TF, then in new scene obtain TA!
{
![Page 16: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/16.jpg)
16
What does this
look like?
Moved to 2D; color-matching functions in geo-mean chromaticity. (9 Planckians, Macbeth ColorChecker, spike sensors, xenon flash SPD)
![Page 17: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/17.jpg)
17
Sony DXC930 sensors, Daylights+F2, actual xenon flash SPD:
“Reference locus”
How to proceed: -Sharpen- Find closest cluster
![Page 18: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/18.jpg)
18
Effect of sharpening:
Poor clusters Better clusters#’ing
Kodak DCS420:
![Page 19: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/19.jpg)
19
Test: can we determine the illuminant?
102 illuminants, Sony camera, Munsell patches
102 illuminants, Sony camera, Macbeth patches
Estimate illum.from Munsell to Macbeth
Nearly 100% correctly identified.
![Page 20: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/20.jpg)
20
Application: White Balance
4 calibration illuminants, HP camera, Macbeth chart(each cluster has 24 dots)
No flash
With flash
- Sharpen- Sample image at 24 locationsevenly over image-Same (“daylight”) color balancefor training and for testing
Image under CWF; CWF+Xenon
![Page 21: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/21.jpg)
21
Overlaps best with CWF, so usewhite patch of Macbeth under CWF for white balance:
“Auto” balance – Wrong.
“Fluor” balance – Correct.
Our color-balance– Much closer.
![Page 22: Practical Scene Illuminant Estimation via Flash/No-Flash Pairs](https://reader034.vdocument.in/reader034/viewer/2022051218/5681581b550346895dc580fd/html5/thumbnails/22.jpg)
22
Thanks!To Natural Sciences and
Engineering Research Council of Canada