personal photo enhancement using example images neel joshi wojciech matusik, edward h. adelson, and...
TRANSCRIPT
![Page 1: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/1.jpg)
Personal Photo Enhancement using Example ImagesNeel JoshiWojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research, Adobe Research, MERL, MIT CSAIL, and UCSD
![Page 2: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/2.jpg)
2Motivation and Approach
It is difficult for most users to fix their images
It’s easier for users to rate their good photos
Use examples of a persons good photos to fix the bad ones automatically
ü X
![Page 3: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/3.jpg)
3Our Approach
Focus on images with faces
Use a known face as a calibration object
Users provide good examples, instead performing manual edits
X
X
![Page 4: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/4.jpg)
4Previous Work
Deblurring and Upsampling/Super-Resolution Poisson image/noise models [Richardson 1972; Lucy 1974]; Sparse
gradient priors [Fergus et al. 2006; Levin 2006; Levin 2007]; Sparse wavelet coefficients [de Rivaz 2001]; Spatially Varying [Whyte et al. 2010; Gupta et al. 2010]; Baker and Kanade 2000; Freeman et al. 2000; Freeman et al. 2002; Liu et al. 2007; Dai et al. 2007; Fattal 2007
Denoising Sparse wavelet coefficients [Simoncelli and Adelson 1996; Portilla et
al. 2003], Anisotropic diffusion [Perona and Malik 1990], Field of Experts [Roth and Black 2005];, Baker and Kanade 2000; Freeman et al. 2000; Freeman et al. 2002; Liu et al. 2007; Dai et al. 2007; Fattal 2007
White-Balancing/Color Correction Finlayson et al. 2004, 2005; Weijer et al. 2007
Using photo collections Baker and Kanade 2000, Liu et al. 2007 , Dale et al. 2009
Hardware Methods Joshi et al. 2010, Raskar et al. 2008, Levin et al. 2008, Veeraraghavan
et al. 2007, Levin et al. 2007, Raskar et al. 2006, Ben-Ezra et al. 2005, Ben-Ezra and Nayar 2004
![Page 5: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/5.jpg)
5Specific vs. General Priors
We use an identity specific prior
Generic Image Prior Multi-Image
Field of Experts [Roth and Black]
Sparse Prior [Levin et al.]
Example Based[Freeman et al.]
Our Approach
X
Photo Collections[Dale et al.]
![Page 6: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/6.jpg)
6Facespace
Faces are a subspace of all images Eigenfaces -- Turk and Petland 1987
Person-specific space is relatively small
The range of images can be captured with a few good examples
![Page 7: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/7.jpg)
7Personal Image Enhancement Pipeline
FACE DETECTION ALIGNMENT
GLOBAL AND LOCALENHANCEMENT
FINAL ENHANCED IMAGE
GOOD IMAGES
BADIMAGE
INTRINSIC IMAGE
DECOMPOSITION
INTRINSIC IMAGE
DECOMPOSITION
![Page 8: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/8.jpg)
8Intrinsic Images [Land and McCann 1971,Barrow and Tenenbaum 1978]
Separation into Lighting, Texture, Color Layers
Use base/detail decomposition of Eisemann and Durand 2004
Input Image
Chroma R Detail/Texture
Chroma G Lighting
![Page 9: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/9.jpg)
9Image Enhancements
Blur (Global)
Color/Exposure Balance (Global)
Super-Resolution/Up-sampling
![Page 10: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/10.jpg)
10Image Enhancements
Blur
Color/Exposure Balance
Super-Resolution/Up-sampling
![Page 11: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/11.jpg)
11Blur Formation
=
Blurry image
Blur kernel(Point-Spread
Function)
+
Zero Mean Gaussian NoiseSharp image
Convolution
![Page 12: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/12.jpg)
12Blur Estimation Goal
=
Blurry image
Blur kernel
+
Zero Mean Gaussian NoiseSharp image
Known
Unknown
Known s
![Page 13: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/13.jpg)
13Deblurring: Multiple Possible Solutions
=
Blurry image
Sharp image Blur kernel
=
=
![Page 14: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/14.jpg)
14Eigenspace
Identity Specific Images are used to build an aligned eigenspace
Mean Face Eigenvectors * 3 *s + Mean Face
Eigenvectors * -3 *s + Mean Face
![Page 15: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/15.jpg)
15
2
43
2
8.01
2
,
)(
minarg,
KK
II
IKIBKI
p
T
KI
Eigenspace used for Blind Deconvolution
Eigenspace used as a linear constraint
Robust norm
Sparsity and smoothness priors on the Kernel
Solved using an Multi-Scale EM style algorithm
B = Blurry ImageI = Sharp PredictionL = Eigenbasis vectors = m Mean Vector
r(.) = Robust Norms = Noise standard deviationl = Regularization parameterp < 1
Data Term Sparse Prior
![Page 16: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/16.jpg)
16Image Enhancements
Blur
Color/Exposure Balance
Super-Resolution/Up-sampling
![Page 17: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/17.jpg)
17Image Enhancements
Blur
Color/Exposure Balance
Super-Resolution/Up-sampling
![Page 18: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/18.jpg)
18Color Correction: Multiple Possible Solutions
= XObserved image
White-balanced Image Lighting Color
= X
![Page 19: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/19.jpg)
19White Balance and Exposure Correction
Diagonal white balancing matrix (scales r and g independently)
Exposure adjustment scales lighting layer
rCC rrC
rr
minargCr = r scaleCg = g scaleCL = L scale
mr = Mean r Vectormg = Mean g VectormL = Mean L Vector
r(.) = Robust Norm
gCC ggC
gg
minarg
LCC LLC
LL
minarg
![Page 20: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/20.jpg)
20Image Enhancements
Blur
Color/Exposure Balance
Super-Resolution/Up-sampling
![Page 21: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/21.jpg)
21Image Enhancements
Blur
Color/Exposure Balance
Super-Resolution/Up-sampling
![Page 22: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/22.jpg)
22Face Correction: Patch Based [Freeman et al. 2000, Liu et al. 2007]
• High-frequencies hallucinated by minimizing the energy of patch-based Markov network
• Two types of energies:
• external potential — to model the connective statistics between two linked patches in and .
• internal potential — to make adjacent patches in smooth.
• Energy minimization by raster scan [Freeman et al. 2000]
gHI
)(vI gH
)(vN lH
lHI
)(vS
)(vI lH
I
LHI
GHI
LHI
![Page 23: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/23.jpg)
Results
![Page 24: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/24.jpg)
24Camera Motion Blur (Global Correction)
Good Example Images
![Page 25: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/25.jpg)
25Exposure Correction and White-Balancing
Good Example Images
![Page 26: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/26.jpg)
26Defocus Blur (Local Correction)
Good Example Images
![Page 27: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/27.jpg)
27Upsampling (Local Correction)
Good Example Images
![Page 28: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/28.jpg)
Comparisons
![Page 29: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/29.jpg)
29Comparisons to Previous Work
Our ResultFergus et al. 2006
![Page 30: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/30.jpg)
30Comparisons to Color Constancy [Weijer et al. 2007 ]
Grayworld Shades of Gray
Our ResultsGrayedgeMaxRGB
![Page 31: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/31.jpg)
31Using Generic FacesOur Result Liu et al. 2007
Our Result
Liu et al.
Generic (10)
Generic (50)
Generic Faces (10)
Generic Faces (50)
![Page 32: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/32.jpg)
32
Our Result Liu et al. 2007 Generic Faces (10)Generic Faces (50)
Using Generic FacesInput
![Page 33: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/33.jpg)
33Discussion/Future Work
Latent photo may not be well modeled by the Eigenspace
All parts of the Eigenspace may not be equally likely
A prior on the distribution within the Eigenspace
Better non rigid alignment/morphable model
Personalized Enhancement on camera/phone
![Page 34: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/34.jpg)
34Contributions
We use good examples of known face images for corrections
Faces are used as calibration objects for global corrections
We can further improve the faces in images
Identity-specific priors out-perform generic priors
ü
![Page 35: Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research,](https://reader035.vdocument.in/reader035/viewer/2022062423/56649d1b5503460f949f13e0/html5/thumbnails/35.jpg)
35Thank You!
http://research.microsoft.com/en-us/um/people/neel/personal_photos/