image restoration with neural networks · several types of noise involved in the image formation:...
TRANSCRIPT
![Page 1: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/1.jpg)
Work with Hang Zhao, Iuri Frosio, Jan Kautz
IMAGE RESTORATION WITH NEURAL NETWORKSOrazio Gallo
![Page 2: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/2.jpg)
![Page 3: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/3.jpg)
![Page 4: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/4.jpg)
MOTIVATIONThe long path of images…
Demosaic Denoise
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black
Level
Metering
AF/AE
Image Signal Processor (ISP)
![Page 5: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/5.jpg)
DEMOSAICINGcolors by interpolation
Image c
redit
: W
ikip
edia
Image c
redit
: M
arc
Levoy
![Page 6: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/6.jpg)
DENOISINGSeveral types of noise involved in the image formation:
• Photon shot noise
• Dark current (AKA thermal noise)
• Photo-response non-uniformity
• Vignetting
• Readout noise:
• Reset noise (charge-to-voltage transfer)
• White noise (during voltage amplification amplification)
• Quantization noise (ADC)
![Page 7: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/7.jpg)
DENOISING
![Page 8: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/8.jpg)
MOTIVATION
Demosaic Denoise
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black
Level
Metering
AF/AE
Demosaicing before denoising changes the
statistics of the noise. And the best de-noising
algorithms require to know what the noise looks like.
![Page 9: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/9.jpg)
MOTIVATION
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black
Level
Metering
AF/AE
Denoising first can change the color reproduction
accuracy as the three channels may be denoised
differently.
![Page 10: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/10.jpg)
PSF CFA Noise
[1] Heide et al., ACM SIGGRAPH Asia 2012 (ToG)
FLEXISP1
A Flexible Camera Image Processing Framework
![Page 11: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/11.jpg)
CAN WE DO IT WITH A NEURAL NETWORK?
Can we do it with a neural network, which moves
the heavy lifting to the training stage and inference
is very quick?
![Page 12: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/12.jpg)
JOINT DEMOSAICING AND DENOISING
![Page 13: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/13.jpg)
JOINT DEMOSAICING AND DENOISINGNetwork architecture
convolu
tion
convolu
tion
convolu
tion
bilin
ear
inte
rpola
tion
![Page 14: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/14.jpg)
MEASURING IMAGE QUALITY
Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/
Original
![Page 15: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/15.jpg)
Higher sensitivity to errors in texture-less regions!
MEASURING IMAGE QUALITY
Wang, et al. "Image quality assessment: from error visibility to structural similarity." IEEE TIP (2004)
![Page 16: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/16.jpg)
MEASURING IMAGE QUALITY
Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/
Original
0.988 0.662
![Page 17: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/17.jpg)
MEASURING IMAGE QUALITYHigher sensitivity to errors in texture-less regions!
![Page 18: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/18.jpg)
JOINT DEMOSAICING AND DENOISINGNetwork architecture
convolu
tion
convolu
tion
convolu
tion
bilin
ear
inte
rpola
tion
![Page 19: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/19.jpg)
JOINT DEMOSAICING AND DENOISINGNetwork training
Training data 31 x 31 patches from 700, 999x666 RGB
images (MIT-Adobe FiveK dataset)
Input - noisy image (realistic noise model)
- bilinear interpolation
Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM
![Page 20: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/20.jpg)
Ground truth
![Page 21: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/21.jpg)
Noisy
![Page 22: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/22.jpg)
![Page 23: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/23.jpg)
![Page 24: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/24.jpg)
RESULTSVisual comparison (+ unsharp masking)
Noisy BM3D (state of the art) Ground truth
![Page 25: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/25.jpg)
Noisy
![Page 26: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/26.jpg)
![Page 27: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/27.jpg)
![Page 28: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/28.jpg)
RESULTSVisual comparison (+ unsharp masking)
Noisy BM3D (state of the art) Ground truth
![Page 29: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/29.jpg)
JOINT DEMOSAICING AND DENOISING: RESULTS
Average image quality metrics on the testing dataset
![Page 30: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/30.jpg)
DOES IT GENERALIZE?
JPEG ARTIFACT REMOVAL&
SUPER-RESOLUTION
![Page 31: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/31.jpg)
JPEG ARTIFACT REMOVALNetwork training
Training data 31 x 31 patches from 700, 999x666 RGB
images (MIT-Adobe FiveK dataset)
Input JPEG compressed image, 25% quality
Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM
![Page 32: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/32.jpg)
JPEG ARTIFACT REMOVAL: RESULTSVisual comparison (+ unsharp masking)
Ground truthL1 + MS-SSIML2JPEG
![Page 33: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/33.jpg)
JPEG ARTIFACT REMOVAL: RESULTSNumerical comparison
Average image quality metrics on the testing dataset
![Page 34: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/34.jpg)
SUPER-RESOLUTIONNetwork training
Training data 31 x 31 patches from 700, 999x666 RGB
images (MIT-Adobe FiveK dataset)
Input 2x downsampled image + upsampled
with bilinear interpolation
Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM
![Page 35: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/35.jpg)
SUPER-RESOLUTION: RESULTSVisual comparison (+ unsharp masking)
L1 + MS-SSIML2Low rez
![Page 36: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/36.jpg)
SUPERRESOLUTION: RESULTSNumerical comparison and literature
![Page 37: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/37.jpg)
LEARNINGS?
![Page 38: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/38.jpg)
LEARNINGSA closer look at the different losses
![Page 39: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/39.jpg)
LEARNINGSand
![Page 40: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/40.jpg)
LEARNINGSand
0.3939
0.3896
• seems to have more convergence issues.
• converges faster and speeds up the convergence or other losses, too.
![Page 41: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/41.jpg)
LEARNINGSA closer look at the different losses
![Page 42: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/42.jpg)
LEARNINGSSSIM and MS-SSIM
“Higher sensitivity to errors in texture-less regions!”
• Multi-scale is helpful when dealing with transition regions.
![Page 43: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/43.jpg)
LEARNINGSA closer look at the different losses
![Page 44: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/44.jpg)
RESULTSWhy mixing MS-SSIM and ?
![Page 45: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/45.jpg)
CONCLUSIONS
• Even a shallow network can produce state-of-the-art results…
• …if you train it carefully.
• Perceptually-motivated loss functions can help!
• But you have to be aware of their limitations!
What have we learnt?
![Page 46: IMAGE RESTORATION WITH NEURAL NETWORKS · Several types of noise involved in the image formation: • Photon shot noise • Dark current (AKA thermal noise) • Photo-response non-uniformity](https://reader034.vdocument.in/reader034/viewer/2022050315/5f77ec35728f776dd1343996/html5/thumbnails/46.jpg)
Thanks!
Zhao, Gallo, Frosio, and Kautz,“Loss Functions for Image Restoration with Neural Networks”,
IEEE Trans. on Comp. Imaging, 2017