single image interpolation via adaptive non-local sparsity ... · single image interpolation via...
TRANSCRIPT
The Proposed Algorithm We suggest training a dictionary using the LR image itself and restore each decimated patch by a sparse
composition over the dictionary using a weighted version of the Simultaneous OMP.
The restored image is obtained by averaging the HR patches, followed by a simple projection of the known pixels in 𝐱 on the outcome.
Notations:
W𝑖,𝑗 sets a high weight for known pixels and a low one for the
unknown ones, multiplies by exp − 𝑥𝑖 − 𝑥𝑗 1/𝑐𝑤 .
𝛼𝑖𝑟 𝐀𝑖
𝑠𝑝
0,∞ counts the non-zero elements rows in the matrix 𝛼𝑖
𝑟 𝐀𝑖𝑠𝑝
.
𝛼𝑖𝑟 is the representation of the non-weighted version of the reference patch (stabilizer).
𝐀𝑖𝑠𝑝
is the representation of the weighted versions of the reference patch and its K – Nearest Neighbors.
𝐑𝑗 is an operator that extracts the jth-patch from the image.
The proposed two-stage algorithm:
1. First stage: Joint sparse-coding using the K-nearest “strong” patches and reconstructing the image using the “strongest” patches.
2. Second stage: Use all the patches (“strong” and “weak”), both in the sparse-coding and the reconstruction steps.
A Basic Observation The more known pixels within a patch, the better the restoration.
The number of known pixels depends on its location (“strong” and “weak” patches).
We suggest “increasing” the number of known pixels based on the self- similarity assumption (e.g. the bright patches are the K-Nearest Neighbors of each dark patch).
Motivation and Goals
Adaptive sparse representation modeling is a promising image prior, which has been shown to be powerful in filling-in missing pixels in an image.
Processing groups of related patches together (based on the self-similarity assumption) exploits their correspondence and leading often times to improved results.
{yaniv,matanpr,elad}@{tx,cs,cs}.technion.ac.il
The Interpolation Problem
Given a Low-Resolution (LR) image 𝐲 = 𝐔L𝐱 , where 𝐱 is the High-Resolution (HR) image and 𝐔L decimates the image by a factor of L along the horizontal and vertical dimensions, our goal is to recover 𝐱 from 𝐲.
Single Image Interpolation via Adaptive Non-Local
Sparsity-Based Modeling
The Core Idea A common patch-based image restoration scheme:
Problems…
Each patch is interpolated independently.
Sparse-coding tends to err due to small number of existing pixels.
Yaniv Romano The Electrical Engineering Department
Matan Protter The Computer Science Department
Michael Elad The Computer Science Department
Technion – Israel Institute of Technology
This research was supported by the European Research Council under EU’s 7th Framework Program, ERC Grant agreement no. 320649, and by the Intel Collaborative Research Institute for Computational Intelligence.
Dictionary Update
Initial Dictionary
Interpolate each patch
Zero-Filled Image
(Not so impressive) HR Image
- Known pixel
- Unknown pixel
LR image
Weighted OMP
Weighted KSVD
LR image
kth iteration interp. image
𝑟
ℎ
×W𝑖,𝑗 𝛼𝑖𝑟
𝑖=1𝑁 , 𝐀𝑖
𝑠𝑝
𝑖=1
𝑁≔ argmin 𝛼𝑖
𝑟 𝐀𝑖𝑠𝑝
0,∞
𝑁
𝑖=1
s. t ∀𝑖 𝐃𝛼𝑖𝑟 − 𝐑𝑖𝐱
𝑒𝑠𝑡22 + 𝐃 𝐀𝑖
𝑠𝑝
𝑗− 𝐑𝑗𝐱
𝑒𝑠𝑡
W𝑖,𝑗
2≤ T𝑖
𝑗∈𝑆𝑖
Joint weighted sparse-coding (weighted SOMP)
𝐃 , 𝛼𝑖 𝑖=1𝑁 , 𝐀𝑖 𝑖=1
𝑁 ≔
argmin 𝐃 𝛼𝑖 − 𝐑𝑖𝐱 𝑒𝑠𝑡
2
2+
𝑁
𝑖=1
𝐃 𝐀𝑖 𝑗 − 𝐑𝑗𝐱 𝑒𝑠𝑡
W𝑖,𝑗
2
𝑗∈𝑆𝑖
𝑁
𝑖=1
s. t ∀𝑖 𝑆𝑢𝑝𝑝 𝛼𝑖 𝐀𝑖 = 𝑆𝑢𝑝𝑝 𝛼𝑖𝑟 𝐀𝑖
𝑠𝑝
Dictionary Update (weighted KSVD)
𝐱 ≔
argmin 𝐃 𝐀𝑖𝑠𝑝
𝑗− 𝐑𝑗𝐱
W𝑖,𝑗
2
𝑗∈𝑆𝑖
𝑁
𝑖=1
Image Reconstruction
s. t. 𝐲 = 𝐔L𝐱
𝐃 ← 𝐃
𝐱 𝑒𝑠𝑡 ← 𝐱
Interpolate the missing
pixels ? S𝑖
Visual Results Interpolation by a factor of 2 (75% missing pixels)
Visual Results Interpolation by a factor of 3 (≈89% missing pixels)
…
𝑐
ℎ
L𝑐
L𝑟
0≠
0≠
0≠
0≠
Results
Peak Signal to Noise Ratio (PSNR) [dB] = 20 log10 255 MSE , Higher is better.
Method Cubic SAI SME PLE NARM Ours Cubic SAI SME PLE NARM Ours
Average PSNR over 18 well-
known images 28.98 29.51 29.62 29.62 29.98 30.09 25.52 25.83 25.95 26.08 26.21 26.44
State-of-the-art Performance
SAI SME
NARM Proposed
Original
PLE
Original SAI SME PLE NARM Proposed
Original SAI SME PLE NARM Proposed
References KSVD: M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries”, IEEE TIP, 2006.
SAI: X. Zhang and X. Wu, “Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation,” IEEE TIP, 2008.
SME: S. Mallat and G. Yu, “Super-resolution with sparse mixing estimators,” IEEE TIP, 2010.
PLE: G. Yu, G. Sapiro, and S. Mallat, “Solving inverse problems with piecewise linear estimators: from Gaussian mixture models to structured sparsity,” IEEE TIP, 2012.
NARM: W. Dong, L. Zhang, R. Lukac, and G. Shi, “Sparse representation based image interpolation with nonlocal autoregressive modeling,” IEEE TIP, 2013.
Grouping
Original
SAI
SME
PLE
NARM
Proposed