image and video upscaling from local self examples
DESCRIPTION
Image and Video Upscaling from Local Self Examples. Gilad Freedman Raanan Fattal Hebrew University of Jerusalem. Background and o verview Algorithm description L ocal self similarity Non-dyadic filter bank Filter design Results. Single i mage upscaling. 1. l arge 2. realistic - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/1.jpg)
Image and Video Upscaling from Local Self Examples
Gilad FreedmanRaanan Fattal
Hebrew University of Jerusalem
![Page 2: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/2.jpg)
Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
![Page 3: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/3.jpg)
Single image upscaling
1. large2. realistic3. faithful4. fast
![Page 4: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/4.jpg)
Previous workparametric image model example based
Freeman et al. 2002
generic looking edges
Sun et al. 2008
Shan et al 2008
Fattal 2007
Glasner et al. 2009
noisyresult
![Page 5: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/5.jpg)
New approach: locale example based
corner step edge
line step edge
non smooth shading
local self similarity small upscaling ratios
1/2
4/5
new non-dyadic filter bank
local self similarity
local search speed
locality too few examples
Increase exemplar quality and size
maintain search locality
novel components:
1. local self similarity2. non-dyadic filter bank
![Page 6: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/6.jpg)
Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
![Page 7: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/7.jpg)
Local self-examples upscaling
low pass original image high pass
interpolated image
frequency content
![Page 8: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/8.jpg)
low pass high pass
interpolated image
For each patch:
Search a local area for best example
Take corresponding
patch
Add to interpolated
image
frequency content
Local self-examples upscaling
![Page 9: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/9.jpg)
Local self-examples upscaling
low pass high pass
interpolated image
Repeat for all patches, to fill the high frequenciesfrequency content
![Page 10: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/10.jpg)
OverviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
![Page 11: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/11.jpg)
Local self similarity
cropped downscaled
![Page 12: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/12.jpg)
Local self similarity
Patches in original image can matched locally with ones in downscaled version
![Page 13: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/13.jpg)
Local examples are enough
full imageimage database
query db image local
4.0 2.9 3.55
1.6 1.05 1.05
2.7 2.05 2.05
3.3 2.96 3.06
6.5 5.61 5.61
best matches
![Page 14: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/14.jpg)
Visual assessment – external, exact NN, local
Large externalexample database
Searching theentire image
Searching localregions in image
external database
global search
local search
![Page 15: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/15.jpg)
Comparison of example search methods
![Page 16: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/16.jpg)
Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
![Page 17: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/17.jpg)
Need for non-dyadic scalingslarge ratios mixed ratiossmall ratios
![Page 18: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/18.jpg)
Dyadic filters
1:2
full frequency
content
higher half
lower half
dyadic filter bank
![Page 19: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/19.jpg)
Non-dyadic filter bank
1:2
4:5
small scaling ratios
better examples
full frequency
content
higher part
lower part
non-dyadic filter bank
![Page 20: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/20.jpg)
Non-dyadic filters: downscaling
1. convolve with 2 filters2. subsample each by 3
dyadic case:
example for the 2:3 ratio:
1. convolve with one filter2. subsample by 2
![Page 21: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/21.jpg)
Non-dyadic filters: upscaling
1. zero upsample by 22. convolve with 2 filters3. sum
dyadic case:
example for the 2:3 ratio:
1. zero upsample by 12. convolve with 1 filter
![Page 22: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/22.jpg)
Use of the filters in upscalingUpscaling using inverse scaling
filters
Smoothing by downscaling
and upscaling
![Page 23: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/23.jpg)
Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
![Page 24: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/24.jpg)
When interpolating, smooth areas come from inputUniformly spaced grids should remain uniform
1. Uniform stretch
0
255
brig
htne
ss
grid coordinates
![Page 25: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/25.jpg)
2. Consistency
upsample downsample
The interpolated image, if downscaled should be equal to the input. Formally,
Previous methods achieve consistency by solving large linear systems to achieve this property
![Page 26: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/26.jpg)
3. PSF modeling
Large image - small camera point spread
function
Small image - large camera point spread
function
Difference between point spread functions
![Page 27: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/27.jpg)
4. Low frequency spanfr
eque
ncy
When upsampling don’t add new frequencies
Upsampling filter should be low-pass
original interpolated
![Page 28: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/28.jpg)
5. Singularities preservation
≈blurred Image interpolated image
similar amount of blur
![Page 29: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/29.jpg)
Real time video upsampling on GPUmain GPU memory
GPU cores
NTSC to full HD @ 24 fps
Search and filter-banksare both local operations
![Page 30: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/30.jpg)
Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
![Page 31: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/31.jpg)
Bicubic x3(zoomed in)
Ours X3(zoomed in)
![Page 32: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/32.jpg)
Bicubic x3 Ours X3
![Page 33: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/33.jpg)
Genuine Fractals™ x4 Ours X4
![Page 34: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/34.jpg)
Glasner et al. 2009 x4 Ours X4
![Page 35: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/35.jpg)
![Page 36: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/36.jpg)
![Page 37: Image and Video Upscaling from Local Self Examples](https://reader033.vdocument.in/reader033/viewer/2022050821/5681677c550346895ddc7f2e/html5/thumbnails/37.jpg)
Thank you!
Paper & additional results can be found at: www.cs.huji.ac.il/~giladfreedmn