From a single joint network of TAU and TAD

• Just recursively apply the scaling network g and f

Heewon Kim, Myungsub Choi, Bee Lim, and Kyoung Mu Lee

Department of ECE, ASRI, Seoul National University, Seoul, Korea

Task-Aware Image Downscaling

https://cv.snu.ac.kr

ComputerVisionLabSeoul National University

Problem Statement

INTRO & MOTIVATION OUR APPROACH

SR-Aware Downscaling

Normal Downscaling• Resizing an image to a smaller scale while preserving the visual appearance.• Simple downscaling operations are widely used in real world applications.• However, the inverse problems of downscaling are highly ill-posed.

Super-Resolution• Previous works on super-resolution are mostly trained with pairs (original HR image - bicubic

downscaled LR image)• Images upscaled with GAN look realistic, but can be different from the original image.• Informative low-resolution images are much easier to restore.

Image Colorization• Upscaling in the channel dimension can be applied to colorizing gray-scale images.• Adding important color information to the gray-scale image can greatly alleviate color-

ambiguity.

Loss Function

𝜃𝑓∗, 𝜃𝑔

∗ = 𝑎𝑟𝑔𝑚𝑖𝑛𝜃𝑓,𝜃𝑔1

𝑁

𝑛=1

𝑁

𝐿𝑡𝑎𝑠𝑘 𝑓𝜃𝑓 𝑔𝜃𝑔 𝐼𝑛𝐻𝑅 , 𝐼𝑛

𝐻𝑅

𝐿𝑡𝑎𝑠𝑘 = 𝐿𝑆𝑅 𝑓 𝐼𝑇𝐴𝐷 , 𝐼𝐻𝑅 + 𝝀𝐿𝑔𝑢𝑖𝑑𝑒 𝐼𝑇𝐴𝐷 , 𝐼𝑔𝑢𝑖𝑑𝑒

(We use L1 loss for both 𝐿𝑆𝑅 and 𝐿𝑔𝑢𝑖𝑑𝑒.)

Architecture & Training

Convolutional encoder-decoder model trained with a guidance image

w.r.t. the task of interest.

When we want to downscale an image:

• Just apply the downscaling network g: 𝐼𝑇𝐴𝐷 = 𝑔𝜃𝑔∗ 𝐼𝐻𝑅

When we want to upscale it later in time:

• Just apply the upscaling network f: 𝐼𝑇𝐴𝑈 = 𝑓𝜃𝑓∗ 𝐼𝑇𝐴𝐷

Upscaling a 240p TAD image to 1080p takes only 0.14s !! (on Titan Xp)

Inference

EXPERIMENTS

Colorization-Aware Downscaling

TAD preserves important information by adding task-aware knowledge at the downscaling stage.

It can be used for efficient restoration of image.

Image Upscaling (Tasks of Interest)

Downscaling vs. Compression

Deep Image Compression• Towards Image Understanding from Deep Compression without Decoding [1]

• Output of [1] : encoded bitstream only its paired decoder can convert it to an image.• Output of TAD (ours) : downscaled image can be used as a thumbnail image as-is.

Task-Aware Downscaling(TAD)• Our new downscaling approach that preserves important information w.r.t. the task of interest.• Makes the inverse problem less ill-posed and easier.

Analysis of TAD

Comparison with the SotA Super-Resolution

Extreme Super-Resolution

Only 15x12 pixels !!

More Results

Comparison to standard image compression methods (compressed output file size & reconstructed performance)

Hyper-parameter 𝜆• Control image quality trade off between HR and LR