Information Theory in an Industrial Research Lab

Marcelo J. Weinberger

Information Theory Research GroupHewlett-Packard Laboratories – Advanced Studies

Palo Alto, California, USA

with contributions from the ITR group

Purdue University – November 19, 2007

Information Theory research in the industry

MissionResearch the mathematical foundations and practical applications of information theory, generating intellectual property and technology for “XXX Company” through the advancement of scientific knowledge in these areas

Apply the theory and work on the applications makes obvious sense for “XXX Company” research labs;But why invest on advancing the theory?

some simple answers which apply to any basic research area: long-term investment, prestige, visibility, give back to society...

this talk will be about a different type of answer:differentiating technology vs. enabling technology

Main claim: working on the theory helps developing analytical tools that are needed to envision innovative, technology-differentiating ideas

Case studies

JPEG-LS:

from universal context modeling to a lossless image compression standard

DUDE (Discrete Universal DEnoiser):from a formal setting for universaldenoising to actual image denoisingalgorithms

Error-correcting codes in nanotechnology:the advantages of interdisciplinary research

2-D information theory:looking into the futureof storage devices

compressstore,transmit

de-compress

Input Output

010010...010010...

S

001

010

100

111

0 1 S

001

010

100

111

0 1

The goal: upon observing , choose to optimize some fidelity criterion (e.g.: minimize number of symbol errors, squared distance, etc.)

A natural extension of work on prediction/compression

Applications: image and video denoising, text correction, financial data denoising, DNA sequence analysis, wireless communications…

discretesource

discretememoryless

channel (noise)

denoiser

nxxx ,,, 21 nzzz ,,, 21 nxxx ˆ,,ˆ,ˆ 21

nxxx ˆ,,ˆ,ˆ 21 nzzz ,,, 21

Discrete Universal DEnoising (DUDE)

DUDE: how it’s done

pass 1: - gather statistics on symbol occurrences per context pattern

- estimate noiseless symbol distribution given context pattern and noisy sample (posterior distribution)

pass 2: denoise each symbol, based on estimated posterior

who do you believe? what you see, or what the global stats tell you?

precise decision formula proven asymptotically optimalcontext template size must be carefully chosen

zi

“context” samples

sample being denoised

datasequence

noisychannel

nxxx ,,, 21 nzzz ,,, 21

Key component of DUDE: Model the conditional distribution P(Zi|context of Zi) and infer P(Xi| Zi and context of Zi) from it

Main issue: large alphabet large number of model parameters high learning cost

Leveraged “semi-universal’’ approach from image compression: rely on prior knowledge. Main tools:

predictioncontexts based on quantized dataparameterized distributions

State-of-the-art for “salt-and-pepper” noise removal Competitive for Gaussian and ``real world” noise removal, but still room for improvement

The main challenge in image denoising

Application 1: Image denoising

Best previous result in the literature: PSNR = 35.6 dB @ error rate=30% (Chan,Ho&Nikolova, IEEE IP Oct’05)

error rate=30% PSNR=10.7 dB(``salt and pepper” noise)

dude-denoised PSNR=38.3 dB

Application 2: Denoiser-enhanced ECC

Suitable for wireless communications Leaves overall system ``as-is’’, but

enhances receiver by denoising signal prior to error correction (ECC) decoding

Allows to design a “better receiver” that will recover signals other receivers would reject as undecodable

transmitted codeword

decodable region forregular ECC (handlescode redundancy,structured)

received noisy codeword

denoising (handles source redundancy,

natural)

non-enhanced(no reception) DUDE-enhanced