information theory in an industrial research lab marcelo j. weinberger information theory research...
Post on 14-Jan-2016
215 Views
Preview:
TRANSCRIPT
Information Theory in an Industrial Research Lab
Marcelo J. Weinberger
Information Theory Research GroupHewlett-Packard Laboratories – Advanced Studies
Palo Alto, California, USA
with contributions from the ITR group
Purdue University – November 19, 2007
Information Theory research in the industry
MissionResearch the mathematical foundations and practical applications of information theory, generating intellectual property and technology for “XXX Company” through the advancement of scientific knowledge in these areas
Apply the theory and work on the applications makes obvious sense for “XXX Company” research labs;But why invest on advancing the theory?
some simple answers which apply to any basic research area: long-term investment, prestige, visibility, give back to society...
this talk will be about a different type of answer:differentiating technology vs. enabling technology
Main claim: working on the theory helps developing analytical tools that are needed to envision innovative, technology-differentiating ideas
Case studies
JPEG-LS:
from universal context modeling to a lossless image compression standard
DUDE (Discrete Universal DEnoiser):from a formal setting for universaldenoising to actual image denoisingalgorithms
Error-correcting codes in nanotechnology:the advantages of interdisciplinary research
2-D information theory:looking into the futureof storage devices
compressstore,transmit
de-compress
Input Output
010010...010010...
S
001
010
100
111
0 1 S
001
010
100
111
0 1
The goal: upon observing , choose to optimize some fidelity criterion (e.g.: minimize number of symbol errors, squared distance, etc.)
A natural extension of work on prediction/compression
Applications: image and video denoising, text correction, financial data denoising, DNA sequence analysis, wireless communications…
discretesource
discretememoryless
channel (noise)
denoiser
nxxx ,,, 21 nzzz ,,, 21 nxxx ˆ,,ˆ,ˆ 21
nxxx ˆ,,ˆ,ˆ 21 nzzz ,,, 21
Discrete Universal DEnoising (DUDE)
DUDE: how it’s done
pass 1: - gather statistics on symbol occurrences per context pattern
- estimate noiseless symbol distribution given context pattern and noisy sample (posterior distribution)
pass 2: denoise each symbol, based on estimated posterior
who do you believe? what you see, or what the global stats tell you?
precise decision formula proven asymptotically optimalcontext template size must be carefully chosen
zi
“context” samples
sample being denoised
datasequence
noisychannel
nxxx ,,, 21 nzzz ,,, 21
Key component of DUDE: Model the conditional distribution P(Zi|context of Zi) and infer P(Xi| Zi and context of Zi) from it
Main issue: large alphabet large number of model parameters high learning cost
Leveraged “semi-universal’’ approach from image compression: rely on prior knowledge. Main tools:
predictioncontexts based on quantized dataparameterized distributions
State-of-the-art for “salt-and-pepper” noise removal Competitive for Gaussian and ``real world” noise removal, but still room for improvement
The main challenge in image denoising
Application 1: Image denoising
Best previous result in the literature: PSNR = 35.6 dB @ error rate=30% (Chan,Ho&Nikolova, IEEE IP Oct’05)
error rate=30% PSNR=10.7 dB(``salt and pepper” noise)
dude-denoised PSNR=38.3 dB
Application 2: Denoiser-enhanced ECC
Suitable for wireless communications Leaves overall system ``as-is’’, but
enhances receiver by denoising signal prior to error correction (ECC) decoding
Allows to design a “better receiver” that will recover signals other receivers would reject as undecodable
transmitted codeword
decodable region forregular ECC (handlescode redundancy,structured)
received noisy codeword
denoising (handles source redundancy,
natural)
non-enhanced(no reception) DUDE-enhanced
top related