Extremal Problems of Information Combining

Alexei Ashikhmin

Information Combining: formulation of the problem Mutual Information Function for the Single Parity Check Codes More Extremal Problems of Information Combining Solutions (with the help of Tchebysheff Systems) for the Single Parity Check Codes

Joint work with Yibo Jiang, Ralf Koetter, Andrew Singer

Encoder

Channel

Channel

Channel

APPDecoder

Information Transmission

Density function of the channel is not known

We only know

Optimization Problem

We assume that

and that the channel is symmetric

Problem 1

Among all probability distributions such that

determine the probability distribution that maximizes (minimizes)

the mutual information at the output of the optimal decoder

Var

iabl

e no

des

proc

essi

ng

Che

ck n

odes

pro

cess

ing

Interleaver

Inpu

t fro

m c

hann

el

From variable nodes

To variable nodes Decoder of single parity check code

Problem is Solved Already

1. I.Land, P. Hoeher, S.Huettinger, J. Huber, 2003

2. I.Sutskover, S. Shamai, J. Ziv, 2003

erasure

Repetition code: The Binary Erasure Channel (BEC) is the best

The Binary Symmetric Channel (BSC) is the worst

Single Parity Check Code:

is Dual of Repetition Code The Binary Erasure Channel (BEC) is the worst

The Binary Symmetric Channel (BSC) is the best

Our Goals

We would like to solve the optimization problem for the Single Parity Check Codes directly (without using duality)

Get some improvements

Soft Bits

We call soft bit, it has support on

Channel

erasure

Binary symmetric channel,

Gaussian Channel:

Channel

Channel

Channel

Decoder Single ParityCheck Code

Encoder Single ParityCheck Code

E.Sharon, A. Ashikhmin, S. Litsyn Results:

Properties of the moments

Lemma 1. is nonnegative and nonincreasing

2. The ratio sequence is nonincreasing

Lemma

In the Binary Erasure Channel all moments are the same

Problem 2

Among all T-consistent probability distributions on [0,1]

such that

determine the probability distribution that maximizes

(minimizes) the second moment

Solution of Problem 2

Theorem

Among all binary-input symmetric-output channel

distributions with a fixed mutual information

Binary Symmetric Channel maximizes

and

Binary Erasure Channel minimizes

the second moment

Proof: We use the theory of Tchebysheff Systems

Lemma

Binary Symmetric, Binary Erasure and an arbitrary channel

with the same mutual information have the following layout of

Lemma

Let and

1)

2) if for and for

then

This is exactly our case

satisfy conditions of the previous lemma

Problem 1 on extremum of mutual information

and

Problem 2 on extremum of the second moment

are equivalent

Extrema of MMSE

It is known that the channel soft bit is the MMSE estimator fo

the channel input

Theorem Among all binary-input symmetric-output channels withfixed the Binary Symmetric Channel has the minimum MMSE: and the Binary Erasure Channel has the maximum MMSE:

Channel

How good the bounds are

Problem 3

1)

2)

Among all T-consistent channels find that maximizes

(minimizes)

Channel

Channel

Channel

Decoder Single ParityCheck Code

Encoder Single ParityCheck Code

Problem 4

Among all T-consistent probability distributions on [0,1]

such that

1)

2)

determine the probability distribution that maximizes

(minimizes) the fourth moment

Theorem The distribution with mass at , mass at

and mass at 0 maximizes

The distribution with mass at , mass at

and mass at 1 minimizes

Extremum densities

Maximizing

Minimizing:

Lemma

Channel with minimum and maximum and an arbitrary

channel with the same mutual information have the followin

layout of

Problem 3 on extremum of mutual information

and

Problem 4 on extremum of the fourth moment

are equivalent

Assume that

and is the same as in AWGN channel with this

Tchebysheff Systems

Definition

A set of real continues functions is called

Tchebysheff system (T-system) if for any real the linear

combination has at most distinct roots at

Definition

A distribution is a nondecreasing, right-continues function

The moment space, defined by

( is the set of valid distributions), is a closed convex cone.

For define

Problem For a given find

that maximizes (minimizes)

Theorem

If and are T-systems,

and then the extrema are attained uniquely with

distrtibutions and with finitely many mass points

Lower principalrepresentation

Upper principalrepresentation

Soft Bits

We call soft bit, it has support on

Lemma (Sharon, Ashikhmin, Litsyn)

If then

Channel

Random variables with this property are called T-consistent

Find extrema of

Under constrains

Theorem

Systems and are T-systems on [0,1].

---------------------------------------------------------------------------------

the distribution that maximizes

has only one mass point at :

has probability mass at

and at

This is exactly the Binary Symmetric Channel