3/23/04 1 information rates for two-dimensional isi channels jiangxin chen and paul h. siegel center...

3/23/04 1

Information Rates for Two-Dimensional ISI Channels

Jiangxin Chen and Paul H. Siegel

Center for Magnetic Recording Research

University of California, San Diego

DIMACS Workshop

March 22-24, 2004

3/23/04 2DIMACS Workshop

Outline

• Motivation: Two-dimensional recording

• Channel model

• Information rates

• Bounds on the Symmetric Information Rate (SIR)• Upper Bound

• Lower Bound

• Convergence

• Alternative upper bound

• Numerical results

• Conclusions


Two-Dimensional Channel Model

• Constrained input array

• Linear intersymbol interference

• Additive, i.i.d. Gaussian noise

1

0

1

0

21

],[],[],[],[n

l

n

k

jinljkixlkhjiy

]j,i[x

],[ jih

)( 2,0~],[ Njin


Two-Dimensional Processes

• Input process:

• Output process:

• Array

upper left corner:

lower right corner:

]j,i[XX

]j,i[YY

11 nj,mij,iY

]j,i[Y

]nj,mi[Y 11


Entropy Rates

• Output entropy rate:

• Noise entropy rate:

• Conditional entropy rate:

n,m,n,m YH

mnlimYH 11

1

02

1eNlogNH

NHX|YHmn

limX|YH n,m,

n,m,

n,m

1111

1


Mutual Information Rates

• Mutual information rate:

• Capacity:

• Symmetric information rate (SIR): Inputs are constrained to be independent, identically distributed, and equiprobable binary.

NHYHX|YHYHY;XI

],[ jixX

Y;XImaxC

XP


Capacity and SIR

• The capacity and SIR are useful measures of the achievable storage densities on the two-dimensional channel.

• They serve as performance benchmarks for channel coding and detection methods.

• So, it would be nice to be able to compute them.


Finding the Output Entropy Rate

• For one-dimensional ISI channel model:

and

where

n

nYH

nYH 1

1lim

nnn yYpEYH 111 log

nYYYY n ,2,11


Sample Entropy Rate

• If we simulate the channel N times, using inputs with specified (Markovian) statistics and generating output realizations

then

converges to with probability 1 as .N nYH 1

Nknyyyy kkkk ,,2,1,][,,]2[,]1[ )()()()(

N

k

kypN 1

)(log1


Computing Sample Entropy Rate

• The forward recursion of the sum-product (BCJR)algorithm can be used to calculate the probability of a sample realization of the channel output.

• In fact, we can write

where the quantity is precisely the normalization constant in the (normalized) forwardrecursion.

n

i

ii

n y|yplogn

yplogn 1

111

11

11i

i y|yp

nyP 1


Computing Entropy Rates

• Shannon-McMillan-Breimann theorem implies

as , where is a single long

sample realization of the channel output

process.

ny1

YHyplogn .s.a

n 11

n


SIR for Partial-Response Channels


Capacity Bounds for Dicode


Markovian Sufficiency

Remark: It can be shown that optimized Markovian processes whose states are determined by their previous r symbols can asymptotically achieve the capacity of finite-state intersymbol interference channels with AWGN as the order r of the input process approaches .

(J. Chen and P.H. Siegel, ISIT 2004)


Capacity and SIR in Two Dimensions

• In two dimensions, we could estimate by

calculating the sample entropy rate of a very large

simulated output array.

• However, there is no counterpart of the BCJR

algorithm in two dimensions to simplify the

calculation.

• Instead, we use conditional entropies to derive upper

and lower bounds on .

YH

YH


Array Ordering

• Permuted lexicographic ordering:

• Choose vector , a permutation of .

• Map each array index to .

• Then precedes if

or and .

• Therefore, row-by-row ordering column-by-column ordering

21 k,kk 21,

21 t,t kk t,t21

21 t,t 21 s,s

kk ts11

kk ts11

kk ts22

:2,1k :1,2k


Two-Dimensional “Past”

• Let be a non-negative

vector.

• Define to be the elements

preceding inside the region

(with permutation k )

4321 l,l,l,ll

j,il,k YPast

j,iY

42

31

lj,lilj,liY


Examples of Past{Y[i,j]}


Conditional Entropies

• For a stationary two-dimensional random field Y on the integer lattice, the entropy rate satisfies:

(The proof uses the entropy chain rule. See [5-6]) • This extends to random fields on the hexagonal

lattice,via the natural mapping to the integer lattice.

j,i,kj,i YPastYHYH |


Upper Bound on H(Y)

• For a stationary two-dimensional random field Y,

where

1Ul,k

kHminYH

j,il,kj,i YPastYHYH |Ul,k 1


Two-Dimensional Boundary of Past{Y[i,j]}

• Define to be the boundary

of .

• The exact expression for

is messy, but the geometrical concept is

simple.

j,il,k YPast

j,il,kStrip Y

j,il,kStrip Y


Two-Dimensional Boundary of Past{Y[i,j]}


Lower Bound on H(Y)

• For a stationary two-dimensional hidden Markov field Y,

where

and is the “state information” for

the strip .

1Ll,k

kHmaxYH

j,iYl,kStX,j,il,kj,i YPastYHYH |Ll,k 1

j,iYl,kStX

j,iYl,kStrip


Sketch of Proof

• Upper bound:

Note that

and that conditioning reduces entropy.

• Lower bound:

Markov property of , given “state

information” .

j,i,kj,il,k YPastYPast

j,iY

j,iYl,kStX


Convergence Properties

• The upper bound on the entropy rate is

monotonically non-increasing as the size of the

array defined by increases.

• The lower bound on the entropy rate is

monotonically non-decreasing as the size of the

array defined by increases.

1Ul,kH

4321 l,l,l,ll

1Ll,kH

4321 l,l,l,ll


Convergence Rate

• The upper bound and lower bound

converge to the true entropy rate at least

as fast as O(1/lmin) , where

1Ul,kH

k orderingcolumn -by-columnfor ,, min

k ordering row-by-rowfor , ,, min

321

431min lll

llll

1Ll,kH

YH


Computing the SIR Bounds

• Estimate the two-dimensional conditional entropies

over a small array.

• Calculate to get

for many realizations of output array.

• For column-by-column ordering, treat each row

as a variable and calculate the joint

probability row-by-row

using the BCJR forward recursion.

BAP

BAH

BPBAP ,,

iY

mYYYP ,,, 21


2x2 Impulse Response

• “Worst-case” scenario - large ISI:

• Conditional entropies computed from 100,000 realizations.

• Upper bound:

• Lower bound:

(corresponds to element in middle of last column)

5.05.0

5.05.0],[1 jih

1 ,log

2

1min 0

10,3,7,7,1,2 eNH U

01

0,3,7,7,1,2 log2

1eNH L


Two-Dimensional “State”


SIR Bounds for 2x2 Channel


Computing the SIR Bounds

• The number of states for each variable increases exponentially with the number of columns in the

array.

• This requires that the two-dimensional impulse response have a small support region.

• It is desirable to find other approaches to computing bounds that reduce the complexity, perhaps at the cost of weakening the resulting bounds.


Alternative Upper Bound

• Modified BCJR approach limited to small impulse response support region.

• Introduce “auxiliary ISI channel” and bound

where

and is an arbitrary conditional

probability distribution.

2Ul,kHYH

ydjilk

jiyqjilk

jiypH yPastyPastUlk

,

,,log,

,,, |2

,

ji

lkjiyq yPast ,

,, |


Choosing the Auxiliary Channel

• Assume is conditional probability distribution of the output from an auxiliary ISI channel

• A one-dimensional auxiliary channel permits a calculation based upon a larger number of columns in the output array.

• Conversion of the two-dimensional array into a one-dimensional sequence should “preserve” the statistical properties of the array.

• Pseudo-Peano-Hilbert space-filling curves can be used on a rectangular array to convert it to a sequence.

ji

lkjiyq yPast ,

,, |


Pseudo-Peano-Hilbert Curve

j,iYPastj,iY l,,,,, 478712


SIR Bounds for 2x2 Channel

Alternative upper bounds --------->


3x3 Impulse Response

• Two-DOS transfer function

• Auxiliary one-dimensional ISI channel with memory

length 4.

• Useful upper bound up to Eb/N0 = 3 dB.

011

121

110

101],[2 jih


SIR Upper Bound for 3x3 Channel


Concluding Remarks

• Upper and lower bounds on the SIR of two-dimensional finite-state ISI channels were presented.

• Monte Carlo methods were used to compute the bounds for channels with small impulse response support region.

• Bounds can be extended to multi-dimensional ISI channels

• Further work is required to develop computable, tighter bounds for general multi-dimensional ISI channels.


References

1. D. Arnold and H.-A. Loeliger, “On the information rate of binary-input channels with memory,” IEEE International Conference on Communications, Helsinki, Finland, June 2001, vol. 9, pp.2692-2695.

2. H.D. Pfister, J.B. Soriaga, and P.H. Siegel, “On the achievable information rate of finite state ISI channels,” Proc. Globecom 2001, San Antonio, TX, November2001, vol. 5, pp. 2992-2996.

3. V. Sharma and S.K. Singh, “Entropy and channel capacity in the regenerative setup with applications to Markov channels,” Proc. IEEE International Symposium on Information Theory, Washington, DC, June 2001, p. 283.

4. A. Kavcic, “On the capacity of Markov sources over noisy channels,” Proc. Globecom 2001, San Antonio, TX, November2001, vol. 5, pp. 2997-3001.

5. D. Arnold, H.-A. Loeliger, and P.O. Vontobel, “Computation of information rates from finite-state source/channel models,” Proc.40th Annual Allerton Conf. Commun., Control, and Computing, Monticello, IL, October 2002, pp. 457-466.


References

6. Y. Katznelson and B. Weiss, “Commuting measure-preserving transformations,” Israel J. Math., vol. 12, pp. 161-173, 1972.

7. D. Anastassiou and D.J. Sakrison, “Some results regarding the entropy rates of random fields,” IEEE Trans. Inform. Theory, vol. 28, vol. 2, pp. 340-343, March 1982.

3/23/04 1 information rates for two-dimensional isi channels jiangxin chen and paul h. siegel center...

Documents

san diego dimacs workshop

dicode slide

noise entropy rate

conditional entropy

computing sample entropy

channel output process

gaussian noise slide

channel coding