partial ordering and stochastic resonance for discrete ... iasi 2012.pdfpartial ordering and...

Partial Ordering and Stochastic

Resonance for Discrete Memoryless Channels

Dr. Paul Cotae

University of the District of Columbia, Washington DC pcotae@udc.edu

1 Technical University of Iasi 05/24/2012

Outline Introduction, Motivation and State of the Art

(2,3) DMC Mutual Information and Capacity

(2,3) DMC and particular cases

Partial Ordering of BSC/SE Stochastic Resonance in BSC/SE

Stochastic Resonance and Capacity Bounds

Conclusions

2 Technical University of Iasi 05/24/2012

Introduction Information theory associated with communication channels Relations between the general (2,3)DMC, the BAC/SE, the BSC/SE,

the BSC, and the BEC

Partial ordering for the BSC/SE under specific noise element constraints

Stochastic Resonance effect on the BSC/SE

Optimal noise power level required to obtain a maximum capacity channel for a given threshold decision

3

Technical University of Iasi 05/24/2012

Technical University of Iasi 05/24/2012 4

Underwater Sensor Network Architecture

Motivation Which DMC, for a given set of resources and noise constraints, is “better”

for communication?

How to use noise to increase the capacity of a communication channel?

This comparison and increase in channel capacity are necessary for wireless sensor communication links at the physical layer operating in very noisy environment and very limited computational power.

Stochastic Resonance observed in many non-linear systems.

Consider Helgert and Pinsker limits capacity bounds

5 Technical University of Iasi 05/24/2012

Communication Channel

Technical University of Iasi 05/24/2012 6

Continuous Channel Discrete Channel

Entropy, Mutual Information and Capacity

7 Technical University of Iasi 05/24/2012

H(X/Y) H(Y/X) I(X,Y)

Partial Ordering

Technical University of Iasi 05/24/2012 8

N1

N2

N3

N4

S

C(N1,S)

C(N2,S)

C(N3,S)

C(N4,S)

Stochastic Resonance(SR) SR phenomenon is a non-linear effect wherein a communication system can transmit the information with improved efficiency in the presence of additive noise.

9 Technical University of Iasi 05/24/2012

N1

N2

N3

N4

S

C(N1,S)

C(N2,S)

C(N3,S)

C(N4,S)

State of the Art

Technical University of Iasi 05/24/2012 10

.

( ) ( )

2

( ) ( ) ( ) ( )

2

( ) ( ) log 1 2

( , )

log 2 2

h a h ba b

ah b bh a bh a ah ba b a b

ah b bh a a ba b

C a b

a b

−−

− −− −

− + + ≠ −

=

+ ≠

1( , )

1a a a a

M a bb b b b

− = = −

2 2( ) log (1 ) log (1 )h x x x x x= − − − −

Technical University of Iasi 05/24/2012 11

Capacity of the (2,2) DMC

Partial Ordering of (2,2) DMCs

Technical University of Iasi 05/24/2012 12

Technical University of Iasi 05/24/2012 13

SR in (2,2) DMC

(2,3) DMC

14 Technical University of Iasi 05/24/2012

(2,3) DMC Mutual Information and Capacity

15 Technical University of Iasi 05/24/2012

H(X/Y) H(Y/X) I(X,Y)

Binary Asymmetric Channel with Symmetric Erasure(BAC/SE) Capacity

and

16 Technical University of Iasi 05/24/2012

using the Kuhn-Tucker condition

Binary Symmetric Channel with Symmetric Erasure (BSC/SE)Capacity

Technical University of Iasi 05/24/2012 17

BSC and BEC

Technical University of Iasi 05/24/2012 18

Technical University of Iasi 05/24/2012 19

Partial Ordering of BSC/SE

Technical University of Iasi 05/24/2012 20

Partial Ordering of BSC/SE (cont.)

Technical University of Iasi 05/24/2012 21

Partial Ordering of BSC/SE: Result-I

Technical University of Iasi 05/24/2012 22

23

Majorization

1 2( , ,..., )na a a=a 1 2( , ,..., )nb b b=b

a b

1 1

k ki i

i ia b↓ ↓

= =≤∑ ∑ 1,2,..., 1k n= −

1 1

n ni i

i ia b↓ ↓

= ==∑ ∑

Technical University of Iasi 05/24/2012

Partial Ordering of BSC/SE (cont.)

Technical University of Iasi 05/24/2012 24

a+b constant, and a1<a2, then (a1,b1) is majorized by (a2,b2)

Technical University of Iasi 05/24/2012 25

Schur Convexity: Result-II-a

( ) ( )ϕ ϕ⇒ ≤a b a b

Theorem 4.2.3: If a+b is constant, then the BSC/SE is Schur convex

Schur Convexity: Result-II-b

Technical University of Iasi 05/24/2012 26

Theorem 4.2.4: for a - b constant, if a1>a2, then C(a1,b1)<C(a2,b2)

Technical University of Iasi 05/24/2012 27

Partial Ordering of BSC/SE: Summary

SR in BSC/SE

28 Technical University of Iasi 05/24/2012

Review of Gaussian, Laplace, Cauchy noise

Technical University of Iasi 05/24/2012 29

SR in BSC/SE

30 Technical University of Iasi 05/24/2012

Technical University of Iasi 05/24/2012 31

SR in BSC/SE: Results

SR in BSC/SE-Gaussian Noise

32 Technical University of Iasi 05/24/2012

SR in BSC/SE- Gaussian Noise

33 Technical University of Iasi 05/24/2012

Forbidden Interval

Technical University of Iasi 05/24/2012 34

SR and Channel Capacity Bounds Pinsker Lower (L) and Helgert Upper (U)bounds

For n=3

For the (2,n) DMC

35 Technical University of Iasi 05/24/2012

SR and Pinsker Bound

36 Technical University of Iasi 05/24/2012

SR and Helgert Bound

37 Technical University of Iasi 05/24/2012

SR and Channel Capacity Bounds

38 Technical University of Iasi 05/24/2012

Optimal Values for Threshold and Noise Level

critical points of the capacity bounds

39 Technical University of Iasi 05/24/2012

Optimal Values for Threshold and Noise Level

40 Technical University of Iasi 05/24/2012

Conclusion We analyzed the general (2,3) DMC and established the relations between the BAC/SE, BSC/SE,

BSC and the BEC.

We provided a partial ordering for the BSC/SE under specific conditional probability between channel input and channel outputs constraints using directional derivative and majorization theory.

We studied the threshold based stochastic resonance behavior of binary-input ternary-output DMC in presence of Gaussian, Laplace and Cauchy noise.

We demonstrated that in presence of the noise with even probability distribution function the general (2,3) DMC become the BSC/SE.

We used the lower and upper capacity bounds to illustrated the stochastic resonance. We derived analytically the optimum noise power level depending on the threshold level that will

maximize their capacity.

41 Technical University of Iasi 05/24/2012

References C.E Shannon and W.W Weaver, “The Mathematical Theory of Communication,” University of Illinois press, Urbana, IL, 1949. Thomas M. Cover and Joy A. Thomas,” Elements of Information Theory,” Second Edition, John Wiley and Sons, 2006. Paul Cotae “Blind Doppler Estimation and Compensation in the Underwater Communications” Technical report submitted to the Naval

Research Laboratories, Washington DC, Sept 2009. François Chapeau-Blondeau,” Noise Enhanced Capacity via Stochastic Resonance in an Asymmetric Binary Channel,” Physical Review

E, 55(2):2016-2019, 1997. François Chapeau-Blondeau, ”Periodic and Aperiodic Stochastic Resonance with Output Signal-to-noise Ratio Exceeding that at the

Input,” International Journal of Bifurcation and Chaos, vol.9, No1:267-272, 1999. K. Chatzikokolaskis and K. Martin ,”A monotonicity principle for information theory”,Electronic Notes in Theoretical computer science

218(2008)111-129. Paul Cotae and Ira S. Moskowitz,” On the Partial Ordering of the Discrete Memoryless Channels Arising in Cooperative Sensor

Networks,” 4th IFIP International Conference on New Technologies, Mobility and Security, NTMS 2011, pp.1-6, Paris, France, February 2011.

H.J. Helgert,”A Partial Ordering of Discrete Memoryless Channels”, IEEE Transactions on Information Theory , vol. IT-3,no.3, pp.360-365,July 1967.

Ira S. Moskowitz, Paul Cotae, Pedro N. Safier, and Daniel L. Kang,” Capacity Bounds and Stochastic Resonance for Binary Input Binary Output Channels,” Proc. of the IEEE Computing, Communications & Applications conference, ComComAP 2012, Hong Kong,pp.61-66,Jan. 2012.

François Chapeau-Blondeau, ”Stochastic Resonance and the Benefit of Noise in Nonlinear Systems,” Noise, Oscillators and Algebraic Randomness – From Noise in Communication Systems to Number Theory. pp. 137-155; M. Planat, ed., Lecture Notes in Physics, Vol. 550, Springer (Berlin) 2000.

Paul Cotae, Ira S. Moskowitz and Myong H. Kang, ”Eigenvalue Characterization of the Capacity of Discrete Memoryless Channels with Invertible Channel Matrices, “Proceedings IEEE 44th Annual Conference on Information Sciences and Systems-CISS 2010, Princeton University,pp.1-6, March 2010.

Ira S. Moskowitz,” Approximations for the Capacity of Binary Input Discrete Memoryless Channels,” In Proceedings 44th Annual Conf. on Information Science and Systems, CISS 2010,Princeton ,NJ,USA, March 2010.

42 Technical University of Iasi 05/24/2012

References(cont.) Ira S. Moskowitz,” An Approximation of the Capacity of a Simple Channel,” 43rd Annual Conf. on Information Science and Systems, CISS

2009,Baltimore ,MD,USA, June 2009. Paul Cotae and T.C. Yang “A cyclostationary blind Doppler estimation method for underwater acoustic communications using direct-

sequence spread spectrum signals,” 8th International conf. on Communications, COMM 2010, Bucharest, July 2010. Ira S. Moskowitz and Paul Cotae,” Channel Capacity Behavior for Simple Models of Optical Fiber Communication,” 8th International conf.

on Communications, COMM 2010, Bucharest, pp.1-6, July 2010. Roland Kamdem, Paul Cotae and Ira S. Moskowitz, ”Threshold Based Stochastic Resonance for the Binary-Input Ternary-Output Discrete

Memoryless Channel,” accepted at 7th IASTED International Conf. on Communication, Internet and Information Technology, CIIT 2012,Baltimore, May 2012.

François Chapeau-Blondeau,” Stochastic Resonance and the Benefit of Noise in Nonlinear Systems,” Noise, Oscillators and Algebraic Randomness – From Noise in Communication Systems to Number Theory. pp. 137-155; M. Planat, ed., Lecture Notes in Physics, Vol. 550, Springer (Berlin) 2000.

Sanya Mitaim, Bart Kosko,”Adaptive Stochastic Resonance in Noisy Neurons Based on Mutual Information ,” IEEE Trans on Neural Networks Vol.15,No.6, Nov 2004.

Mark Damian McDonnell, “Theoretical Aspects of Stochastic Signal Quantization and Suprathreshold Stochastic Resonance,” PhD thesis, The University of Adelaide, Australia, Feb.2006.

Bart Kosko, Sanya Mitiam, Ashok Patel and Mark M. Wilde, ”Applications of Forbidden Interval Theorem in Stochastic Resonance, ” In Applications of Nonlinear Dynamics ,Understanding Complex Systems, pages 71-89.Springer-Verlag, 2009.

Ira S. Moskowitz, Paul Cotae, Pedro N. Safier,”Algebraic Information Theory and Stochastic Resonance for Binary-Input Binary-Output Channels ,” Proceedings IEEE 46th Annual Conference on Information Sciences and Systems-CISS 2012, Princeton University, March 17-19, pp.1-6, March 2012.

Paul Cotae, et. all ”Design and Application of a Distributed and Scalable Underwater Wireless Sensor Acoustic Networks” 2009 Annual ASEE Global Colloquium on Engineering Education, Budapest, Hungary, Oct. 12-15, 2009.

R. A. Silverman,”On Binary Channels and their Cascades,” IRE Trans. Information Theory, Vol.1,no.,pp.19-27,Dec 1995. C.E. Shannon, ”A note on a partial ordering for communication channels, ”Information and control,Vol.1, pp.390-397,1958. H.J. Helgert, “A Partial Ordering of Discrete Memoryless Channel,” IEEE Transactions on Information Theory, vol.1, no.3, pp.360-365,

July 1967. A.W. Marshall and I. Olkin, Inequalities: Theory of Majorization and its applications .New York: Academic, 1979. Patrick M. Fitzpatrick “Advanced Calculus,” American Mathematical Society, 2009.

Technical University of Iasi 05/24/2012 43