on the entropy of continuous probability distributions

8/10/2019 On the Entropy of Continuous Probability Distributions

1/3


2/3

121

TABLE I

TABLE OFDIFFERENTIALENTROPIES

t

7

DISTRIBUTIONS

E"tP.?py

Name

Density Function

(in nats)

f(x) = xp-I(1 - x)q-l/B(P.q) i 0 5 x 5 1

log E3(p,q) - (p-l) IIJI(P)-*(P+q)l

Beta

Where B(p,q) = r(p)r(q)/r(P+q) : P. 4 ' 0

- (q-l) [$ (q)-a(p+q)]

Cauchy

f(X) = (i/n) (h2+x2)-l i -m < x < m, i > 0

log(4nh)

Chi

f(x) ={2/[2"'2 0" r(n/2)]Ix"-l e-x2'(202);x>0 log[or(n/2)/ JZ] _

y $(n/2) + n/2

and n is a positive integer

Chi-square

f(X) = .(42)-l .-x/(202),[ 2n/2 ?r(n/Z)]; x,0 loq[202

r(n/2)] + (l-n/2) yw2) + n/2


Erlang

f(x) = [p/(*-l, ] 2-l e-BX ; x, 6 > 0

(1 - n) $'(n) + loqCr(n,L3] + n


Exponential

f(X) = o-1 e-x/a : x, '3 > 0

.J /2

" z/2

F

f(X) =

"I '

2

,(v,/2)-1

log[(",/V*,B(V,/2.V2/2~l+(l-~~/2)*~~,/2)

B(V,/2.Vz/2)

(" +v x) ("1+"2)'2

;x>o --==-A

and ", , "2

are positi& integer

-(l+"~/2)$(v,/2)+[(v,+v*~/~~[~vlfvz)/2]

Gamma f(x) = xa-l e-X'@/[fi"r (a)] i x, a, B ) 0

log@r(u)] + (1-a) $(a) + a

Laplace

f(x) = (l/2) q-1 .-lx-ei' J ; -m < x < m, @.o

1 + log(2 v )

Logistic

f(x) = e-x (1 + e-X)-2 ; -m < x < -

2

Lognormal

f(x) = [,xsrzll)]

-1 e-uogx-m)2/(202) : x > 0

m + (l/2) log(2neo2)

I

Maxwell-Bolt7.ma

ftx) = [4n-1/283/2]x2 e-flX2

i x. 6 > 0

(l/2) log (n/B) + y - l/2

/

Normal

f(x) = Lomi,,,-l e-x2/(2o2) -m < x < ml a>0

(1/2)loq(2neo2)

I

Gsnerallzed-Noml,f(x) = [2&2/r(a/ 2,] xa-l e-6x2: x,a,

6>0 log~(u/2)/(261' 2 )] - [(a-1)/2] $W2) +a/2

Pare to

f(x) = a ka/xa+' ; x z k > 0, a > 0

loq(k/a) + 1 +1/a

Rayleigh

f(x) = (x/b') e-x2'(2b2' ; x, b > 0

1 + log(B/J2) + y/2

f(x) = (1 + x2/")-("Cl)'2

;--m


3/3

122

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-24, NO. 1, JANUARY 1978

This reduces the problem of calculating h(X) to that of calcu-

lating Ehi (X).

Shannon [ll] was the first to compute the entropy of the uni-

form, the exponential and the normal distributions. Golomb [5]

computed the entropy for the Pareto distribution; the entropy

for the squared Cauchy distribution appeared in Bergers book

[l]. For the Laplace, logistic, lognormal, and Weibull distribu-

tions, the Eh;(X)s can be found in the books of Johnson and

Kotz [6], [7] expressed in terms of elementary functions. We add

to this list the following distributions: Cauchy, Maxwell-

Boltzmann, Rayleigh, Triangular.

For each of the remaining distributions listed in this paper

(beta, chi, &i-square, F, gamma, generalized-normal, student-t),

at least one of the terms Ehi(X) is not expressible in terms of

elementary functions. The main object of this correspondence

is to point out that a single special function, the logarithmic de-

rivative # of the gamma function, may be used to specify h(X)

for all these distributions.

The definitions of the #-function and of Eulers constant y (to

enough precision for numerical calculations) are as follows:

Hz) = $log I%) = -y + (2 -

l)$(k + l)(z + iz)]-1

(4)

y = 0.5772156649 -. - .

(5)

For more details on the #-function, see [lo].

The entropy h(X), given by (3),can be easily obtained by

using

a)

b)

4

4

[2, p. 314, (S)] and [2, p. 316, (22)] for the beta distribu-

tion,

[3, p. 233, (8)], [2, p. 314, (8)], and [2,316, (23)] for the F-

distribution,

[2, p. 315, (9)] and [2, p. 312, (l)] for the gamma distribu-

tion,

[2, p. 312, (l)], [2, p. 315, (9)], and [2, p. 133, (3)] for the

generalized normal distribution.

The results are summarized in Table I.

ACKNOWLEDGMENT

The authors would like to thank Prof. T. Berger and the ref-

erees for their helpful comments and suggestions.

Ill

121

[31

141

[51

[3

[71

is1

PI

IlO1

Iill

REFERENCES

T. Berger, Rate Distortion Theory.

Englewood Cliffs, NJ, Prentice-Hall,

1971.

A. ErdBlyi et al., Tables of Integral Transforms, vol. I. New York:

McGraw-Hill, 1954.

A. Erdblyi et al., Tables of Integral Transforms, vol. II. New York:

McGraw-Hill. 1954.

D. V. Gokhale, Maximum entropy character ization of some distributions,

in

Statistical Distributions

in

Scientific

Work, vol. 3, Patil, Katz, and Ord,

Eds. Boston, MA: Reidel , 1975, pp. 299-304.

S. W. Golomb, The information generating function of a probability distri-

bution,

IEEE Trans. Inform. Theory,

vol. IT-12, pp. 75-77, Jan. 1966.

N.

L. Johnson and

S.

Katz,

Distributions in Statistics: Continuous Uniuariate

Distributions--I.

New York:

Wiley, 1970.

N.

L. Johnson and

S.

Katz,

Distributions in Statistics: Continuous Uniuariate

Distributions- 2. New York: Wiley, 1970.

A. M. Kagan, Ju. V. Linnik, and C. R. Rao,

Characterization Problems in

Mathematical Statistics.

New

York: Wilev. 1973.

J. H. C. Lisman and M. C. A. van Zuylen,

%Jote on the generation of most

probable frequency distributions, Statistica Neerlandica, vol. 26, pp. 19-23,

1972.

W. Magnus, F. Oberhettinger, and R. P. Soni, Formulas

and Theorems for

the Special Functions of Mathematical

Physics.

New York: Springer-Verlag,

1966.

C. E. Shannon, A mathematical theory of comunication (concluded), Bell

Syst. Tech. J.,

vol. 27, pp. 629-631,1948.

Advent of Nonregularity in Photon-Pulse Delay

Estimation

ISRAELBAR-DAVID, MEMBER,IEEE,AND MOSHELEVY

Abstract-The mean-square error of the delay estimate of a

photon pulse is known to decrease as Q- if the pulse envelope is

smooth, but as Qe2 if it has sharp edges, thereby belonging to

nonregular estimation cases (Q denotes the expected photon

count). The transition from the Q-l to the Qm2 law is investigated

for trapezoidal pulse models, and is found to occur in the region

where the standard deviation of the error is on the order of the

width of the pulse slopes. Thus for values of Q below this region,

practical pulses are effectively rectangular, and their estimation

problem may be qualified as nonregular.

This note relates to one facet of the problem of estimation of

photon-pulse delay with direct-detection receivers, which has

been exposed in a previous contribution [l] to which we refer the

reader for appropriate background. It has been shown that,

asymptotically with the expected photon count Q in the pulse,

the mean-square error (mse) of the delay estimate decreases only

as Q-l with pulses that have smooth envelopes, i.e., regular

estimation cases [2], but as Q-z with pulses that have sharp edges.

The latter envelopes pertain to nonregul ar estimation problems

and, apparently, the nonregularity changes the dependence from

the Q-l to the Qe2 law. As many practical pulse envelopes are

more readily associated with a rectangular model than with a

smooth one, the way in which the signal shape affects the tran-

sition between these laws has been suggested for further inves-

tigation [l, item (3) of Section II-C]. Reformulated, the question

is how inclined need a slope be in order that the pulse may qualify

as a rectangular one?

Herein, we analyze the effect by modeling real-life pulse en-

velopes by trapezoids of varying slope as illustrated in Fig. 1. Let

the pulse envelope X(t) be given by

-(l + a)D I t I -D

-DItID

[(l +

a)D - t]X,

D 5 t I

(1

+ a)D

a-D

0,

otherwise.

(1)

Then the expected photon count in the pulse is

Q= I:::&:,

(t) dt = (a + 2)0X,,

the received envelope is X(t - T), and the observables are the

instants tk, k = 1,2, . . - ,

L, of the photoelect ron emissions at the

direct detection receivers output. The minimum m.s.e. estimate

of the delay r for such an envelope is not expressible explicitly

in terms of tk, as it fortunately is for rectangular ones [I]. We

therefore adopt here the estimate which is optimum for the

rectangular envelope (with (Y = 0 in (1)) and is given [ 11, or L I

Manuscript received March 2.1977 ; revised May 6.1977. This correspondence

incorporates results from a Master of Science theses submitted by M. Levy to the

Senate of the Technion-Israel Institute of Technology, Haifa, Israel, in October

1976.

The authors are with the Faculty of Electrical Engineering, Technion- Israel

Institute of Technology, Haifa, Israel.

0018-9448/78/0100-0122$00.75 0 1978 IEEE

on the entropy of continuous probability distributions

Documents