a partial ordering for binary channels.pdf

8/12/2019 A partial ordering for binary channels.pdf

1/11

1960 IRE TRANSACTIONS ON INFORMATION THEORY 529

A Partial Ordering for Binary Channels*N. ABRAMSONt, MJXXMBER, RE

Summary-The properties of iterated binary channels are investi-gated. An ordering (defined by the symbol 2) of communication write a 2 by 2 transition probability matrix for C,,channels with two possible inputs and any number of possible PI, Pll . * - Pl An-1outputs is defined. For any two such channels C, and CZ, this I (2)ordering has the property that if C1 3 Cz, the minimum average Pm P2, *. * P, 2-1loss when using c, willbe less than the.minimum average lossusing CZ, independent of the losses assigned to the various errors, where P,j is the conditional probability of xi given si.and indePendent of the statistics of the source. This ordering is Pii is given by the equationapplied to 1) the general binary channel, 2) the iterated binarysymmetric channel, and 3) the unreliable binary symmetric channel Pij = i=lwhen used with many iterations. Curves allowing one to use the (3)ordering are given, and an example using these curves is worked i=2out. where

I. COMPARISON OF BINARY CHANNELSA. Introduction

ONSIDER the use of a noisy binary channel totransmit information with a high degree of relia-bility. There are two common solutions to thisproblem--block coding and iteration. In this paper wedefine an ordering of binary channels pertinent when theyare used with iteration.Let a binary source emit symbols at the rate of oneper second with the probability of a zero equal to w,

wi = number of ones in xi. (4)In order to decide which of the si was sent from aninspection of the received message, it is clearly notnecessary to know exactly which of the xi was received.It is necessary only to know Wi* That is, Wi is a sufficientstatistic for the determination of the input message. Weneed, therefore, only distinguish n + 1 different outputsof the channel C, - wi = 0, 1, . . . n. By considering thechannel C, to have two possible inputs-s, and s,-andn + 1 possible outputs-O, 1, 2, . . . n-we may thenwrite a simpler transition probability matrix as shownx 1and the probability oi a one equal to (1 - ;). Let us belowdefine a channel C by the transition probability matrix

second. We may then transmit the message symbolsand let C transmit at the rate of n binary symbols perfrom the source by iteration; that is, by transmitting

_ ---groups of either n zeros or n ones through the channel C.

. .

When used in this manner, we may think of the channel Cas generating another communication channel C,. C, is

. .

a channel with two possible input. messages-r, zeros andn ones-and 2 possible output messages-the set ofall n digit binary numbers. We shall refer to the channelC, as the nth-semi-extension of C.

The problem of deciding which of the si was actuallysent on the basis of which of the outputs (0, 1 . . . n)is received can be treated by the methods of statisticaldecision theory. We define a loss matrix

C. The Decision Theory problem

L = Ll L2I 1l22where Lii is the amount we lose if we decide that si wassent, if si was the true message sent. Then, rememberingthat the a priori probability of s, is w, and the a prioriprobability of s2 s (1 - w), we may compute the optimum(Bayes) decision procedure-that procedure which mini-mizes the expected loss. Let us call the minimum valueof the expected loss R,(w, L).

B. A Suf/icient Statistic for C,Let us call the two possible input messages to C,,s1 (n zeros), and sZ (n ones). We also define the 2 possibleoutput messages as z,,, x1, . . . zznel where Xi correspondsto t,he n digit binary number j. Then it is possible to

* Received by the PGIT, December 14,1959. The work describedin this report was supported by the Office of Naval Res., ContractNonr 225(24).f Stanford Electronics Labs,, Stanford, Calif.1 p is the probability of receiving a zero, if a zero is transmitted,and p is the probability of receiving a zero if a one is transmitted.We sha : assume throughout, the paper that q 2 $ znd p 2 3.

Now, if we are given the choice of using C,, the nth-semi-extension of C or CA, the mth-semi-extension ofsome other channel C, it is a simple matter, in principle,to determine which channel to use. We need only computeR,(w, L) and RL(w, L) and choose the channel with thesmaller minimum expected loss. This method will generatea complete ordering of all possible semi-extensions of all


2/11

530 IRE TRANSACTIONS ONpossible binary channels. This ordering will, in general,depend upon w and L. The surprising fact is that it ispossible to assign a (partial) ordering to the C, whichis based upon minimum expected loss, yet which isindependent of w and independent of L. That is, in certaincases, it is possible to say that C, is better than CL (thatit will produce a smaller minimum average loss) inde-pendent of the statistics of the source (w) and independentof the losses we may assign to the different errors (L).If R,(w, L) is less than RL(o, L) for all w and all L,we say that C, is more informative than CL, written C, > CA.In the language of statistical decision theory, thedetermination of the transmitted message of C, from thereceived message is a simple hypothesis testing problem.There are two hypotheses: H,, or s1 was sent, and H,,or sZ was sent. The observation of which of the outputs(0, 1, 2, ... n) is received, is said to constitute an experi-ment, and the out,puts (0, 1, 2, . . . n) are called thepossible outcomes of the experiment.It is important to note that by giving the transitionprobability matrix (5), we have completely defined theexperiment corresponding to C,. Henceforth, we shallrefer to this matrix as the experiment matrix. The experi-ment matrix for the problem of testing hypotheses H,and H, is just a 2 X k matrix (P,,), where k is the numberof possible outcomes of the experiment. Pii is the con-ditional probability of outcome j, given that H, is true.The problem of comparison of experiments-or equiva-lently comparison of experiment matrices-has beentreated extensively in the statistical literature.2-5 Weshall follow the treatment of this subject as given byBlackwell. A detailed explanation of this treatment ispresented,6 and a condensed version is given7 in twoarticles by this author.For the sake of completeness, we shall repeat thecondition for the comparison of two experiments in thenext section.D. Comparison of Dichotomous Experiments

Let P be an experiment with k possible outcomes(x1, X2, ..* x,) used to test the hypotheses H, and H,.P may be defined by the two by Ic mat.rix

2 D. Blackwell and M. Girshick, Theory of Games and StatisticalDecisions, John Wiley and Sons, Inc., New York, N. Y.; 1954.* D. Blackwell, Equivalent comparisons of experiments,Ann. Math. Stat., vol. 24, pp. 265-272; 1953.4 D. Lindley, On a measure of the information provided by anexperiment, Ann. Math. Stat., vol. 27, pp. 986-1005; 1956.5 R. Bradt and S. Karlin, On the design and comparison ofcertain dichotomous experiments,pp: 390-409; 1956. Ann. Math. Stat., vol. 27,6 N. Abramson, Application of Comparison of Experimentsto Radar Detection and Coding Problems, Stanford ElectronicsLabs., Stanford, Calif., Tech. Rept. No. 41; July 23, 1958.7 N. Abramson, The application of comparison of experi-ments to detection problems, 1958 IRE NATIONAL CONVENTIONRECORD, pt. 4, pp. 22-26.

INFORMATION THEORYwhere Pii = P, (xi/Hi ) ; let

CYj= P,j + p,j;define

Decem

Fdt) = c ai{%:2 5 I>

(the notation on the right side of (9) indicates that any t we sum only those CY~uch that Pl,/aj < t); finaldefineK,(t) = s, Fp(w) dw. (

Then, if Q is some other experiment with the same numbof hypotheses, but not necessarily the same number outcomes,P > Q if, and only if

KP(t) >_ K,(t) for all tin [0, I]. (As suggested by (II), we shall call KP(t) the comparisfunction of experiment P.From the above definition, it may be seen that if P > then Q C P, read Q is less informative than P. Furthemore, ifPI& and Q>R

thenP > R.

That is, the relationship defined by > is transitivFinally, note that in general, given any two experimenP and Q, we cannot say that either P > Q or Q > When neither of these relations hold between experimenP and Q, we say that P and Q are noncomparable. other words, the relationship defined by > definespartial ordering over all 2 by Ic Markov matrices, equivalently over all channels with two possible tranmitted messages and any number (not necessarily same for the chamlels being compared) of possible receivmessages.

II. COMPARISON OFTHE GENERALBINARY CHANNEL

A. The Comparison Function for C,Let us apply the condition given in Section I-D two simple cases. First, consider the channel C, wh

n = 1. That is, each signal, zero or one, is sent as itreceived, and there is really no iteration (or codinat all. The experiment matrix for C, is just the transitio* This ordering may also be extended to include more genexperiments. See footnote 3.9 We shall adopt the notation

and F,(t) for Fe,(t)K,(t) for Kc,(t).


3/11

1960probability matrix

Abramson: A Partial Ordering for Binary Channelsw 10 --

.5 --

531

Q 1-qi I-1-P (12)From (8), we have

w=q+pa 2 = (1 - a> +

and from (9), remembering that0F,(t)J114) 1 I2

where

0 - PI,>,q 2 $ and p I +,

t < 6t 5 t < tC2 (13)t (2 5 t

(144t (2) A Q-. q+p. (14b)

Finally, we may plot K,(f), the comparison function ofCl, in Fig. 1.B. The Comparison of Two Binary Channels

From (11) we see that C, > C{ for any two noniteratedbinary channels C, and C{ if, and only ifK,(t) 2 K:(t) for all t in [0, 11; (15)

and after a little algebra, we see that a necessary andsufficient condition for (15) to hold ist (l) I t(l) (164

and t (2) 2 tC2). (16b)Finally, using (14), we may express (16) as

and1-P 1 - p-.1-q%-q

(174

Eq. (17) may be expressed graphically as in Fig. 2.Any channel C, with experiment matrix as in (12)may be thought of as a point in the unit square. Strictlyspeaking, since we have assumed that q >_ 3 and p I $we should only consider points in the upper left-handquarter of the unit square. In Fig. 2 we have fixed thepoint (p, q) corresponding to the channel C{. The setof channels whose parameters p and q satisfy both (17a)and (17b) will then lie in the shaded region of Fig. 2.

10This simple example, until (16), is taken with some notationalchanges, from Blacks-ell (see footnote 2). It is repeated here forpedagogical reasons.

n.0n) ~//----yi

0 I .5 1 1.0 t$11 4(2)Fig. l-Comparison function for the general binary channel.

0 0.5 1P--L

Fig. P-Comparison of Ci and Ci.

These channels are all the channels which are moreinformative than C{. The set of channels whose parametersp and q satisfy (17a) and (17b) with the inequality signsreversed will lie in the dotted region of Fig. 2. Thesechannels are all the chamlels which are less informativethan Cf. The channels corresponding to the unshadedand undotted regions of Fig. 2 are those chamlels whichare noncomparable with C:.C. The Shannon Ordering

For the case of the noniterated binary chamrel, itis interesting to compare the partial ordering presentedin the previous section with a different partial orderingdiscussed by ShannonI We shall use the symbol >- todenote Shannons ordering. It is easily shown that if wehave two channels Cl and C{, then Cl > Ci impliesCl >- C{. The converse, however, is not true. For thecase of the binary symmetric channel, it may be seen thatthe two orderings are equivalent.III. COMPARISON OFTHEITERATED BSC

A. The Comparison Function for the Iterated BXCNow consider the channel C,, the nth semi-extensionof Cl. The experiment matrix for C, is given in (5). Inprinciple we may start with the experiment matrix andcalculate its comparison function K,(t). To compare C,zl1 C. Shannon, A note on a partial ordering for communicationchannels, Information and Control, vol. 1, pp. 390-397; December,1958.


4/11

532 IRE TRANSACTION8 ON INFORMATION THEORY Decewith Cd the mth semi-extension of C: we need only examine

K(t) - K(t)and note whether this difference is non-negative or non-positive in the interval [0, 11. In practice, however, forlarge n such a procedure can become quite tedious. Forthe case of the nth semi-extension of a binary symmetricchannel (BSC), however, there does exist an easy methodof obtaining the comparison function. For the BSC t,hetransition probability matrix is

and the experiment matrix of the C,, the nth semi-exten-sion of this channel is just.

I(1 PIP>$1 PYP $1 PY(PY(PT(PI (n;) o-(l - PI ($P)V -PI . . * (1 -PI,>

09Now, using the notation of Section I-D we see that

ffj = ( _ 1,[( - p)n-f+yp)f-* + (1 _ p)+1(p)++ll\l j = (1,2, **. n + 1) Jl(20)and defining

rd-


5/11

1960 Abramson: A Partial Ordering for Binary Channels 533then

K,,.(t) I K,,, (t) for t in [0, 11.Lemma 1 states that, in order to compare iterations ofBSCs, we need only compute the comparison functionsfor 0 I t I 3.Next note the form of K,(t) as given in (29). K,(t)

is a piecewise linear function of t in the interval [0, 11.Furthermore K,(t) is continuous in this interval, whileF,(t), the derivative of K,(t), is discontinuous at thepoints t(l), f@), . . . t(n+l). We shall refer to these pointsas the breakpoints of K,(t).The breakpoints of K,(t) are easily obtained from (23~).In addition by inspection of (23~) we may immediatelywrite Lemma 2.

Lemma 2: Let K,(t) and K,-2(t) be the comparisonfunctions of the nth and n - 2 th semi-extensions ofthe same BSC. The set of n + 1 breakpoints of K,(t)consist ofa) the set of n - 1 breakpoints of K,-,(t),b> 11 + r-*

1 1-= 1 --.1 + rn 1 + r-nLemma 2 states that if we have the n - 1 breakpointsof K,-,(t), it is only necessary to compute one additionalbreakpoint to obtain the n + 1 breakpoints of K,(t).Lemma 2 suggests that a simple method of obtaining acomparison function for large n might be to start withsmaller values of n and to proceed by induction. Thisindeed will be the method we shall use but first we needtwo additional results.Lemma 3: If K,(t) is the comparison function for thenth semi-extension of a BSC then, for n an odd integer

,$ K,(t)I = 1.t=1,2The last result we need before we show how to obtainthe comparison function for the nth semi-extension ofa BSC is Lemma 4.Lemma 4: Let K,(f) and K,-,(t) be the comparisonfunctions of the nth and n - lth semi-extensions of

the same BSC. Let t:l), t:) , . . . tp+l) be the n + 1 break-points of KIL(t). ThenK,(p) = K,-,(p)

for n = 2,3, **aand j = 1, 2, .a. n + 1.

Lemma 4 tells us that K,(t) will equal K,-,(t) at thebreakpoints of K,,(t). Thus, if we know K,-,(t) we knowthe value of K,(t) at its breakpoints. K,(t) is, however,a linear function of t between any two successive break-

points so that the values of K,(t) at its breakpoints aresufficient to determine the function everywhere.C. Construction of the Comparison Function for anIterated BXC

We shall now illustrate the application of Lemmas 1,2, 3 and 4 to the construction of a comparison function.As an example, let us say we wish to construct K,(t) forn = 10 and r = 0.5. Note that by Lemma 1, we are onlyinterested in K,(t) for 0 I t 5 +. Now the breakpointsof K,,(t) and K,(t) may be computed from (23~). InTable I we have listed for these two comparison functionsall the breakpoints which occur in the interval [0, +].

TABLE IBREAKPOINTS OF Kl~(g) AND KgcL) FOR T = 0.5

Kl,,(") Kg(')0.0010.0040.1500.5900.2000.500

0.0020.0080.3000.1110.333

By Lemma 2 the breakpoints of K,(t) occurring in theinterval [0, $1 may be obtained by dropping t,he firstentry in the first column of Table I. The breakpointsof K6(t) may be obtained by dropping the first two entriesin the first column, etc. The breakpoints for K7(t), KS(t),K3(t) and K,(t) are obtained from the second columnin Table I in the same manner.The first step in the construction of K,,(t) then is toindicate the breakpoints of K,(t), K,(t), . . . K,,(t) inthe interval [0, $1. These breakpoints, which by Lemma 2consist of the eleven numbers in Table I, are shown byvertical lines in Fig. 3.Having drawn these lines, it is a simple matter toconstruct K,(t) for t in [0, 41. The only breakpoint ofK,(t) in [0, $] is the last entry in the second column ofTable I (t = 0.333). By Lemma 3, K,(t) must have a slopeof unity at t = 4. We therefore draw the line startingat this breakpoint with unit slope. This line is equalto K,(t) for 0.333 5 t I 0.500. K,(t) is equal to zerofor 0 5 t 5 0.333. K,(t) may now be drawn immediatelyas shown in Fig. 3 by a direct application of Lemma 4.In exactly the same manner, by using Lemma 4, weconstruct K,(t) to K,,(t) without further calculation.12In Figs. 4-6 we have drawn the comparison functionsfor some other values of r.D. Example

As an example of the application of these figures, con-sider the use of the third extension of a BSC with proba-

I2 t is necessary o employ the slope condition given in Lemma3 to obtain the last segmentof Kj(t) for odd j.


6/11

IRE TRANSACTIONS ON INFORMATION THEORY Dece

Fig. 3-Comparison functions, K,(t), for r = 0.5. Fig. 4-Compari son functions, K%(t), for r = 0.6.

Fig. 5-Comparison functions, K%(t), for T = 0.7. Fig. B-Compari son functions, K,(t), for r = 0.8.

bility of error equal to 0.375 (r = 0.6). By superimposingFig. 4 on Fig. 5 it may be seen thatK,,.,(t) 2 JL.40 o


7/11

1960 Abramson: A Partial Ordering for Binary Channels 535Theorem 3: Let C,, and C: be two BSCs with proba-bilities of error p and p respect,ively. Let C, be the nthsemi-extension of C,. Thena) for n odd

C, > C( if and only ifFig. 7-The maximum and minimum comparison functions. k-lPI > (334

parison function less than the MNCF at any point in [0, I].The MXCF corresponds to a channel which can t,ransmitbinary information with zero error. The MNCF cor-responds to a channel where the output is statisticallyindependent of the input.

b) for n evenC,, > C: if and only if

cn-1 3 c:. (33b)B. Some Comparison RegionsIV. COMPARISON OF THE ITERATED BSC WITH C:

A. Two Comparison TheoremsIn this section we shall investigate the comparison of C%,the nth semi-extension of one BSC, with C{, some othernoniterated BSC. By means of such a comparison it will

be possible to compare iterated BSCs in general. That is

In most of the remainder of Section IV we shall beinterested in investigating the properties of highly un-reliable BSCs when used with a large number of itera-tions. Accordingly, we shall find it convenient to definethe BXC parameter E by

if c, > c:and C: > C;,

then we know thatc, > c:.

The reason for comparing two iterated BSCs throughthe artifice of introducing another, noniterated BSC maybe seen in the following 1emma.14Lemma 5: Let C, be the nth semi-extension of theBSC C, and let C: be some other BSC. Thena) C: > C, if and only if

t < tit:..r - (30)b) C, > C{ if and only if

KA) 2 Kl,A)). (31)We are now in position to state the two central resultsof Section III. First, by using in (30) the expression fort given in (23c), and then writing the result in termsof p, we obtain Theorem 2, as follows:Theorem 2: Let C, and C: be two BSCs with proba-bilities of error p and p respectively. Let C, be the nth

semi-extension of C,. ThenC: > C, if and only if

(P)p 5 --*(PI + (1 - PI,>Next we note that for n even t:2+1 = 3, so thatby Lemma 4, K,(s) = K,-,(q), again for even n. Thento get K,-,(s), we use (29), setting t = 3, and finallyobtain:I4 Lemma 5 is proved in the appendix.

p = $(l - E) (34)where p is the probability of error of the BSC. We recallthat

Pr=l--p cmso that

1-cr=FTe (35)For a highly unreliable BSC the BSC parameter E willapproach zero.Let E and E be the BSC parameters of C, and C: re-

spectively. Then (32) may be written asCf > C, if and only if

1 - t C, if and only if

(3,-bwhere we have defined

n0 = 0 for m > n.m

Likewise, we may rewrite (33a) in terms of E and Eas follows: For n oddC, > C{ if and only if

E < 1 - @-I ,.g2 (k $1 + $(I - P+l. (37)


8/11

536 IRE TRANSACTIONS ON INFORMATION THEORY DecemIn Figs. 8-11 we have plotted in the E, e plane regionswhere C{ > C, and where C, > C: for n = 3,5,7 and 9.Also included in these figures is the equicapacity line-the line which gives for any 2 the value of E such thatthe Shannon chamlel capacity of C, (in bits per unittime) is equal to that of C:. The channel capacity of C,is given by

n[l + P log p + (1 - PI log (1 - $1 (38)where we have multiplied by n so that one digit fromthe source may be transmitted by C, (using n iterations)in the same time that it may be transmitted by C:. The

channel capacity of C: is, of course,1 + p log p + (1 - p) log (1 - p).

It is interesting to note that the equicapacity lies in the noncomparable regions of Figs. 8-11 excfor a small range of values of 6 corresponding to higreliable BSCs. This range of values vanishes rapidly n increases. In this range, however, we have the teresting phenomenon of two channels C, and C, wthe channel capacity of C, greater than that of C yet with the average loss using Cn (in an iterated mannalways greater than that of C.

jg - No of iterations = 5 /*I I I I I I I l/l I

0 .2 .3 4 .5 .6 .7 .s . 9 E-Fig. S-Comparison of the iterated BSC with the noniterated BSC. Fig. 9-Comparison of the iterated BSC with the noniterated

.6

.5

A

.3

.2

.f

0 0

.s No of iterations = 9 -

.7

Fig. lO-Comparison of the iterated BSC with the noniterated BSC. Fig. 11-Comparison of the iterated BSC with the noniterated


9/11

1960 Abramson: A Partial Ordering for Binary ChannelsC. Iteration of Unreliable BSC s

One final question of interest is the. behavior of highlyunreliable channels when used with many iterations. Thatis, we are interested in comparing C, with C: when eapproaches zero and n is a large number.Under these conditions (36b) reduces toC{ > C, if and only if

IE 5 5.n (40)We may also let c approach zero in (37). After a gooddeal of algebraic manipulation and the application ofStirlings formula, this will reduce to

C, > C: if and only if

Proof: From (10)537

(43)and letting

v=l-w

K,(t) = j-, F,O - v) dv, (44)but from (28) we see that if K(t) is the comparison func-tion of the nth semi-extension of a BSC, then

Fn(1 -v)=2- F,(v) for v in CO,11 (45)except on a finite set of points, so that

J I--.2 ;z (41) - Fnb)l dv (46)Finally from (38) and (39) we obtain (again for small e):The capacity of C, [from (38)] is equal to the capacity

of C: [from (39)] when+. (42)

= 2t - K,(l) + Kdl - 0= (2t - 1) + K,(l - 0.

Lemma 1 then follows directly from (46).Lemma 2:

We may indicate the results of (40)-(42) as shown in Let K,(t) and K,-z(t) be the comparison functions ofFig. 12. the nth and n - 2th semi-extensions of the same BSC.No. of iterations = n

The set of n + 1 breakpoints of K,(t) consist oft

slops a) the set of n - 1 breakpoints of K,,-)(t)eb> 11 + ren4 1 1-=l-----.1 + rn 1 + rPn

Proof: Lemma 2 follows directly from (23~).Lemma 3:If K,(t) is the comparison function for the nth semi-e + extension of a BSC then, for n an odd integer

Fig. 12-Comparison of the iterated unreliable BSC with thenoniterated BSC. [ 1K*(t) =l.t=1,2APPENDIX Proof: For n odd we may use (29) where j = (n + 1)/2.Some Proofs Then, taking a derivative with respect to t, we get

Lemma 1: (n+l)/ZLet K,,,(t) and K,,,,(t) be the comparison functions [ 1KS0 (47)t=1,2 = 2 a@).of the nth and mth semi-extensions of the BSCs C, ,,and C, .I, respectively. Finally, from (19) we see that the right side of (47)is just one.IfK,,,(t) 2 fL,,dt) for tin IX, 31 Lemma 4:

thenK,,.(t) 2 K,,,.(t) for tin [O, 11.

Let K,(t) and K,-,(t) be the comparison functions ofthe nth and n - lth semi-extensions of the same BSC.Let t(l) p -.. pi117s I n , 11 be the n + 1 breakpoints of K,(t).


10/11

538Then

IRE TRANSACTIONS ON IATFORMATION THEORYI

Decem

K,(t;) = K,-,(t,?) F,,(t)for n = 2,3, a*.

Fn I ,(t)

and j= 1,2, .-an+ 1.Proof: The first step is to note that

K,(t:l)) = K,-,(t:) = 0;next we show that

K,(t:) = K,-,(t:).K,(t) is just the integral of F,(t). In Fig. 13 we haveplotted the first section of F,(t) and Ill,_l(d). Referringto this figure we see that K,(tp) is just the area underF,(t) from 0 to tz while K,-,(tp)) is the area underFmA1(t) in this same interval. Thus to show that thesetwo areas are equal we need only show that the shadedarea A, in Fig. 13 is equal to t,he shaded area H,. That is,we must prove that

[Pl - p] [oL:l)] = [t: - ti,] [o&y1 - cp]. (50)Substituti ng for the tj and $ in this equation willthen prove (49).We might continue in this manner and show that areasA, and B, of Fig. 13 are equal so that K,(t) and K,-,(b)are equal for t = t3,. It is easier, however, to tacklethe general problem immediately. That is, referring toFig. 14, we shall show that the fact that area AieI isequal to area B,-* implies that area Ai is equal to area Bi.In terms of the quantities in Fig. 14 we wish to showthat if[L,-,][t: - t:li;] = [oL::;1) - L,_J[ty - ty] (51)then we must haveit+) - t;:& :: - cp + Lj-,]

= pn(l: - ty] [cp - L;-,I. (52)From (51) we may obtain an expression for L,+,

(j-1)L.- = &-1) L&L- .

_ t-lI 1 1 ,;?I _ t;i-" ,

after substitution of (53) in (52) and a good deal ofalgebra WC obtain

finally, we substitute for tik aud aiL and verify (54).Let us summarize what we have done. First we haveshown thatKn(t:j)) = K,-,(t:) (55)

for j = 1 and 2. Eq. (55) would hold for all j if we couldprove that the areas we have labeled Ai and Bi wereequal. Finally we proved that Ai = Bi if Ai-, = Bj-l:since we had already proved A, = B, this completedthe proof and (55) holds for all j.

Fig. 13-The first section of F,(t) and Fn-l(t).

F (t] end F, _,(t)

fI

-- ___________ -------r----

(j +I).?

L--l-

----- III I I II I I I I1 I I II I I I I$ - 1) t~J--IO ,(i) ,(j)n n-1 &+=I t-n

Fig. 14-A middle section of F,,(t) and F,-l(t).

Theorem 1:Let K,(t) be the comparison function of the nth sextension of a BSC with transition probability mar1-P P\

i P 1 -Pia) ifp < 3

lim K,(t) = t o


11/11

1960 Wonham: Probability Density of the Output of a Low-Pass System 539Proof: From (29) we have (for n odd)

(n+1)/2 (n+l)/ZK,,(+) = 5 go? - g ack)tk * (56)But for n odd

(n+1)/2z Q(k) = 1 (57)and(n+1)/2 (k)t(k-) = (nx2 (k 1&l - p)k--l(p)n-k+l . (57)

The sum on the right of (57) is just the probabilityt,hat a random variable n, having a binomial distributionof mean (n) (1 - p) and variance (n) (1 - p)(p) willbe less than n/2. If p < 3, (n)(l - p) > n/2 and wemay write (57) as(n+1)/2

< (PM - -p>.- n($ - p),>Where the last step is obtained from the Bienayme-Tchebycheff inequality. Now we use (56) and (57) in(55), let n approach infinity, and we obtain (for n odd

and p < 4)lim K,(s) = $. (59)w-m

Furthermore, since for a fixed r, C,,, > C,, (59) mustalso hold for eveil values of n. Finally, since K,(O) = 0

and K,(l) = 1 for all n, and d/dt K,(t) is a nondecreasingfunction of t, (59) proves part a) of the theorem.Part b) is then proved simply by noting that if p = 3,we have r = 1, and the only breakpoint of K,,(t) is att = 3.Lemma 5:Let C, be the nth semi-extension of the BSC C, and

let C{ be some other BSC. Thena) C: > C, if and only if

t :;, _< t;;;,b) C, > C: if and only if

K,.(+) 2 ~WG).Proof: From Lemma 1 we see that we need only prove

4 C(t) 2 K(t) in P, $1if and only if

t:y < t(l), - n.7,b> K,(t) 2 K(t) in LO, 1

if and only ifK,,(3) 2 Kl..G).

Note that tjfi, is the only breakpoint of Ii:(t) in[0, +] so that by Lemma 3, the slope of K:(t) is unityin the interval tit:, < t 5 +. The slope of K,(t), however,is never greater than unity in this interval (again byLemma 3). Parts a) and b) of Lemma 5 follow directlyfrom these two facts.

On the Probability Density of the Output of a Low-PassSystem When the Input is a Markov Step Process*

W. M. WONHAMt

Summary--Forward equations are derived for the (N + l)-dimensional Markov process generated when a Markov step signal[s(t)) is the input to an Nth -order system of the form dX/df =U( X; s). As examples, the joint probability densities of input andoutput are found for a symmetric three-level signal smoothed byan RC low-pass filter, and partial results are obtained for a doublyintegrated Rice telegraph signal.

* Received by the PGIT, January 8, 1960; revised manuscriptreceived, June 2, 1960.t Dept. of Engrg., University of Cambridge, Cambridge, Eng.

INTRODUCTIONF THE INPUT to a (possibly nonlinear) system isa Markov process, and the system belongs to acertain class of functionals, an equation for thecharacteristic function of the output distribution can bewritten using the methods of Darling and Siegert.

i D. A. Darling and A. J. F. Siegert, A systematic approachto a class of problems in the theory of noise and other r andomphenomena-I, IRE TRANS. ON INFORMATION THEORY, vol. IT-3,pp. 32-37; March, 1957.

a partial ordering for binary channels.pdf

Documents