near shannon limit error - correcting coding and decoding

8/6/2019 Near Shannon Limit Error - Correcting Coding and Decoding

1/7

NEAR SHANNON LIMITERROR - CORRECTINGCODING AND DECODING :TURBO-CODES (1 )Clau de Berrou, Alain Glavieux and Punya Thitimajshima

Claude Berrou, Integrated Circuits for Telecommunication LaboratoryAlain Glavieux and Punya Thitimajshima, Digital Communication Laboratory

Ecole Nationale SuNrieure des Ttlkommunications de Bretagne, France(1) Patents No 9105279 (France), No 92460011.7 (Europe), No 07/870,483 (USA)

Abstract - This paper deals with a new class of convolutionalcodes called Turbo-codes,whose performances in terms ofBit Error Rate (BER) are close to the SHANNON limit. TheTurbo-Code ncoder is built using a parallel concatenation oftwo Recursive Systematic Convolutional codes and theassociated decoder, using a feedback decoding rule, isimplemented as P pipelined identical elementary decoders.

I - INTRODUCTIONConsider a binary rate R=1/2 convolutional encoder withconstraint length K and memory M=K-1. The input to theencoder at time k is a bit dk and the corresponding codewordCk is the binary couple ( X k , Y k ) with

K -Ii =O

x k = z g I i d k - ; m d . 2 g l i =0,1 (la)K- 1Yk = zg2idk- i m d . 2 82i =0,1 ( I b )i = Owhere G I: {gli) ,G2: ( g 2 i } are the two encoder generators,

generally expressed in octal form.It is well known, that the BER of a classical NonSystematic Convolutional (NSC) code is lower than that of aclassical Systematic code with the same memory M at largeSNR. At low SNR, it is in general the other way round. Thenew class of Recursive Systematic Convolutional (RSC)codes, proposed in this paper, can be better than the best NSCcode at any SNR for high code rates.

A binary rate R=1/2 RSC code is obtained from aNSC code by using a feedback loop and setting one of thetwo outputs X k or Yk equal to the input bit dk . For an RSCcode, the shift register (memory) input is no longer the bit dkbut is a new binary variable ak. If Xk=dk (respectivelyYk=dk), the output Y k (resp. X k ) is equal to equation (lb)(resp. la ) by substituting ak for dk and the variable ak isrecursively calculated as

K - 1ak = dk + 1 i a,- ; m d . 2 ( 2 )i = lwhere yi is respectively equal to g l i if Xk=dk and to g 2 i if

Yk=dk.Equation (2) can be rewritten asK 1i= O

dk = Z 7 ; U k - i tmd.2. (3 )One RSC encoder with memory M=4 obtained from an NSCencoder defined by generators G1=37, G2=21 is depicted inFig. 1. Generally, we assume that the input bit dk takesvalues 0 or 1 with the same probability. From equation (2),we can show that variable ak exhibits the same statisticalproperty0-7803-0950-2/93/$3.00Q1993IEEE 106.1

Pr{ak =El, . .ak- l =Ek - l)= Pr(d k = E ) = 1 / 2 (4 )with E is equal to

K -1I= 1E = C y i e i md . 2 E =0,1. ( 5 )Thus the trellis structure is identical for the RSCcode and the NSC code and these two codes have the samefree distance df However, the two output sequences (Xk} nd

{ Y k ) do not correspond to the same input sequence (dk) forRSC and NSC codes. This is the main difference between thetwo codes.When punctured code is considered, some outputbits X k or Y k are deleted according to a chosen puncturingpattern defined by a matrix P . For instance, starting from arate R=1/2 code, the matrix P of rate 2/ 3 punctured code is

L J

Fig. 1a Classical No n Systematic code. Yk

Fig.1b Recursive Systematic code. 'k


2/7

I1 - PARALLEL CONCATENATION OF RSC CODESWith RSC codes, a new concatenation scheme,called parallel concatenation can be used. In Fig. 2, anexample of two identical RSC codes with parallelconcatenation is shown. Both elementary encoder (C l andC2 ) inputs use the same bit dk but according to a differentsequence due to the presence of an interleaver. For an inputbit sequence {&), encoder outputs Xk and Yk at time k arerespectively equal to dk (systematic encoder) and o encoderC1 output Y l k , or to encoder C2 output Y2k. If the codedoutputs ( Y l k , Y 2 k ) of encoders C1 and C 2 are usedrespectively n l times and n 2 timesand soon, the encoder C1rateR 1and encoder C2 rateR2 are equal to

(6)n, + n ,R, = 2n, + nl *

*k

SystemabCode (37,21)

I

Fig. 2 Recursive Systematic codeswith parallel concatenation.

The decoder DE C depicted in Fig. 3a, is made up of twoelementary decoders (DEC1 and DEC2) in a serialconcatenation scheme. The first elementary decoder DECl isassociated with the lower rate R1 encoder C1 and yields asoft (weighted) decision. The error bursts at the decoderDECl output are scattered by the interleaver and the encoderdelay L1 is inserted to take the decoder DECl delay intoaccount. Parallel concatenation is a very attractive schemebecause both elemenmy encoder and decoder use a singlefrequency clock.For a discrete memoryless gaussian channel and abinary modulation, the decoder DE C input is made up of acouple Rk of two random variables nk andyk, at time k

x k = ( 2 4 - )+ , ( 7 4Y =w, u + q , , (76)where ik and q k are two independent noises with the samevariance 02 .The redundant information yk is demultiplexedand sent to decoder DECl when Y k = Y l k and toward decoderDEC2 when Yk = Y 2k . When the redundant information of a

given encoder ( C l or C2) is not emitted, the correspondingdecoder input is set to zero. This is performed by theDEMUXDNSERTION block.It is well known that soft decoding is better thanhard decoding, therefore the first decoder DE Cl must deliverto the second decoder DEC2 a weighted (soft) decision. TheLogarithm of Likelihood Ratio (LLR), Al(dk ) associatedwith each decoded bit dk by the first decoder D E C l is arelevant piece of information for the second decoder DEC2

(8)P,{dk = 1 observution)" dk' = Log P,{dk = O/observution)where P,{dk = i lobservation), i = 0, 1 is the a posterioriprobability (APP) of the data bitdk.

- _ -DEMUWINSERTIONFig. 3a Principle of the decoderaccording toa serial concatenation scheme.

I11 - OPTIMAL DECODING OF RSC CODES WITHWEIGHTED DECISIONThe VITJZRBI algorithm is an optimal decodingmethod which minimizes the probability of sequence errorfor convolutional codes. Unfortunately this algorithm is notable to yield the APP for each decoded bit. A relevantalgorithm for this purpose ha s been proposed by BAHL et al.[l]. This algorithm minimizes the bit error probability indecoding linear block and convolutional codes and yields theAPP for each decoded bit. For RSC codes, the BAHL et al.algorithm must be modified in order to take into account theirrecursive character.

I11 - 1 Modified BAHL el al. algorithm for RSC codesConsider a RSC code with constraint length K; t

time k the encoder stateSk is represented by a K-uple

Also suppose that the information bit sequence ( d k ) is madeup of N independent bits dk, taking values 0 and 1 with equalprobability and that the encoder initial state So and final stateSN areboth equal to zero, i.eThe encoder output codeword sequence, notedgaussian memoryless channel whose output is the sequenceR: = (R,.......R~ ....... N) where Rk =(Xk,yd is defined byrelations (7a) and (7b).

s& (u&.a&-, ......a,_,+, ). (9 )

so =sN= 0,0 .....0)=0. (10)Cy = {C,......Ck .......CN)is the input to a discrete

1065


3/7

The APP of a decoded data bit dk can be derivedfrom the joint probability AL(m) defined byand thus, the APP of a decodeddata bit dk is equal to

P r[ dk = i / R ; ] = Z . A i ( m ) , i = o , L (12)From relations (8)and (12), he LLR A (d k associated witha decoded bit dk can be written as

n , (m ) =e {dk = i , s k = m / R Y ] (11)

m

mFinally the decoder can make a decision by comparingA (d k to a threshold equal to zerod k = l if A ( d k ) > o& = o $ A ( d k ) < o . ( 1 4 )

In order to compute the probability A i ( m ) , let us introducethe probability functions a; ( m ) p& m ) and yi(Rk,m, m )

Thus we obtain

(19)Taking into account that events after time k are notinfluenced by observation R: and bit dk if state Sk is known,the probability &( m) is equal

A,(m) = al(m)Pk(m). (20)The probabilities a ; ( m ) and & ( m ) can be recursivelycalculated from probability y i ( R k ,m, m ) .From annex I, weobtaina ; ( m ) = mi=O

1Z y i (R k ,m ,m)a : - , (m )zzz i (R k ,m ,m)a f _ , (m ) (21)1 1m mi=Oj=O

and~ Y i ( R & + i , m , m ) P & + i ( m )

* (22)k (m ) = mi=O1 1EZ Z.Z .y i (R k+l ,m ,m)a i (m )m mi=Oj=OThe probability yi(Rk,m,m ) can be determined fromtransition probabilities of the discrete gaussian memoryless

channel and transition probabilities of the encoder trellis.From relation (17), yi (R,, m, m ) is given byY ; (R k ,m ,m)=p (R k / d k = i ,& =m,s k-1 = m )

q(dk = / s k = m,s&-l = m)z(s&= m/ S & - l= m) (23)where p(./.) is the transition probability of the discretegaussian memoryless channel. Conditionally to(dk = i, Sk = m , Sk-1= m ),xk and y k are two uncorrelatedgaussian variables and thus we obtain

p(Rk /dk = i, s k = m, s k - 1 = m ) =p(x k /dk = i, k = m, k-1 = m )p ( y & dk = ,s&= m, Sk-1 = m). ( 2 4 )

Since the convolutional encoder is a deterministic machine,q (d k = i / S k = m,S & -] m ) is equal to 0 or 1. Thetransition state probabilities x(sk = /s&-l m ) of th etrellis are defined by the encoder input statistic.Generally,P, (dk = 1) = P, id k = 0) = 1/2 and since thereare two possible transitions from eachstate,n(Sk= m/Sk-l = m) = 1 /2 for each of thesetransitions.Different steps of modified BAHL et al. algorithminitialized according to relation (12)-Step 0 : Probabilities a ; ( m ) and P N ( m ) are

a i (0)=1 a ; ( m ) = O h # O , i = O , l (25a)PN(0) 1 p(m) = 0 Vm # 0. 05b)-Step 1 : For each observation Rk. the probabilities

af(nz)and yi(Rk,m,m ) are computed using relations (21)and (23) respectively.

-Step 2 : When the sequence R; has beencompletely received, probabilities p k ( m ) are computed usingrelation (22), and probabilities a ; ( m ) and & ( m ) aremultiplied in order to obtain & ( d .inally the LLRassociated with each decoded bit dk is computed fromrelation (1 ).IV- HE EXTRINSIC INFORMATION OF THE RSCDECODERIn this chapter, we will show that the LLR h(dk)

associated with each decoded bit dk , s the sum of the LLRof dk at the decoder input and of another information calledextrinsic information, generated by the decoder.Using the LLR A(dk)definition (13) and relations (20) and(21), we obtain

A ( d k )= Log J - T 0 . ( 2 6 )Since the encoder is systematic (Xk= dk), the transitionprobability p(xk / d k = i,s&= m,Sk- , = m ) in expressiony i (Rk ,m ,m) is independent of state values Sk and Sk-1.Therefore we can factorize this transition probability in thenumerator and in the denominator of relation (26)

1CC Cy1(Rk,m,m)a:_,(m)pk(m)ZZm j = O m, m)a:-,(m)p,(m)

1066


4/7

feedback loop I

-,_ - - . leaving

DEMUWINSERTIONJIdecoded output

Fig. 3b Feedback decoder (under0 internal delay assumption). GkV- 1 Decoding with a feedback loop

1zzz l ( y k , , m ) a i - l ( m ) P k ( m )Z Z o y k ,m, )a:-,( m )bkm)Lo g j ; O . (27)m m j = 0

Conditionally to dk = l (resp. dk =O), variables xk aregaussian with mean 1 (resp. -1 ) and variance 02 , hus theLLR A (dk) is still equal to

2A ( d k ) = T x k + wk (28)dwherewk = A ( d k ) Ix1=o =

1zz y l ( y k , m , m ) a ~ - l ( m ) P k ( m )zz : o ( y k ,m,m)aJ- , (m)P, (m)

jrO . (29)m m j = 0Wk is a function of the redundant information introduced by

the encoder. In general Wk has the same sign as dk; thereforewk may improve the LLR associated with each decoded databit dk . This quantity represents the extrinsic informationsupplied by the decoder and does not depend on decoderinput xk .This property will be used for decoding the twoparallel concatenated encoders.

V - DECODING SCHEME OF PARALLELCONCATENATIO N CODESIn the decoding scheme represented in Fig. 3a,decoder D E C l computes *LLRAl(dk) for each transmittedbit dk from sequences (xk} and {yk } hen the decoder DEC2performs the decoding of sequence(d k } from sequences(A1 dk) ) and ( y k ] .Decoder DECl uses the modified BAHLet al. algorithm and decoder DEC2 may use the VITERBIalgorithm. The global decoding rule is not optimal becausethe first decoder uses only a fraction of the availableredundant information. Therefore it is possible to improve theperformance of this serial decoder by using a feedback loop.

We consider now that both decoders DECl andDEC2 use the modified BAHL et al. algorithm. We haveseen in section IV that the LLR at the decoder output can beexpressed as a sum of two terms if the decoder inputs wereindependent. Hence if the decoder DEC2 inputs Al(dk) andy 2k are independent, the LLR A2(dk) at the decoder DEC2output can be written aswith

A2 (dk 1= f A1 (dk )) + wzk2

cr2

(30)

(31)l ( d k ) = - X + w1kFrom relation (29), we can see that the decoder DECzextrinsic information W2k is a function of the sequence{A1 (d , ) ]n+k Since A 1 (d,) depends on observationRp.extrinsic information W2k is correlated with observations . I C ~and ylk. Nevertheless from relation (29), the greater I -k is,the less correlated are A I (d,) and observations xk , yk. Thus,due to the presence of interleaving between decoders DECland DEC2, extrinsic information W2k and observationsxbylk are weakly correlated. Therefore extrinsic informationW2kand observationsxk , ylk ca n be jointly used for carryingout a new decoding of bit dk, the extrinsic informationzk= W a acting as a diversity effect in an iterative process.

In Fig. 3b, we have depicted a new decoding schemeusing the extrinsic information W2k generated by decoderDEC2 in a feedback loop. This decoder does not take intoaccount the different delays introduced by decoder DECland DEC2 and a more realistic decoding structure will bepresented later.The fist decoder DECl now has three data inputs,(xk, y l k , z k ) and probabilities a i k ( m ) and&,(m) arecomputed in substituting Rk = { x k , y l k ] by Rk =&k, y lk , a) nrelations (21) and (22). Taking into account that Q isweaklycorrelated with xk and y l k and supposing that ,Q can beapproximated by a gaussian variable with variance a: # 02,the transition probability of the discrete gaussian memorylesschannel can be now factored in three terms

1067


5/7

p ( R k / d k = i , & =m,sk-1 = m ' ) = P ( X k / . ) P ( Y k / . ) P ( Z k / . ) ( 3 2 )The encoder C1 with initial rate R I , hrough the feedbackloop, is now equivalent to a rate R'1 encoder with

( 3 3 )The first decoder obbins an additional redundant informationwith zk that may significantly improve its performances; theterm Turbo-codes is given for this iterative decoder schemewith reference to the turbo engine principle.With the feedback decoder, the LLR hl (dk) generated bydecoder DECl is now equal to

where Wlk depends on sequence ( z ~ ) ~ + ~s indicatedabove, information zk has been built by decoder DEC2 at theprevious decoding step. Therefore zk must not be used asinput information for decoder DE C2. Thus decoder DEC2input sequences at step p ( p 2 2 ) will be sequences

( 3 5 )Finally from relation ( 3 0 ) , decoder DEC2 extrinsicinformationzk= Wzk, after deinterleaving, can be written as

and the decision at the decoder DE C output isThe decoding delays ~ introduced by decoder D E C(DEC=DECl+DECz), the interleaver and the deinterleaverimply that the feedback information zk must be used throughan iterative process as represented in Fig. 4a, 4b. In fact, theglobal decoder circuit is composed of P pipelined identicalelementary decoders (Fig. 4a). The pth decoder D E C(Fig. 4b) input, is made up of demodulator output sequences( x ) ~nd C y ) p through a delay line and of extrinsic information( z ) ~ enerated by the @-1)th decoder DE C. Note that thevariance 0; of the extrinsic information and the variance ofi,d k1must be estimated at each decoding step p .

V-2 InterleavingThe interleaver uses a square matrix and bits { d k }are written row by row and read pseudo-randomly. This non-uniform reading rule is able to spread the residual errorblocks of rectangular form, that may set up in the interleaverlocated behind the first decoder D E C 1 , and to give thegreater free distance as possible to the concatenated (parallel)code.

V I - RESULTSFor a rate R=1/2 encoder with constraint length K= 5 ,generators G 1=37, G2=21 and parallel concatenation(Rl=R2=2/3) ,we have computed the Bit Error Rate (BER)after each decoding step using the Monte Carlo method, as afunction of signal to noise ratio E b/No where Eb is theenergy received per information bit dk and No is the noisemonolateral power spectral density. The interleaver consistsof a 256x256 matrix and the modified BAHL et al. algorithmhas been used with length data block of N=65536 bits. In

- -emodulator v vFig. 4a Modular pipelined decoder, corresponding to aniterative processus of the feedback decoding.

(41I

($1 I6 DELAY UNE Ivt i,

(YIP1

htamabtedmddanpltig. 4b Decoding module (level p).

order to evaluate a BER equal to we have considered128 datablocks i.e. approximatively 8 x106 bits dk. The BERversus E b / N o , for different values of p is plotted in Fig. 5 . For any given signal to noise ratio greater than 0 dB, he BERdecreases as a function of the decoding step p. The codinggain is fairly high for the first values of p @=1,2,3) andcarries on increasing for the subsequent values of p. Forp=18for instance, the BER is lower than at & / N o = 0,7 dB.Remember that the Shannon limit for a binary modulationwith R=1/2, is P , = 0 (several authors take P e = 1 0 - 5 as areference) for @,/NO= 0 dB . With parallel concatenation ofRSC convolutional codes and feedback decoding, theperformances are at 0,7 dB from Shannon's limit.The influence of the constraint length on the BERhas also been examined. For K greater than 5, at&,/No= 0, 7 dB, the BER is slightly worst at the first @ = 1 )decoding step and the feedback decoding is inefficient toimprove the final BER. For K smaller than 5 , atEb/ N o= 0,7 dB, the BER is slightly better at the firstdecoding step than for K equal to 5 , but the correctioncapacity of encoders C1 and C2 is too weak to improve theBER with feedback decoding. For K=4 (i.e. &stateelementary decoders) and after iteration 18,a BER of isachieved at & / N o = 0,9dB. For K equal to 5, we have testedseveral generators (GI , G 2 ) and the best results wereachieved with G1=37, G2=21.

1068


6/7

0 1 2 3 4 5 Eb/No (dB)theoreticalh i tFig. 5 Binary error rate given by iterative decoding (p-I, ...,18 )of code of fig. 2 (rate:1/2); nterleaving 256x2 56.

L

BER could increase during the iterative decoding VI1 CONCLUSIONForthatprocess. In order to overcome this effect, we have divided theextrinsic information zk by [ 1+ Oli,(d,( ]with 0 = 0,15.

In Fig. 6, the histogram of extrinsic information (TIhas been drawn for several values of iteration p, with all databits equal to 1 and for a low signal to noise ratio(&,/No= 0,8 dB). For p =l (first iteration), extrinsicinformation (z) is very poor about bit dk , furthermore thegaussian hypothesis made above for extrinsic information( z ) ~ ,s not satisfied! Nevertheless when iteration p increases,the histogram merges towards a gaussian law with a meanequal to 1.For instance, for p13, xtrinsic information (z)becomes relevant information concerning data bits.

In this paper, we have presented a new class ofconvolutional codes called Turbo-codeswhose performancesin terms of BER are very close to SHANNON'S limit. Thedecoder is made up of P pipelined identical elementarymodules and rank p elementary module uses the datainformation coming from the demodulator and the extrinsicinformation generated by the rank @ l) module. Eachelementary module uses a modified BAHL et aI. algorithmwhich is rather complex. A much simpler algorithm yieldingweighted (soft) decisions has also been investigated forTurbo-codes decoding [2],whose complexity is only twicethe complexity of the VITERBI algorithm, and withperformances which are very close to those of the BAHL etal. algorithm. This new algorithm will enable encoders and

1069


7/7

decoders to be integrated in silicon with error correctingoerformances unmatched at the Dresent time.

INORMALIZED -1 0 +I t 2 IANNEX I :EVALUATION OF PROBABILITIES af ( m )

SAMPLESFig. 6 Histograms of extrinsic information z afteriterations W 1,4,13 at EWNo = 0.8dB;all information bits d = l .

AND P k ( m )From relation (15)probability a; ( m ) is equal to

The numerator of a ; ( m ) can be expressed from state Sk-1and bit dk -1.Pr{dk =i , sk =m ,Rk / R: - ' ]=

By using BAYES rule, we can writePr{dk = i,sk= m, Rk / R : - ' ] =

e [d k = i,sk= m, k / dk -' = J ,sk-l= m' , R:-']. ( A3 )By taking into account that events after time (k-1) are notinfluenced by observation RF-' and bit dk-1 if state Sk-1 sknown and from relation (17)we obtain

Pr{dk =m.Rk/R:-'] =1

Z1 i ( Rk , m 'm ) aL _ , ( m ' ) . A 4 )m ' j = OThe denominator can be also expressed from bit dk and states k

and from relation (A4),we can write :1070

1 1Pr{Rk /R:- '] = Z E Z: 1 i (Rk ,m'm)a; - , (m ' ) . (A6)Finally probability a; m) can be expressed from probabilityai.-lm ) by the following relation

m m' i=O j = O

1ZZ i R k ,m' m )ai 1 (m' )XZ z y i ( R k , m ' m ) & ( m t ) * ( A 7 )m )= " i = O1 1m m ' i= O j = O

In the same way, probability Pk ( m ) an be recursivelycalculated from probability Pk+l (m). From relation (16),wehave

' r {R?+l l S k = m ] -P k ( m )= pr {@+I

By using BAYES rule, the numerator is equal to

P k ( " ) = "i=O . (A10)8 Rk+l / R / ]In substituting k by (k+I) in relation (A6), the denominatorof (A10) is equal to

Finally probability Pk (m)can be expressed from probabilityPk+l (m'),by the following relation

1Z y i ( Rk + l , m , m ' ) Pk + l ( m ' )p k i m ) = m'i=O 1 . (A12)1 Y i ( Rk + l ) "m ) aL ( " )

m m ' i=O j=OREFERENCES[ l ] L.R. Bahl, J. Cocke, F. Jeinek and J. Raviv,"Optimal decoding of linear codes for minimizing symbol

error rate", IEEE Trans. Inform. Theory,vol. IT-20,pp. 248-287,March 1974.[2 ]C. Berrou, P. Adde, E. Angui and S . Faudeil, "Alow complexity soft-output Viterbi decoder architecture", oappear at ICC' 93 .

near shannon limit error - correcting coding and decoding

Documents