DIGITAL BASEBAND TRANSMISSION AND

RECORDING

DIGITAL BASEBAND TRANSMISSION AND

RECORDING

Jan W.M. Bergmans

Philips Research Eindhoven, The Netherlands

.. SPRINGER-SCIENCE+BUSINESS MEDIA, B.V.

A C.I.P. Catalogue record for this book is available from the Library of Congress

ISBN 978-1-4419-5164-9 ISBN 978-1-4757-2471-4 (eBook) DOI 10.1007/978-1-4757-2471-4

Printed an acid-free paper

AlI Rights Reserved © 1996 Springer Science+Business Media Dordrecht Originally published by Kluwer Academic Publishers in 1996 No part of the material protected by this copyright notice may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording or by any information storage and retrieval system, without written permis sion from the copyright owner.

CONTENTS

PREFACE xi

1 MATHEMATICAL PRELIMINARIES 1 1.1 Introduction 1.2 Deterministic signals and sequences 1.3 Stochastic signals and sequences 14 1.4 Probability density functions 25 1.5 Averages 29 1.6 Discrete-time minimum-phase functions 31 1.7 Lagrange multipliers 35 1.8 Further reading 37 APPENDIX lA: The Fourier transform for continous-time signals 37 APPENDIX 1B: The Fourier transform for discrete-time signals 40 APPENDIX I C: Spectral characteristics of minimum-phase functions 42 PROBLEMS 43 REFERENCES 46

2 EXAMPLES OF DIGITAL BASEBAND TRANSMISSION SYSTEMS 47 2.1 Introduction 47 2.2 Digital subscriber lines 47 2.3 Digital magnetic recording 55 2.4 Digital optical recording 74 2.5 Further reading 87 APPENDIX 2A: Nonlinear bit-shifts due to bandwidth limitations of the

write path 88 APPENDIX 2B: Nonlinear bit-shifts and write precompensation 90 APPENDIX 2C: Additive models for transition jitter 92 APPENDIX 2D: Nonlinear lSI induced by domain bloom 95 PROBLEMS 96 REFERENCES 98

3 CHARACTERISTICS OF DIGITAL BASEBAND TRANSMISSION 105 3.1 Introduction 105

v

vi

4

DIGITAL BASEBAND TRANSMISSION AND RECORDING

3.2 System model and nomenclature 106 3.3 Optimum modulation and demodulation for an ideal low-pass channel 115 3.4 Capacity of an ideal low-pass channel with white Gaussian noise 117 3.5 Capacity of a category of noisy dispersive channels 119 3.6 Effect of excess bandwidth 120 3.7 A hypothetical maximum-likelihood receiver 122 3.8 Noise whitening 125 3.9 Matched-filter bound 126 3.10 Equivalent discrete-time channel models 129 3.11 Performance of maximum-likelihood receivers 132 3.12 Further reading 137 APPENDIX 3A: Signal-to-noise ratios of NRZ and Bi-Phase on a Lorent-

zian channel 138 APPENDIX 3B: Optimum distribution of transmit power across frequency 142 APPENDIX 3C: Maximum-likelihood detection 143 APPENDIX 3D: Performance of the maximum-likelihood sequence de-

tector in the presence of colored Gaussian noise 144 PROBLEMS 147 REFERENCES 150

BASEBAND MODULATION CODES 153 4.1 Introduction 153 4.2 Intersymbol interference, Nyquist-l functions and eye patterns 155 4.3 Effect of modulation code on intersymbol interference 158 4.4 Effect of modulation code on signal-to-noise ratios 163 4.5 Effect of modulation code on timing recovery 166 4.6 Effect of modulation code on adaptation 170 4.7 Code mechanization 172 4.8 Run-length-limited codes 174 4.9 Modulation codes with spectral zeros 185 4.10 Linear modulation codes 195 4.11 Nonlinear codes with spectral zeros 201 4.12 Further reading 210 APPENDIX 4A: Analysis of baseline wander 211 APPENDIX 4B: Scramblers and pseudorandom sequences 212 APPENDIX 4C: Effect of coding on signal-to-noise ratios 215 PROBLEMS 220 REFERENCES 224

CONTENTS vii

5 LINEAR EQUALIZATION 231 5.1 Introduction 231

5.2 Zero-forcing linear equalizer 232

5.3 ZF LE with minimum noise enhancement 237

5.4 Minimum mean-square error linear equalizer 240

5.5 Linear equalization for run-length-limited codes 246

5.6 Resistance to channel parameter variations 249

5.7 Transmit filters 250

5.8 Implementation issues 253

5.9 Further reading 257

APPENDIX 5A: Effect of bias on receiver performance 257

APPENDIX 5B: Structure of the MMSE transmit filter 259

PROBLEMS 260

REFERENCES 263

6 PARTIAL-RESPONSE AND DECISION-FEEDBACK EQUALIZATION 265 6.1 Introduction 265

6.2 Partial-response equalization 266

6.3 Decision-feedback equalization 277 6.4 TomlinsonlHarashima precoding 283

6.5 In-loop noise whitening 286

6.6 Decision-feedback equalization for run-length-limited codes 287

6.7 Further reading 291 APPENDIX 6A: Derivation of the discrete-time MMSE DFE 291

APPENDIX 6B: MMSE FSE and DFE of finite length 292 APPENDIX 6C: Optimum d = 1 DFE 294

PROBLEMS 295 REFERENCES 298

7 VITERBI DETECTION 301 7.1 Introduction 301 7.2 Dynamic programming 302 7.3 Two-state Viterbi detection 304 7.4 Effect of transmission code and channel on trellis diagram 306 7.5 Characteristics of Viterbi detection 311 7.6 Channel memory truncation by means of prefiltering 315 7.7 Sequence feedback 324 7.8 Set partitioning 331 7.9 Restricted-delay detection 335

viii DIGITAL BASEBAND TRANSMISSION AND RECORDING

7.10 Further reading 344 APPENDIX 7 A: Viterbi detection for charge-constrained codes 344 APPENDIX 7B: Effects of a restricted detection delay 350 APPENDIX 7C: Performance of YD with prefilter 356 APPENDIX 7D: Performance of YD with sequence feedback 358 APPENDIX 7E: Exploitation of run-length constraints 360 PROBLEMS 363 REFERENCES 368

8 ADAPTIVE RECEPTION 373 8.1 Introduction 373 8.2 Structure of adaptive receivers 375 8.3 Automatic gain control 382 8.4 Non-data-aided adaptive equalization 398 8.5 Zero-forcing equalizer adaptation 400 8.6 Minimum mean-square error equalizer adaptation 412 8.7 Comparison of ZF and MMSE adaptation 420 8.8 Adaptive detection 421 8.9 Implementation example 427 8.10 Further reading 429 APPENDIX 8A: Analysis of the non-data-aided closed-loop AGe of

Fig. 8.9 430 APPENDIX 8B: Effect of a nonlinear voltage-controlled amplifier on adapt-

ation properties 432 APPENDIX 8C: Decoupled ZF adaptation 432 APPENDIX 8D: Behavior of single-tap ZF MMSI loop 433 APPENDIX 8E: Properties of the MMSE adaptation loop 435 PROBLEMS 438 REFERENCES 445

9 BASICS OF TIMING RECOVERY 451 9.1 Introduction 451 9.2 Timing-recovery schemes 452 9.3 Tracking 461

9.4 Acquisition 465

9.5 Aided acquisition 469 9.6 Acquisition limits 472

9.7 Further reading 476 APPENDIX 9A: All-digital timing-recovery scheme of Fig. 9.7 476

CONTENTS ix

APPENDIX 9B: Phase- and frequency-locked loops for a sinusoid embed-ded in noise 482

APPENDIX 9C: Data-aided maximum-likelihood one-shot phase estimator 489 APPENDIX 90: One-shot maximum-likelihood frequency estimator 492 PROBLEMS 494 REFERENCES 496

10 A CATALOG OF TIMING-RECOVERY SCHEMES 499 10.1 Introduction 499 10.2 Maximum-likelihood timing recovery 500 10.3 Timing recovery based on threshold crossings 508 10.4 Nonlinear spectral line methods 524 10.5 Early-late timing recovery 533 10.6 MMSE and sampled-derivative timing recovery 535 10.7 Zero-forcing timing recovery 544 APPENDIX lOA: Analysis of maximum-likelihood timing recovery 562 APPENDIX lOB: Analysis of timing recovery based on threshold crossings 568 APPENDIX 10C: Analysis of square-law timing recovery 571 APPENDIX 100: Analysis of minimum mean-square error timing recovery 573 APPENDIX WE: Analysis of zero-forcing timing recovery 575 PROBLEMS 577 REFERENCES 587

11 PHASE-LOCKED LOOPS 591 11.1 Introduction 591 11.2 PLL structure 591 11.3 Linear analysis of first-order PLL 594 11.4 Linear analysis of high-gain second-order PLL 602 11.5 Linear analysis of general second-order PLL 609 11.6 Nonlinear PLL behavior 610 11.7 Design example 612 11.8 Further reading 614 APPENDIX l1A: Equivalent discrete-time model of cascade of analog loop

filter and VCO 615 APPENDIX lIB: Effect of loop delay on stability 618 PROBLEMS 620 REFERENCES 623

INDEX 625

PREFACE

The advent of the information age has spurred the development of communication tech­nologies for digital transmission and storage of voice, audio, video and data signals. Application areas of these technologies range in bit rate from bits per second to gig­abits per second, and span such diverse media as radio links, telephone lines, coaxial cable, magnetic disk and tape, optical disc, and optical fiber. In all of these systems, data mayor may not be modulated onto a carrier wave before being transmitted or stored. The adjectives 'carrier-modulated' and 'baseband' are commonly used to distinguish between these two possibilities. Digital baseband transmission is used, for example, in narrowband ISDN, in optical fiber transmission, and in virtually all digital storage systems.1 Key ingredients are modulation coding, equalization, detection, adaptation and timing recovery. It IS the objective of this book to give a coherent, in-depth and up-to-date overview of these and related ingredients. The book is an outgrowth of an internal course at Philips Electronics, Eindhoven, The Netherlands. It is considerably broader in scope than books that focus primarily on a single transmission ingredient, like [6], [31], [21], [18], [10], [35], [16], [23], [28] and [24]. Even so the treatment extends beyond that of classical textbooks on digital communications, like [22], [25], [32], [1], [17], [19] and [15]. Special emphasis is given to transmission impairments that are characteristic of digital baseband transmission, such as linear and nonlinear in­tersymbol interference, random jitter of recorded transitions, and dynamic fluctuations of the time base and the characteristics of the transmission channel.

To the best of my knowledge this is the first book to deal exclusively with digital baseband transmission, and a considerable fraction of the presented material has not appeared in book form before. Some material (notably parts of Chapters 4, 7,8 and 9) appears here in print for the first time. Even so there is inevitably a certain overlap with the many excellent text books on information theory, coding, adaptation, synchroniza­tion and digital communication, like [1]-[37]. The reader is likely to benefit from prior exposure to one or more of these works. Conversely, much of the material presented here is probably also of interest to the 'carrier-modulated' engineering community.

An attempt has been made to make the material accessible to a wide audience. To this end, the development of the subject matter is almost entirely self-contained, and mathematical formalism is de-emphasized wherever possible in favour of an exposition of the underlying concepts. The reader is assumed to have an elementary background in mathematics and signal analysis, roughly at the level of an undergraduate engineering curriculum. Required knowledge beyond this level is summarized in Chapter 1 and concerns mainly certain aspects of spectral analysis. The presented material should be

lSince storage is merely the transportation of infonnation from one point in time to another, it is, in essence, also a fonn of transmission.

xi

xii DIGITAL BASEBAND TRANSMISSION AND RECORDING

of interest to students in electrical engineering as well as to practicing engineers. The entire book can be covered in a two-semester course. For a one-semester course it will be necessary to skip parts of most chapters in accordance with their relative importance.

Some subjects had to be skipped in order to restrict the size of the book to reasonable limits. Omitted subjects include channel characterization techniques, servo signal pro­cessing, word synchronization, error protection, and implementation issues. It is hard to over-estimate the practical importance of the latter topic and we would be remiss not to refer the reader to such excellent treatises as [15, Section 10.1] and [38, Chapter 10] (both on digital implementation) as well as [39], [40] and [41] (on analog implement­ation). References to these and other omitted topics are listed in the 'Further Reading' sections of each chapter. The reference lists are extensive and are as much as possible up to date.

Chapter 1 discusses the mathematical tools and concepts that are basic to later chapters. Covered are the temporal and spectral effects of filtering, sampling and modulation on both deterministic and random signals; probability density functions; the arithmetic, geometric and harmonic averages of amplitude spectra; properties of discrete-time minimum-phase functions; and constrained optimization via Lagrange multipliers. Prior knowledge that is assumed for this chapter as well as for the rest of the book is an elementary familiarity with calculus, linear systems theory, Fourier analysis, and probability and stochastic processes.

By way of introduction and motivation, Chapter 2 presents brief overviews of three application areas of digital baseband transmission, namely digital subscriber lines, di­gital magnetic recording, and digital optical recording. The treatment of transmission techniques in later chapters is somewhat biased towards these three areas, and in partic­ular towards digital recording. Appendices of the chapter are devoted to various types of bit-shifts that arise in the recording process.

Chapter 3 outlines the basic topology of a digital baseband transmission system, in­troduces elementary notions and terminology, and discusses several fundamental limits to transmission performance, such as channel capacity, the matched-filter bound, and the performance of the maximum-likelihood sequence detector.

Chapter 4 is devoted to baseband modulation codes. It discusses their raison d' etre and their impact on such transmission aspects as intersymbol interference, signal-to­noise ratios, timing recovery, and adaptation. It also discusses two important categories of codes, namely run-length-limited codes and codes that introduce one or more spectal zeros. The latter category is subdivided into linear and nonlinear codes. This distinction is not often made but has important consequences for receiver design and transmission performance. Fundamentals of these categories are described and for each of them a small catalog of worked examples is presented.

Chapters 5 and 6 deal with equalization. Equalization is a bare necessity -as op­posed to a minor IUxury- in many baseband transmission systems, and the depth and scope of our treatment reflect this importance. Chapter 5 is concerned with linear equal­ization subject to both zero-forcing and minimum mean-square error optimality cri­teria. Also addressed are the equalization of run-length-limited codes, the design of transmit filters, optimization of equalizer resistance to channel parameter variations.

PREFACE xiii

and implementation-related issues. Chapter 6 provides a similar coverage of partial­response and decision-feedback equalization. In both chapters, optimum equalizer per­formances are described as spectral averages of the folded signal-to-noise ratio of the channel and the spectrum of the encoded data. These canonical descriptions provide direct insight into performances of the equalizers vis a vis each other and the matched­filter bound, and into the impact of the modulation code on attainable performances.

Recent years have witnessed a rapidly growing interest in near-optimum reception techniques. Many of these are based in some way or another on the Viterbi detector (VD). Chapter 7 attempts to provide a cohesive overview of these techniques. After developing the basic form of the VD and exploring some of its characteristics, it dis­cusses and illustrates four approaches that are aimed at lowering VD complexity at a minor expense to performance. These approaches are based on prefiltering, sequence feedback, set partitioning, and restriction of the detection delay, respectively. They are largely complementary in nature and are often quite effective. Among the presented examples is Ferguson's detector for partial-response systems, which is widely used in digital magnetic recording. The VD first found its way into data transmission systems as an effective technique for dealing with intersymbol interference, and Chapter 7 fo­cuses primarily on this objective. A more recent and expanded objective involves ex­ploitation of any redundancy that is introduced by the modulation code. This topic is discussed in a more restricted fashion, with emphasis on run-length-limited codes and codes that introduce a zero at DC.

Many baseband transmission channels exhibit temporal or piece-by-piece vari­ations that must be dealt with adaptively in the data receiver. Chapter 8 provides an overview of these adaptation techniques. Emphasis is on such 'down-to-earth' topics as automatic gain and slope control, adaptive equalization with digital as well as analog equalizer structures, adaptation of various detector parameters, and more generally on adaptation techniques with a maximum simplicity. Instead of moving beyond the clas­sical LMS adaptation algorithm into the realm of 'advanced' adaptation schemes, the chapter provides an in-depth treatment of zero-forcing (ZF) adaptation, which is often preferred to LMS because of its simplicity.

loint adaptation of multiple parameters is most conveniently described in terms of matrix notation and some parts of Chapter 8, unlike all other chapters in this book, as­sume that the reader has an elementary knowledge of matrix theory.

The last three chapters of the book are concerned with timing recovery. The purpose of timing recovery is to define the instants at which decisions are taken. This task be­comes more difficult and at the same time more critical to transmission performance as the system becomes more efficient in terms, for example, of modulation code or band­width. This is one reason why classical schemes, in which the timing is derived directly from the received signal, are gradually being replaced by more powerful schemes of the data-aided type. The timing-recovery scheme usually takes the form of a delay-locked or phase-locked loop and at the beginning of transmission this loop must be brought into lock. In many instances the time for the loop to acquire lock is an undesirable overhead that should be minimized. Various techniques have evolved to accomplish this.

xiv DIGITAL BASEBAND TRANSMISSION AND RECORDING

The importance of timing recovery seems to warrant the extensive treatment in this book. Chapter 9 is concerned with basics. It presents a classification of timing­recovery schemes, discusses the impact of various system parameters on timing recov­ery, presents fundamental limits on the speed of acquisition, and discusses several ap­proaches to improve acquisition speed of a 'bare' scheme towards these limits.

Chapter 10 presents a rather extensive catalog of schemes. The catalog is sub­divided into several categories of schemes and to establish a baseline it first covers the optimum approach, called maximum-likelihood timing recovery. Further categories are presented in a more or less chronological order, progressing from classical ones to data­aided categories of the minimum mean-square error and zero-forcing type. The latter two categories are presented in such a manner as to expose their close relation to the adaptation techniques of Chapter 8. The catalog focuses on what sets distinct schemes apart, namely the timing-error detector which serves to extract timing-error informa­tion from the incoming signal. The common trait of most schemes is that they take the form of a discrete-time phase-locked loop (PLL). This trait is studied in Chapter 11. This final chapter discusses basic properties of the discrete-time PLL and relates them to those of the more usual continuous-time PLL.

Throughout the book, examples serve the dual purpose of illustrating the principal results and connecting the theory to practical applications. In all chapters the emphasis is on concepts rather than mathematics. For this reason some of the more mathemat­ical subjects have been relegated to appendices. Each chapter contains a problem sec­tion that is meant to deepen and expand the readers understanding of the subject matter. These sections are best regarded as an integral portion of the text.

The present book would not have been written without my long-term association with Philips Research, Eindhoven, The Netherlands, and without the influence and help of many friends and colleagues. In particular I would like to thank Bob Barnes, Steve Brittenham, Theo Claasen, Kevin Fisher, Charlie Gamble, Piet van Gerwen, Henk Holl­mann, Morishi Izumita, Guy Kerpen, Seiichi Mita, Saeed Rajput, Hans Voorman, Frans Willems, and HoWai Wong-Lam for interactions which have inspired parts of the ma­terial presented here, and Ad van den Enden, Henk Hollmann, Nuno Ramalho, Hugo Veenstra, Niek Verhoeckx, Jan van Vlerken, HoWai Wong-Lam and Peter van der Wurf for reviewing parts of the text. Also, I would like to express my gratitude to the manage­ment of Philips Research for giving me the opportunity and permission to publish this book. Finally, I would like to thank my Polish treasure for her support and endurance.

Jan W.M. Bergmans,

Warsaw, August 1995.

PREFACE xv

REFERENCES

[1] S. Benedetto, E. Biglieri, and V. Castellani, Digital Transmission Theory. Englewood Cliffs, NJ: Prentice-Hall, 1987.

[2] W.R. Bennett and J.R. Davey, Data Transmission. New York: McGraw-Hill, 1965. [3] J.AC. Bingham, Theory and Practice of Modem Design. New York: Wiley, 1988. [4] R.E. Blahut, Principles and Practice of Information Theory. Menlo Park, CA: Addison­

Wesley, 1987. [5] R.E. Blahut, Digital Transmission of Information. Menlo Park, CA: Addison-Wesley,

1990. [6] K.W. Cattennole, Principles of Pulse Code Modulation. London: Iliffe Books, 1969. [7] AP. Clark, Principles of Digital Data Transmission (2nd ed.). New York: Halstead Press

- Wiley, 1983. [8] AP. Clark, Adaptive Detectorsfor Digital Modems. London: Pentech Press, 1989. [9] T.M. Cover and J.A Thomas, Elements of Information Theory. New York: John Wiley,

1991. [to] C.EN. Cowan and P.M. Grant, Adaptive Filters. Englewood Cliffs, New Jersey: Prentice­

Hall,1985. [11] K. Feher, Digital Communications. Englewood Cliffs, NJ: Prentice-Hall, 1983. [12] K. Feher (ed.), Advanced Digital Communication Systems and Signal Processing Tech­

niques. Englewood Cliffs, N.J.: Prentice-Hall, 1987. [13] L.E. Franks, Signal Theory. Englewood Cliffs, NJ: Prentice-Hall, 1969. [14] R.G. Gallager, Information Theory and Reliable Communication. New York: John Wiley

& Sons, 1968. [15] R.D. Gitlin, S.B. Weinstein, and I.E Hayes, Data Communication Principles. New York:

Plenum, 1991. [16] S. Haykin, Adaptive Filter Theory. Englewood Cliffs, New Jersey: Prentice-Hall, 1986. [17] S. Haykin, Digital Communications. New York: Wiley, 1988. [18] M.L. Honig and D.G. Messerschmitt, Adaptive Filters: Structures, Algorithms, and Ap-

plications. Boston: Kluwer Academic Publishers, 1984. [19] E.A. Lee and D.G. Messerschmitt, Digital Communications. Boston, MA: Kluwer, 1988. [20] Y.W. Lee, Statistical Theory of Communication. New York: Wiley, 1960. [21] W.C. Lindsey, Synchronization Systems in Communications. Englewood Cliffs, NJ:

Prentice-Hall, 1972. [22] R.W. Lucky, I. Salz, and EJ. Weldon, Jr., Principles of Data Communication. New York:

McGraw-Hill, 1968. [23] H. Meyr and G. Ascheid, Synchronization in Digital Communications - Volume I. New

York: Wiley, 1990. [24] J. Moon and L.R. Carley, Sequence Detection for High-Density Storage Channels.

Dordrecht: Kluwer Academic Publishers, 1992. [25] J.G. Proakis, Digital Communications. New York: McGraw-Hill, 1983 (2nd ed. 1989).

xvi DIGITAL BASEBAND TRANSMISSION AND RECORDING

[26] J .G. Proakis and J .A. Salehi, Communication Systems Engineering. Englewood Cliffs, NJ: Prentice-Hall, 1994.

[27] M.S. Roden, Digital and Data Communication Systems. Englewood-Cliffs, NJ: Prentice­Hall,1982.

[28] K.A Schouhamer Immink, Coding Techniques for Digital Recorders. Englewood Cliffs, NJ: Prentice-Hall, 1991.

[29] M. Schwartz, Information Transmission, Modulation, and Noise. New York: McGraw­Hill, 1959 (4th ed. 1990).

[30] K.S. Shanmugan, Digital and Analog Communication Systems. New York: Wiley, 1979. [31] J.J. Stiffler, Theory of Synchronous Communications. Englewood Cliffs, NJ: Prentice­

Hall,1971. [32] H. Taub and D.L. Schilling, Principles of Communication Systems. New York: McGraw­

Hill,1986. [33] AJ. Viterbi, Principles of Coherent Communications. New York: McGraw-Hill, 1966. [34] A.I. Viterbi and J.K. Omura, Principles of Digital Communication and Coding. New York:

McGraw-Hill, 1979. [35] B. Widrow and S.D. Stearns, Adaptive Signal Processing. Englewood Cliffs, New Jersey:

Prentice-Hall, 1985. [36] J.M. Wozencraft and I.M. Jacobs, Principles of Communication Engineering. New York:

Wiley, 1965. [37] R.E. Ziemer and W. Peterson, Digital Communications and Spread Spectrum. New York:

MacMillan, 1985. [38] A W.M. van den Enden and N .AM. Verhoeckx, Discrete-TIme Signal Processing -An In­

troduction. Hemel Hempstead, Hertfordshire (UK): Prentice-Hall, 1989. [39] P.R. Gray and R.G. Meyer, Analysis and Design of Analog Integrated Circuits. New York:

John Wiley & Sons, 1977 (2nd ed. 1984). [40] R. Gregorian and G.C. Ternes, Analog MOS Integrated Circuits for Signal Processing.

New York: John Wiley & Sons, 1986. [41] Y.P. Tsividis and J.~. Voorman (Eds.), Integrated Continuous-Time Filters. New York:

IEEE Press, 1993.