rill--- -

"I have used Numerical Linear Algebra in my introductory graduate course and I have found it to be almost the perfect text to introduce mathematics graduate students to the subject. I like the choice of topics and the format: a sequence of lectures. Each chapter (or lecture) carefully builds upon the material presented in previous chapters, providing new concepts in a very clear manner. Exercises at the end of each chapter reinforce the concepts, and in some cases introduce new ones. ...The emphasis is on the mathematics behind the algorithms, in the under-standing of why the algorithms work. ...The tent is sprinkled with examples and explanations, which keep the student focused."

Daniel B. Szyld, Department of Mathematics, Temple University

"A beautifully written textbook offering a distinctive and original treatment. It will be of use to all who teach or study the subject."

Nicholas J. Higham, Professor of Applied Mathematics, University of Manchester

"...this is an ideal book for a graduate course in numerical linear algebra (either in mathematics or in computer science departments); it presents the topics in such a way that background material comes along with the course. ...I will use it again next time I teach this course!"

Suely Oliveira, Texas A&M University

This is a concise, insightful, and elegant introduction to the field of numerical linear algebra. Designed for use as a stand-alone textbook in a one-semester, graduate-level course in the topic, it has already been class-tested by MIT and Cornell graduate students from all fields of mathematics, engineering, and the physical sciences. The authors' clear, inviting style and evident love of the field, along with their eloquent presentation of the most fundamental ideas in numerical linear algebra, have made it popular with teachers and students alike.

Numerical Linear Algebra aims to expand the reader's view of the field and to present the core, standard material in a novel way. This makes it a perfect companion volume to the encyclopedic treatment of the topic that already exists in Golub and Van Loan's now-classic Matrix Computa-tions. All of the most important topics in the field, including iterative methods for systems of equations and eigenvalue problems and the underlying principles of conditioning and stability, are covered. Trefethen and Bau offer a fresh perspective on these and other topics, such as an emphasis on connections with polynomial approximation in the complex plane.

Numerical Linear Algebra is presented in the form of 40 lectures, each of which focuses on one or two central ideas. Throughout, the authors emphasize the unity between topics, never allow-ing the reader to get lost in details and technicalities. The book breaks with tradition by begin-ning not with Gaussian elimination, but with the OR factorization a more important and fresher idea for students, and the thread that connects most of the algorithms of numerical linear algebra, including methods for least squares, eigenvalue, and singular value problems, as well as iterative methods for all of these and for systems of equations.

Lloyd N. Trefethen is a Professor of Computer Science at Cornell University. He has won teach-ing awards at both MIT and Cornell. In addition to editorial-positions on such journals as SIAM Journal on Numerical Analysis, Journal of Computational and Applied Mathematics, Numerische Mathematik, and SIAM Review, he has been an invited lecturer at two dozen international conferences. While at Cornell, David Bau was a student of Trefethen. He is currently a Soft-ware Developer at Microsoft Corporation, where he works in the Internet Division.

Man. Society for Industrial and Applied Mathematics

3600 University City Science Center Philadelphia, PA 19104-2688

ISBN 0-89871-361-7

Telephone: 215-382-9800 Fax: 215-386-7999

siam @slam. org http://wwwsiam.org

1101031111111111110111 9 78 898"713619"

BKOT0050

Notation

For square or rectangular matrices A E Cm", m > n:

QR factorization: A = QR

Reduced QR factorization: A = Qi?

SVD: A = UEV*

Reduced SVD: A = UEV*

For square matrices A E Cm x In :

LU factorization: PA = LU

Cholesky factorization: A= R*R

Eigenvalue decomposition: A = XAX-1

Schur factorization: A = UT U*

Orthogonal projector: P = QQ*

Householder reflector: F = I 2 vv* v*v

QR algorithm: Ak = Q(k)R(k), A(k) = (Q(k))T AQ(k)

Arnoldi iteration: AQ.= Qn+ifla, H.= Q:AQ.

Lanczos iteration: AQ.= Q.+it., T.= QT.AQ.

NUMERICAL LINEAR ALGEBRA

NUMERICAL LINEAR ALGEBRA LLOYD N. TREFETHEN Cornell University Ithaca, New York

DAVID BAU3 III Microsoft Corporation Redmond,Washington

Society for Industrial and Applied Mathematics

Philadelphia

Copyright 1997 by the Society for Industrial and Applied Mathematics.

10 9 8 7 6 5 4 3 2

All rights reserved. Printed in the United States of America. No part of this book may be repro-duced, stored, or transmitted in any manner without the written permission of the publisher. For information, write to the Society for Industrial and Applied Mathematics, 3600 University City Science Center, Philadelphia, PA 19104-2688.

Trademarked names may be used in this book without the inclusion of a trademark symbol. These names are used in an editorial context only; no infringement of trademark is intended.

Library of Congress Cataloging-in-Publication Data

Trefethen, Lloyd N. (Lloyd Nicholas) Numerical linear algebra / Lloyd N. Trefethen, David Bau III.

p. cm. Includes bibliographical references and index. ISBN 0-89871-361-7 (pbk.) 1. Algebras, Linear. 2. Numerical calculations. I. Bau, David.

II. Title. QA184.T74 1997 512'.5--dc21 96-52458

Cover Illustration. The four curves reminiscent of water drops are polynomial lemniscates in the complex plane associated with steps 5,6,7,8 of an Amoldi iteration. The small dots are the eigenval-ues of the underlying matrix A, and the large dots are the Ritz values of the Arnoldi iteration. As the iteration proceeds, the lemniscate first reaches out to engulf one of the eigenvalues X, then pinches off and shrinks steadily to a point. The Ritz value inside it thus converges geometrically to X. See Figure 34.3 on p. 263.

Man= is a registered trademark.

To our parents Florence and Lloyd MacG. Trefethen

and Rachel and Paul Bau

Contents

Preface ix

Acknowledgments xi

I Fundamentals 1 Lecture 1 Matrix-Vector Multiplication 3 Lecture 2 Orthogonal Vectors and Matrices 11 Lecture 3 Norms 17 Lecture 4 The Singular Value Decomposition 25 Lecture 5 More on the SVD 32

II QR Factorization and Least Squares 39 Lecture 6 Projectors 41 Lecture 7 QR Factorization 48 Lecture 8 GramSchmidt Orthogonalization 56 Lecture 9 MATLAB 63 Lecture 10 Householder Triangularization 69 Lecture 11 Least Squares Problems 77

III Conditioning and Stability 87 Lecture 12 Conditioning and Condition Numbers 89 Lecture 13 Floating Point Arithmetic 97 Lecture 14 Stability 102 Lecture 15 More on Stability 108 Lecture 16 Stability of Householder Triangularization 114 Lecture 17 Stability of Back Substitution 121 Lecture 18 Conditioning of Least Squares Problems 129 Lecture 19 Stability of Least Squares Algorithms 137

vu

vii'

CONTENTS

IV Systems of Equations 145 Lecture 20 Gaussian Elimination 147 Lecture 21 Pivoting 155 Lecture 22 Stability of Gaussian Elimination 163 Lecture 23 Cholesky Factorization 172

V Eigenvalues 179 Lecture 24 Eigenvalue Problems 181 Lecture 25 Overview of Eigenvalue Algorithms 190 Lecture 26 Reduction to Hessenberg or Tridiagonal Form 196 Lecture 27 Rayleigh Quotient, Inverse Iteration 202 Lecture 28 QR Algorithm without Shifts 211 Lecture 29 QR Algorithm with Shifts 219 Lecture 30 Other Eigenvalue Algorithms 225 Lecture 31 Computing the SVD 234

VI Iterative Methods 241 Lecture 32 Overview of Iterative Methods 243 Lecture 33 The Arnoldi Iteration 250 Lecture 34 How Arnoldi Locates Eigenvalues 257 Lecture 35 GMRES 266 Lecture 36 The Lanczos Iteration 276 Lecture 37 From Lanczos to Gauss Quadrature 285 Lecture 38 Conjugate Gradients 293 Lecture 39 Biorthogonalization Methods 303 Lecture 40 Preconditioning 313

Appendix The Definition of Numerical Analysis 321

Notes 329

Bibliography 343

Index 353

Preface

Since the early 1980s, the first author has taught a graduate course in nu-merical linear algebra at MIT and Cornell. The alumni of this course, now numbering in the hundreds, have been graduate students in all fields of engi-neering and the physical sciences. This book is an attempt to put this course on paper.

In the field of numerical linear algebra, there is already an encyclopedic treatment on the market: Matrix Computations, by Golub and Van Loan, now in its third edition. This book is in no way an attempt to duplicate that one. It is small, scaled to the size of one university semester. Its aim is to present fundamental ideas in as elegant a fashion as possible. We hope that every reader of this book will have access also to Golub and Van Loan for the pursuit of further details and additional topics, and for its extensive references to the research literature. Two other important recent books are those of Higham and Demmel, described in the Notes at the end (p. 329).

The field of numerical linear algebra is more beautiful, and more funda-mental, than its rather dull name may suggest. More beautiful, because it is full of powerful ideas that ar