ActaNumerica(2005), pp. 1137 cCambridgeUniversityPress,2005DOI:10.1017/S0962492904000212 PrintedintheUnitedKingdomNumericalsolutionofsaddlepointproblemsMicheleBenziDepartment of Mathematics and Computer Science,Emory University, Atlanta, Georgia 30322, USAE-mail: benzi@mathcs.emory.eduGeneH.GolubScientic Computing and Computational Mathematics Program,Stanford University, Stanford,California 94305-9025, USAE-mail: golub@sccm.stanford.eduJorgLiesenInstitut f ur Mathematik, Technische Universit at Berlin,D-10623 Berlin, GermanyE-mail: liesen@math.tu-berlin.deWe dedicate this paper to Gil Strang on the occasion of his 70th birthdayLarge linear systems of saddle point type arise in a wide variety of applica-tions throughout computational science and engineering. Due to their indef-initeness and often poor spectral properties, such linear systems represent asignicantchallengeforsolverdevelopers. Inrecentyearstherehasbeenasurge of interest in saddle point problems, and numerous solution techniqueshavebeenproposedforthistypeof system. Theaimof thispaperistopresentanddiscussalargeselectionofsolutionmethodsforlinearsystemsinsaddlepointform, withanemphasisoniterativemethodsforlargeandsparse problems.SupportedinpartbytheNationalScienceFoundationgrantDMS-0207599.SupportedinpartbytheDepartmentofEnergyoftheUnitedStatesGovernment.Supportedinpart by the Emmy Noether Programmof the Deutsche Forschungs-gemeinschaft.2 M.Benzi,G.H.GolubandJ.LiesenCONTENTS1 Introduction 22 Applications leading to saddle point problems 53 Properties of saddle point matrices 144 Overview of solution algorithms 295 Schur complement reduction 306 Null space methods 327 Coupled direct solvers 408 Stationary iterations 439 Krylov subspace methods 4910Preconditioners 5911Multilevel methods 9612Available software 10513Concluding remarks 107References 1091. IntroductionInrecentyears, alargeamountofworkhasbeendevotedtotheproblemof solvinglargelinearsystemsinsaddlepointform. Thereasonforthisinterestisthefactthatsuchproblemsariseinawidevarietyoftechnicalandscienticapplications. Forexample, theever-increasingpopularityofmixedniteelementmethodsinengineeringeldssuchasuidandsolidmechanicshasbeenamajorsourceof saddlepointsystems(Brezzi andFortin 1991, Elman, Silvester and Wathen 2005c). Another reason for thissurgeininterestistheextraordinarysuccessof interiorpointalgorithmsin both linear and nonlinear optimization, which require at their heart thesolution of a sequence of systems in saddle point form (Nocedal and Wright1999, Wright 1992, Wright 1997).Becauseoftheubiquitousnatureofsaddlepointsystems,methodsandresults on their numerical solution have appeared in a wide variety of books,journalsandconferenceproceedings, justifyingacomprehensivesurveyofthe subject. The purpose of this article is to review many of the most prom-ising solution methods, with an emphasis on iterative methods for large andsparse problems. Although many of these solvers have been developed withspecicapplicationsinmind(forexample, Stokes-typeproblemsinuiddynamics), itispossibletodiscusstheminafairlygeneral settingusingstandard numerical linear algebra concepts, the most prominent being per-haps the Schur complement. Nevertheless, when choosing a preconditioner(or developing a new one), knowledge of the origin of the particular problemNumericalsolutionofsaddlepointproblems 3at hand is essential. Although no single best method exists, very eectivesolvershavebeendevelopedforsomeimportantclassesof problems. Wetherefore devote some space to a discussion of saddle point problems arisingin a few selected applications.It is hoped that the present survey will prove useful to practitioners whoarelookingforguidanceinthechoiceofasolutionmethodfortheirownapplication, toresearchersinnumerical linearalgebraandscienticcom-puting, and especially to graduate students as an introduction to this veryrich and important subject.Notation. Wehaveusedboldfacetodenotevectorsinsectionsdescribinguid dynamics applications, where this is traditional, but we have otherwisefollowed the standard practice of numerical linear algebra and employed nospecial notation.1.1. Problem statement and classicationThe subject of this paper is the solution of block 2 2 linear systems of theform _A BT1B2C_ _xy_ =_fg_, or /u = b, (1.1)A Rnn, B1,B2 Rmn, C Rmmwith n m. (1.2)It is obvious that, under suitable partitioning, any linear system can be castintheform(1.1)(1.2). WeexplicitlyexcludethecasewhereAoroneorboth ofB1,B2are zero. When the linear system describes a (generalized)saddle point problem,the constituent blocksA, B1, B2andCsatisfy oneor more of the following conditions:C1 A is symmetric: A = ATC2 the symmetric partofA,H 12(A+AT), is positive semideniteC3 B1 = B2 = BC4 Cis symmetric (C = CT) and positive semideniteC5 C = O (the zero matrix)Notethat C5implies C4. Themost basiccaseis obtainedwhenalltheaboveconditions aresatised. Inthis case Ais symmetricpositivesemidenite and we have a symmetriclinear system of the form_A BTB O_ _xy_ =_fg_. (1.3)This system arises as the rst-order optimality conditions for the followingequality-constrained quadratic programming problem:minJ(x) =12xTAx fTx (1.4)subject to Bx = g. (1.5)4 M.Benzi,G.H.GolubandJ.LiesenInthiscasethevariableyrepresentsthevectorof Lagrangemultipliers.Any solution (x, y) of (1.3) is a saddle point for the LagrangianL(x, y) =12xTAx fTx + (Bx g)Ty,hence the name saddle point problem given to (1.3). Recall that a saddlepoint is a point (x, y) Rn+mthat satisesL(x, y) L(x, y) L(x, y) for any x Rnand y Rm,or, equivalently,minxmaxyL(x, y) = L(x, y) = maxyminxL(x, y).Systemsoftheform(1.3)alsoariseinnonlinearlyconstrainedoptimiz-ation(sequential quadraticprogrammingandinteriorpointmethods), inuid dynamics (Stokes problem), incompressible elasticity, circuit analysis,structural analysis, andsoforth; seethenextsectionforadiscussionofapplications leading to saddle point problems.AnotherimportantspecialcaseiswhenconditionsC1C4aresatised,but not C5. In this case we have a block linear system of the form_A BTB C_ _xy_ =_fg_. (1.6)Problems of this kind frequently arise in the context of stabilizedmixed -nite element methods. Stabilization is used whenever the discrete variables xand y belong to nite element spaces that do not satisfy the LadyzhenskayaBabuskaBrezzi (LBB, or inf-sup) condition (Brezzi and Fortin 1991). An-other situation leading to a nonzero C is the discretization of the equationsdescribing slightly compressible uids or solids (Braess 2001, Chapter 6.3).Systems of the form (1.6) also arise from regularized, weighted least-squaresproblems (Benzi and Ng 2004) and from certain interior point methods inoptimization(Wright1992, Wright1997). OftenthematrixChassmallnorm compared to the other blocks.Intheliterature, thephrasegeneralizedsaddlepoint problemhasbeenusedprimarilytoallowforthepossibilityofanonsymmetricmatrix /in(1.1). In such problems either A ,= AT(with condition C2 usually satised),orB1 ,=B2, or both. The most important example is perhaps that of thelinearizedNavierStokesequations, wherelinearizationhasbeenobtainedby Picard iteration or by some variant of Newtons method. See Ciarlet Jr.,HuangandZou(2003), Nicolaides(1982)andSzyld(1981)foradditionalexamples. We note that our denition of generalized saddle point problemas a linear system of the form (1.1)(1.2), where the blocksA,B1,B2andC satisfy one or more of the conditions C1C5, is the most general possible,and it contains previous denitions as special cases.Numericalsolutionofsaddlepointproblems 5Inthevastmajorityof cases, linearsystemsof saddlepointtypehavereal coecients, andinthispaperwerestrictourselvestothereal case.Complexcoecient matrices, however, doariseinsomecases; see, e.g.,BobrovnikovaandVavasis(2000), MahawarandSarin(2003)andStrang(1986, page 117). Most of the results and algorithms reviewed in this paperadmit straightforward extensions to the complex case.1.2. Sparsity, structure and sizeAlthoughsaddlepointsystemscomeinallsizesandwithwidelydierentstructural andsparsityproperties, inthispaperwearemainlyinterestedinproblemsthatarebothlargeandsparse. Thisjustiesouremphasison iterative solvers. Direct solvers, however, are still the preferred methodinoptimizationandotherareas. Furthermore, directmethodsareoftenused in the solution of subproblems, for example as part of a preconditionersolve. Some of the algorithms considered in this paper are also applicableifoneormoreoftheblocksin /happentobedense, aslongasmatrix-vector products with / can be performed eciently, typically inO(n +m)time. Thismeansthatifadenseblockispresent, itmusthaveaspecialstructure(e.g., Toeplitz, asinBenziandNg(2004)andJin(1996))oritmust be possible to approximate its action on a vector with (nearly) linearcomplexity, as in the fast multipole method (Mahawar and Sarin 2003).Frequently,the matrices that arise in practice have quite a bit of struc-ture. For instance, theA block is often block diagonal, with each diagonalblock endowed with additional structure. Many of the algorithms discussedinthispaperareabletoexploitthestructureoftheproblemtogaine-ciency and save on storage. Sometimes the structure of the problem suggestssolution algorithms that have a high degree of parallelism. This last aspect,however, is not emphasized in this paper. Finally we mention that in mostapplicationsn is larger thanm, often much larger.2. ApplicationsleadingtosaddlepointproblemsAs alreadymentioned, large-scalesaddlepoint problems occur inmanyareasofcomputationalscienceandengineering. Thefollowingisalistofsome elds where saddle point problems naturally arise, together with somereferences:computational uiddynamics(Glowinski 1984, Quarteroni andValli1994, Temam 1984, Turek 1999, Wesseling 2001)constrained and weighted least squares estimation (Bj orck 1996, Goluband Van Loan 1996)constrained optimization (Gill, Murray and Wright 1981, Wright 1992,Wright 1997)6 M.Benzi,G.H.GolubandJ.Lieseneconomics (Arrow, Hurwicz and Uzawa 1958, Duchin and Szyld 1979,Leontief, Duchin and Szyld 1985, Szyld 1981)electrical circuits and networks (Bergen 1986, Chua, Desoer and Kuh1987, Strang 1986, Tropper 1962)electromagnetism(Bossavit 1998, Perugia1997, Perugia, Simonciniand Arioli 1999)nance (Markowitz 1959, Markowitz and Perold 1981)image reconstruction (Hall 1979)image registration (Haber and Modersitzki 2004, Modersitzki 2003)interpolation of scattered data (Lyche, Nilssen and Winther 2002, Sib-son and Stone 1991)linear elasticity (Braess 2001, Ciarlet 1988)meshgenerationfor computer graphics (Liesen, deSturler, Sheer,Aydin and Siefert 2001)mixedniteelement approximations of ellipticPDEs (Brezzi 1974,Brezzi and Fortin 1991, Quarteroni and Valli 1994)model order reduction for dynamical systems (Freund 2003, Heres andSchilders 2005, Stykel 2005)optimal control (BattermannandHeinkenschloss 1998, Battermannand Sachs 2001, Betts 2001, Biros and Ghattas 2000, Nguyen 2004)parameter identication problems (Burger and M uhlhuber 2002, Haberand Ascher 2001, Haber, Ascher and Oldenburg 2000).Quiteoften, saddlepointsystemsarisewhenacertainquantity(suchastheenergyofaphysicalsystem)hastobeminimized, subjecttoasetoflinear constraints. Inthis casetheLagrangemultiplier yusuallyhas aphysical interpretation and its computation is also of interest. For example,in incompressible ow problemsx is a vector of velocities andy a vector ofpressures. In the complementary energy formulation of structural mechanicsx is the vector of internal forces, y represents the nodal displacements of thestructure. For resistive electrical networks y represents the nodal potentials,x being the vector of currents.Insomecases, suchasuiddynamicsorlinearelasticity, saddlepointproblems result fromthe discretizationof systems of partial dierentialequationswithconstraints. Typicallytheconstraintsrepresentsomeba-sic conservation law, such as mass conservation in uid dynamics. In othercases, such as resistive electrical networks or structural analysis, the equa-tions are discrete to begin with. Now the constraints may correspond to thetopology (connectivity) of the system being studied. Because saddle pointequationscanbederivedasequilibriumconditionsforaphysical system,they are sometimes called equilibrium equations. See Strang (1986, 1988) foraverynicediscussionofequilibriumequationsthroughoutappliedmath-ematics. Another popular name for saddle point systems, especially in theNumericalsolutionofsaddlepointproblems 7optimizationliterature, isKKTsystem, fromtheKarushKuhnTuckerrst-order optimality conditions; see Nocedal and Wright (1999, page 328)for precise denitions, and Golub and Greif (2003) and Kjeldsen (2000) forhistorical notes.Systemsoftheform(1.1)(1.2)alsoarisefromnon-overlappingdomaindecomposition when interface unknowns are numbered last, as well as fromFETI-type schemes when Lagrange multipliers are used to ensure continuityattheinterfaces; seeforinstanceChanandMathew(1994), FarhatandRoux (1991), Hu, Shi and Yu (2004), Quarteroni and Valli (1999) and Toselliand Widlund (2004).Itisof coursenotpossibleforustocoverhereall thesedierentap-plications. Wechooseinsteadtogivesomedetailsaboutthreeclassesofproblemsleadingtosaddlepointsystems. Therstcomesfromtheeldof computational uid dynamics, the second from least squares estimation,and the third one from interior point methods in constrained optimization.2.1. Incompressible ow problemsWebeginwiththe(steady-state)NavierStokesequationsgoverningtheow of a Newtonian, incompressible viscous uid. Let Rd(d = 2, 3) bea bounded, connected domain with a piecewise smooth boundary . Givena force eld f: Rdand boundary data g : Rd, the problem is tond a velocity eld u : Rdand a pressure eldp : R such thatu + (u) u +p = f in , (2.1) u = 0 in , (2.2)Bu = g on , (2.3)where> 0 is the kinematic viscosity coecient (inversely proportional tothe Reynolds number Re), is the Laplace operator in Rd, denotes thegradient, isthedivergence, and Bissometypeof boundaryoperator(e.g.,a trace operator for Dirichlet boundary conditions). To determinepuniquely we may impose some additional condition, such as_p dx = 0.Equation (2.1) represents conservation of momentum, while equation (2.2)represents the incompressibility condition, or mass conservation. Owing tothe presence of the convective term (u) u in the momentum equations,the NavierStokes system is nonlinear. It can be linearized in various ways.Anespeciallypopular linearizationprocess is theonebasedonPicardsiteration; see, e.g., Elmanet al. (2005c, Section7.2.2). Startingwithaninitial guess u(0)(with u(0)= 0) for the velocity eld, Picards iterationconstructsasequenceof approximatesolutions(u(k), p(k))bysolvingthe8 M.Benzi,G.H.GolubandJ.Liesenlinear Oseen problemu(k)+ (u(k1) ) u(k)+p(k)= f in , (2.4) u(k)= 0 in , (2.5)Bu(k)= g on (2.6)(k=1, 2, . . .). Notethatnoinitialpressureneedstobespecied. Undercertain conditions on (which should not be too small) and f(which shouldnot be too large in an appropriate norm),the steady NavierStokes equa-tions (2.1)(2.3) have a unique solution (u, p) and the iterates (u(k), p(k))convergetoitas k foranychoiceof theinitial velocityu(0). Werefer toGirault andRaviart (1986) for existenceanduniqueness resultsand to Karakashian (1982) for a proof of the global convergence of Picardsiteration.Hence,at each Picard iteration one needs to solve an Oseen problem ofthe formu + (v) u +p = f in , (2.7) u = 0 in , (2.8)Bu = g on (2.9)withaknown, divergence-freecoecientv. Discretizationof (2.7)(2.9)using, e.g., nitedierences(PeyretandTaylor1983)orniteelements(Elmanet al. 2005c, Quarteroni andValli 1994)resultsinageneralizedsaddlepointsystemof theform(1.6), inwhichxrepresentsthediscretevelocities and y the discrete pressure. Here A = diag(A1, . . . , Ad) is a blockdiagonal matrix, whereeachblockcorresponds toadiscreteconvection-diusion operator with the appropriate boundary conditions. Note thatAisnonsymmetric, butsatisesconditionC2whenanappropriate(conser-vative)discretizationisused. TherectangularmatrixBTrepresentsthediscrete gradient operator whileBrepresents its adjoint, the (negative) di-vergence operator. A nonzeroCmay be present if stabilization is used.The important special case v = 0 corresponds to the (steady-state) Stokesequations:u +p = f in , (2.10) u = 0 in , (2.11)Bu = g on . (2.12)Notethatwithoutlossof generalitywehaveset=1, sincewecanal-waysdividethemomentumequationbyandrescalethepressurepandtheforcingtermf by. TheStokesequationscanbeinterpretedastheEulerLagrange partial dierential equations for the constrained variationalNumericalsolutionofsaddlepointproblems 9problemminJ(u) =12_|u|22 dx _fudx (2.13)subject to u = 0 (2.14)(see, e.g., GreshoandSani (1998, page636)). Throughout this paper,|u|2 = uu denotes the Euclidean norm of the vector u. Here the pres-sure pplays theroleof theLagrangemultiplier. TheStokes equationsdescribetheowof aslow-moving, highlyviscousuid. Theyalsoariseas subproblems in the numerical solution of the NavierStokes equations byoperator splitting methods (Glowinski 2003, Quarteroni and Valli 1994) andas the rst step of Picards iteration when the initial guess used is u(0)= 0.Appropriate discretizationof the Stokes systemleads toasymmetricsaddlepointproblemof theform(1.3)whereAisnowablockdiagonalmatrix, and each of itsd diagonal blocks is a discretization of the Laplaceoperator withtheappropriateboundaryconditions. Thus, Aisnowsymmetric and positive (semi-)denite. Again, a nonzero C may be presentif stabilizationisused. Typical sparsitypatternsfor /aredisplayedinFigure 2.1.An alternative linearization of the NavierStokes equations can be derivedon the basis of the identity(u)u =12(|u|22) u (u).0 100 200 300 400 500 600 700 8000100200300400500600700800nz = 74260 100 200 300 400 500 600 700 8000100200300400500600700800nz = 8194(a) without stabilization (b) with stabilizationFigure 2.1. Sparsity patterns for two-dimensional Stokes problem(leaky lid-driven cavity) using Q1-P0 discretization.10 M.Benzi,G.H.GolubandJ.LiesenSee, for instance, Landau and Lifschitz (1959, page 5) or Chorin and Marsden(1990, page 47) as well as the discussion in Gresho (1991) and Gresho andSani (1998). The corresponding linearized equations take the formu +wu +P= f in , (2.15) u = 0 in , (2.16)Bu = g on , (2.17)where, for the two-dimensional case,(w) = _0 ww 0_ w = v = v1x2+v2x1 P= p +12|v|22 (the so-called Bernoulli pressure)Herethedivergence-freevectoreldvagaindenotestheapproximateve-locityfromthepreviousPicarditeration. SeeOlshanskii (1999)forthethree-dimensional case. Notethatwhenthewind functionvisirrota-tional (v=0), equations (2.15)(2.17) reducetotheStokes prob-lem. Itisworthstressingthatthelinearizations(2.7)(2.9)and(2.15)(2.17), althoughbothconservative(Olshanskii 1999, page357), arenotmathematicallyequivalent. Theso-calledrotationform(2.15)ofthemo-mentum equations, although popular in uid mechanics, has not been widelyknownamongnumericalanalystsuntiltherecentworkbyOlshanskiiandco-workers (Lube andOlshanskii 2002, Olshanskii 1999, Olshanskii andReusken2002), whichshoweditsadvantagesoverthestandard(convect-ive) form. We return to this in Section 10.3.Arelatedproblem, alsoleadingtolargesparselinearsystemsinsaddlepointformupondiscretization,isthepotentialuidowprobleminpor-ousmedia, oftenusedtomodel groundwater contamination(Bear 1972,Maryska, RozloznkandTuma1995). Thisconsistsof aboundaryvalueproblem for a system of rst-order partial dierential equations represent-ing, respectively, DarcysLawforthevelocityelduandthecontinuityequation:Ku +p = 0 in , (2.18) u = q in , (2.19)p = pDon D, un = uNon N, (2.20)wherep is a piezometric potential (uid pressure),Kis the symmetric anduniformly positive denite second-rank tensor of hydraulic permeability ofthe medium, andq represents density of potential sources (or sinks) in themedium. HereDandNaresubsetsoftheboundaryoftheboundedconnected ow domain , with = DN, D ,= , and DN= ; n isthe outward normal vector dened (a.e.)on . When discretized by mixedNumericalsolutionofsaddlepointproblems 11nite elements (RaviartThomas elements being a very popular choice forthis problem), a linear system of the type (1.3) is obtained. The symmet-ric positive denite matrixA is now a discretization of the linear operatormultiplicationbyK,a zeroth-order dierential operator. The condition-ing properties of A are independent of the discretization parameterh (formost discretizations), and depend only on properties of the hydraulic per-meabilitytensor K. Thematrix Brepresents, again, adiscretediver-genceoperatorandBTadiscretegradient. Wenotethatmodellingtheinteraction between surface and subsurface ows leads to coupled (Navier)StokesandDarcysystems(Discacciati, MiglioandQuarteroni 2002, Dis-cacciatiandQuarteroni2004). Problem(2.18)(2.20)isjustoneexampleof arst-order systemformulationof asecond-order linear ellipticPDE(Brezzi andFortin1991). Saddle point systems alsoarise frommixedformulationoffourth-order(biharmonic)ellipticproblems(GlowinskiandPironneau 1979).Inthe course of this brief discussionwe have restrictedourselves tostationary(steady-state)problems. Theunsteady(time-dependent)caseleads to sequences of saddle point systems when fully implicit time-steppingschemesareused, forexamplewhenthetimederivativeutisdiscretizedusingbackwardEulerorCrankNicolsonschemes; see, e.g., Turek(1999,Chapter 2). In the case of Stokes and Oseen, the resulting semi-discrete sys-tems are often referred to as generalizedStokes and Oseen problems. Theliteratureonnumerical methodsforincompressibleowproblemsisvast;see, e.g., Elmanet al. (2005c), Fortin(1993), Glowinski (2003), Greshoand Sani (1998), Gunzburger (1989), Quarteroni and Valli (1994), Temam(1984), Turek (1999) and Wesseling (2001).2.2. Constrained and weighted least squaresLinear systems of saddle point type commonlyarise whensolvingleastsquares problems. Consider the following least squares problem with linearequality constraints:minx|c Gy|2(2.21)subject to Ey = d, (2.22)wherec Rp,G Rpm,y Rm,E Rqm,d Rqandq< m. Problemsof this kind arise, for instance, in curve or surface tting when the curve isrequiredtointerpolatecertaindatapoints; seeBj orck(1996, Chapter5).The optimality conditions for problem (2.21)(2.22) are__IpO GO O EGTETO____ry__ =__cd0__, (2.23)12 M.Benzi,G.H.GolubandJ.LiesenwhereIpistheppidentitymatrixand RqisavectorofLagrangemultipliers. Clearly,(2.23) is a special case of the symmetric saddle pointproblem (1.3).Next we consider the generalized linear least squares problemminx(f Gx)TW1(f Gx), (2.24)wheref Rn, G Rnmwithm0istheregularizationparameterandLiseitherthemmidentityorsometypeof smoothingoperator, suchasarst-ordernitedierenceoperator. SeeBenzi andNg(2004)andGonzalesandWoods(1992) for applications in image processing.2.3. Saddle point systems from interior point methodsHere we show how saddle point systems arise when interior point methodsareusedtosolveconstrainedoptimizationproblems. Ourpresentationisbased on the nice synopsis given in Bergamaschi, Gondzio and Zilli (2004).Consider a convex nonlinear programming problem,minf(x) (2.25)subject to c (x) 0, (2.26)wheref:RnRandc:RnRmareconvexandtwicedierentiable.Introducing a nonnegative slack variable z Rm, we can write the inequalityconstraintasthesystemofequalitiesc (x) + z= 0,andwecanintroducethe associated barrier problem:minf(x) m

i=1ln zi(2.27)subject to c (x) +z = 0. (2.28)Numericalsolutionofsaddlepointproblems 13The corresponding Lagrangian isL(x, y, z; ) = f(x) +yT(c (x) +z) m

i=1ln zi.To nd a stationary point of the Lagrangian we setxL(x, y, z; ) = f(x) +c (x)Ty = 0, (2.29)yL(x, y, z; ) = c (x) +z = 0, (2.30)zL(x, y, z; ) = y Z1e = 0, (2.31)whereZ= diag(z1, z2, . . . , zm)ande = [1 1. . . 1]T. Introducingthediag-onalmatrixY = diag(y1, y2, . . . , ym),therst-orderoptimalityconditionsfor the barrier problem becomef(x) +c (x)Ty = 0, (2.32)c (x) +z = 0, (2.33)Y Ze = e, (2.34)y, z 0. (2.35)This is a nonlinear system of equations with nonnegativity constraints andit can be solved by Newtons method. The barrier parameter is graduallyreduced so as to ensure convergence of the iterates to the optimal solutionof problem (2.25)(2.26). At each Newton iteration, it is necessary to solvea linear system of the form__H(x, y) B(x)TOB(x) O IO Z Y____xyz__ =__f(x) B(x)Tyc (x) ze Y Ze__, (2.36)whereH(x, y) = 2f(x) +m

i=1yi2ci(x) Rnnand B(x) = c (x) Rmn.Here 2f(x)denotestheHessianof fevaluatedatx. Thelinearsystem(2.36) can be reduced to one of smaller dimensions by using the third equa-tiontoeliminatez=Y1e Ze ZY1yfromthesecondequation.The resulting system is_H(x, y) B(x)TB(x) ZY1_ _xy_ =_f(x) +B(x)Tyc (x) Y1e_. (2.37)Apart fromthesign, (2.37) is asaddlepoint systemof theform(1.6).If theobjectivefunctionf(x) andtheconstraints ci(x) areconvex, thesymmetric matrix H(x, y) is positive semidenite, and it is positive deniteif f(x)isstrictlyconvex. Thediagonal matrixZY1isobviouslyposi-14 M.Benzi,G.H.GolubandJ.Liesentivesemidenite. Thecoecientmatrixin(2.37)dependsonthecurrentapproximation (x, y), and it changes at each Newton step.Similar linear systems arise when interior point methods are used to solvelinear and quadratic programming problems. Now the systems to be solvedat each Newton iteration are of the form_H D BTB O_ _xy_ =__,where the nn matrix His symmetric positive semidenite if the problemisconvexandDisa(positive)diagonal matrix. NowHandBremainconstant (H O in linear programming), whileD changes at each Newtoniteration.There are many interesting linear algebra problems arising from the use ofinterior point methods; see in particular Bergamaschi et al. (2004), Czyzyk,Fourer and Mehrotra (1998), Forsgren, Gill and Shinnerl (1996), Fourer andMehrotra(1993), Frangioni andGentile(2004), FreundandJarre(1996),Gill, Murray, Poncele on and Saunders (1992), Nocedal and Wright (1999),Oliveira and Sorensen (2005), Wright (1992) and Wright (1997).3. PropertiesofsaddlepointmatricesThis sectionis devotedtoestablishingbasic algebraic properties of thesaddlepointmatrix /suchasexistenceof variousfactorizations, invert-ibility, spectral properties, and conditioning. Knowledge of these propertiesis important in the development of solution algorithms.3.1. Block factorizations and the Schur complementIf Aisnonsingular,thesaddlepointmatrix /admitsthefollowingblocktriangular factorization:/ =_A BT1B2C_ =_I OB2A1I_ _A OO S_ _I A1BT1O I_, (3.1)where S = (C+B2A1BT1 ) is the Schur complement of A in /. A numberof important properties of the saddle point matrix / can be derived on thebasis of (3.1): we do this in the next three subsections.Also useful are the equivalent factorizations/ =_A OB2S_ _I A1BT1O I_(3.2)and/ =_I OB2A1I_ _A BT1O S_. (3.3)Numericalsolutionofsaddlepointproblems 15Theassumptionthat Aisnonsingularmayappeartoberatherrestrict-ive, sinceAissingularinmanyapplications; see, e.g., HaberandAscher(2001). However, one can use augmented Lagrangian techniques (Fortin andGlowinski 1983, Glowinski and Le Tallec 1989, Golub and Greif 2003, Greif,Golub and Varah 2005) to replace the original saddle point system with anequivalent one having the same solution but in which the (1, 1) blockA isnownonsingular. Hence, nogreatlossofgeneralityisincurred. Weshallreturn to augmented Lagrangian techniques in Section 3.5.Besidesbeinguseful forderivingtheoretical propertiesof saddlepointmatrices, the decompositions (3.1)(3.3) are also the basis for many of themost popular solution algorithms for saddle point systems, as we shall see.3.2. Solvability conditionsAssumingA is nonsingular, it readily follows from any of the block decom-positions (3.1)(3.3) that / is nonsingular if and only if S is. Unfortunately,very little can be said in general about the invertibility of the Schur com-plementS= (C + B2A1BT1 ). It is necessary to place some restrictionson the matricesA,B1,B2 andC.Symmetric caseWe begin with the standard saddle point system (1.3), where A is symmetricpositive denite, B1 =B2andC=O. In this case the Schur complementreduces toS = BA1BT, a symmetric negative semidenite matrix. It isobvious thatS, and thus /, is invertible if and only ifBThas full columnrank (hence, if and only if rank(B) = m), since in this caseSis symmetricnegative denite. Then both problems (1.3) and (1.4)(1.5) have a uniquesolution: if (x, y)isthesolutionof (1.3), xistheuniquesolutionof(1.4)(1.5). It can be shown thatxis theA-orthogonal projection of thesolution x =A1fof the unconstrained problem (1.4) onto the constraintset (= x Rn[ Bx=g. Here A-orthogonal meansorthogonal withrespect to the inner product v, w)A wTAv. We will discuss this in moredetail in Section 3.3.Next we consider the case whereA is symmetric positive denite, B1=B2=B, andC,=Ois symmetric positive semidenite. ThenagainS= (C + BA1BT)issymmetricnegativesemidenite, anditisneg-ativedenite(hence, invertible) if andonlyif ker(C) ker(BT) = 0.Obvious sucient conditions for invertibility are that C be positive deniteorthatBhavefullrowrank. Wecansummarizeourdiscussionsofarinthe following theorem.Theorem3.1. AssumeAissymmetricpositivedenite, B1=B2=B,andCis symmetric positive semidenite. If ker(C) ker(BT) = 0, then16 M.Benzi,G.H.GolubandJ.Liesenthe saddle point matrix / is nonsingular. In particular, / is invertible if Bhas full rank.Now we relax the condition thatA be positive denite. If A is indenite,the following simple example shows that / may be singular, even ifBhasfull rank:/ =__1 0 10 1 11 1 0__ =_A BTB O_.However, / will be invertible ifA is positive denite on ker(B). WhenAissymmetricpositivesemidenite, wehavethefollowingresult(see, e.g.,thediscussionofquadraticprogramminginHadley(1964)orLuenberger(1984, page 424)). Although this is a well-known result, we include a proofto make our treatment more self-contained.Theorem3.2. AssumethatAissymmetricpositivesemidenite, B1=B2 = B has full rank, and C = O. Then a necessary and sucient conditionfor the saddle point matrix / to be nonsingular is ker(A) ker(B) = 0.Proof. Let u=_xy_besuchthat /u=0. Hence, Ax + BTy=0andBx=0. ItfollowsthatxTAx= xTBTy= (Bx)Ty=0. SinceAissymmetric positivesemidenite, xTAx = 0 impliesAx = 0 (see HornandJohnson (1985, page 400)), and thereforex ker(A) ker(B), thusx = 0.Also,y = 0 sinceBTy = 0 andBThas full column rank. Thereforeu = 0,and / is nonsingular. This proves the suciency of the condition.Assumenowthatker(A) ker(B) ,= 0. Takingx ker(A) ker(B),x ,= 0andlettingu =_x0_wehave /u = 0, implyingthat /issingular.Hence, the condition is also necessary.Remark. ItisclearfromtheproofofthistheoremthattherequirementthatA be positive semidenite can be somewhat relaxed: it suces thatAbe denite on ker(B). In fact, all we need is that xTAx ,= 0 for x ker(B),x ,=0. ThisimpliesthatAiseitherpositivedeniteornegativedeniteon ker(B). In any case, the rank ofA must be at leastn m for / to benonsingular.How restrictive is the assumption thatBhas full rank?A rank decientBsignies that some of the constraints are redundant. It is generally easyto eliminate this redundancy. For instance,in the Stokes and Oseen case,whereBTrepresents a discrete gradient, a one-dimensional subspace (con-tainingall theconstantvectors)isoftenpresent. Hence /hasonezeroeigenvalue, corresponding to the so-called hydrostatic pressure mode, due tothe fact that the pressure is dened up to a constant. A similar situationNumericalsolutionofsaddlepointproblems 17occurs with electric networks, where y, the vector of nodal potentials, is alsodened up to an additive constant. The rank deciency inBcan easily beremovedbygroundingoneofthenodes, thatis, byspecifyingthevalueofthepotential(orofthepressure)atonepoint. Oneproblemwiththisapproach is that the resulting linear system may be rather ill-conditioned;seeBochevandLehoucq(2005). Fortunately, sincethesystem /u =bisconsistentbyconstruction, itmaynotbenecessarytoremovethesingu-larityof /. IterativemethodslikeGMRES(SaadandSchultz1986)arelargelyunaectedbythepresenceof asingleeigenvalueexactlyequal tozero, at least when using a zero initial guess,u0 = 0. The reader is referredto Elman et al. (2005c, Section 8.3.4) for a detailed discussion of this issuein the context of uid ow problems; see further the remarks in Olshanskiiand Reusken (2004, Section 4) and Zhang and Wei (2004, Section 4).General caseWhenC=O, anecessary conditionforinvertibilityisprovidedbythefollowingtheorem, aslightgeneralizationof asimilarresultforthecaseB1 = B2: see Gansterer, Schneid and Ueberhuber (2003).Theorem3.3. If the matrix/ =_A BT1B2O_is nonsingular, then rank(B1) = m and rank_AB2_ = n.Proof. Ifrank(B1) 0 for all ( /).Proof. Toprove(i)weobservethatforanyv Rn+mwehavevT/v=vTHv, whereH 12( /+/T) =_H OO C_is the symmetric part of/. Clearly H is positive semidenite, so vT/v 0.To prove (ii), let (, v) be an eigenpair of /, with |v|2 = 1. Then v/v = and (v/v) = v/Tv = . Therefore12v( /+/T)v =+2= Re(). Toconclude the proof, observe thatv( /+/T)v = Re(v)T( /+/T)Re(v) + Im(v)T( /+/T)Im(v),a real nonnegative quantity.To prove (iii), assume (, v) is an eigenpair of/ withv =_xy_. ThenRe() = xHx +yCy= Re(x)THRe(x) + Im(x)THIm(x)+ Re(y)TCRe(y) + Im(y)TCIm(y).This quantity is nonnegative,and it can be zero only ifx = 0 (sinceHisassumedtobepositivedenite)andCy= 0. Butif x = 0thenfromtherstequationinthesystem/v=vwegetBTy= 0,hencey= 0sinceBThas full column rank. Hencev = 0, a contradiction.Figure 3.1(b) displays the eigenvalues of the matrix/ corresponding tothesameOseenproblemasbefore. Ascanbeseen, all theeigenvalueslie in the right half-plane. However, such distribution of eigenvalues is notnecessarily more favourable for Krylov subspace methods, and in fact Krylovsubspacemethodswithoutpreconditioningperform justaspoorly astheydo on the original problem. We revisit this topic repeatedly in the courseof this survey.Numericalsolutionofsaddlepointproblems 25When A = ATand C = CTthe matrix / is symmetric indenite whereas/ is nonsymmetric positive (semi-)denite. Moreover,/ satises/ =/T, (3.12)with dened as in (3.11); that is,/ is -symmetric (or pseudosymmetric:see Mackey,Mackey andTisseur(2003)). Inotherwords,/ is symmetricwithrespecttotheindeniteinnerproductdenedonRn+mby[v, w] wTv. Conversely, any -symmetricmatrixisof theform_A BTB C_forsomeA Rnn, B Rmn, andC RmmwithA =ATandC=CT.Note that the set of all -symmetric matricesJ =__A BTB C_ A = AT Rnn, B Rmn, C = CT Rmm_isclosedunder matrixadditionandunder theso-calledJordanproduct,dened asT( 12(T ( +( T).The triple (J, +, ) is a non-associative, commutative algebra over the reals.It is known as the Jordan algebra associated with the real Lie groupO(n, m, R) of -orthogonal (or pseudo-orthogonal ) matrices, i.e., the groupof all matrices Q Rn+mthat satisfy the condition QT Q = ; see Am-mar,Mehl andMehrmann(1999)andMackey etal. (2003). Thespectraltheory of these matrices has been investigated by several authors. A Schur-like decomposition for matrices in J has been given in Ammar et al. (1999,Theorem 8), and properties of invariant subspaces of -symmetric matriceshave been studied in Gohberg, Lancaster and Rodman (1983).Besides being mathematically appealing, these algebraic properties haveimplicationsfromthepointofviewofiterativemethods: seeforexampleFreund, GolubandNachtigal (1992, page80), whereitisshownhow -symmetry can be exploited to develop transpose-free variants of basic Krylovmethodsusingshortrecurrences. Thisisof coursenotenoughtojustifyusingthenonsymmetricform/when /issymmetric, sinceinthiscaseonemayaswell useasymmetricKrylovsolverontheoriginal (symmet-ric) formulation;see Fischer and Peherstorfer (2001), Fischer etal. (1998)and Section 9 below. Nevertheless, there are some advantages in using thetransformedlinearsystem/v=binsteadof theoriginal one, especiallywhencertainpreconditionersareused; seeBenzi andGolub(2004), Sidi(2003)andSection10.3below. ItcanbeshownthatwhenAandCaresymmetric, at most 2m of then + m eigenvalues of/ can have a nonzeroimaginary part (Simoncini 2004b);furthermore, in some important specialcases it turns out that the eigenvalues of/ are all real and positive. Thisimplies the existence of a nonstandard inner product on Rn+mwith respect26 M.Benzi,G.H.GolubandJ.Liesentowhich/issymmetricpositivedenite, adesirablepropertyfromthepointofviewofiterativemethods. Thefollowingresultgivesasucientcondition (easily checked in many cases) for the eigenvalues of/ to be real.Theorem3.7. Assume thatA is symmetric positive denite,B1 = B2 =Bhasfull rank, andC=O. Let S=BA1BT, andlet ndenotethesmallesteigenvalueof A. If n 4|S|2, thenall theeigenvaluesof thematrix/ in (3.10) are real and positive.Proof. See Simoncini and Benzi (2005).We note that the conditions expressed in Theorem 3.7 are satised, for in-stance, for the stationary Stokes problem under a variety of nite dierencesand nite element discretization schemes.A more detailed analysis is available in the special situation that A = I,where>0isapositivescalingparameter, B1=B2=B, andC=O.Denote the resulting usual saddle point matrix by /+ , and the alternativeformulation with negative (2,1) block by / , i.e.,/=_ I BTB O_. (3.13)Thefollowingtheoremcharacterizestheinuenceofthechoiceof+or as well as on the eigenvalues of the matrices / .Theorem3.8. SupposethatthematrixBhasrankm r, anddenotethe nonzero singular values ofBby1 mr.1Then +m eigenvalues of /+in (3.13) are given by(i) zero with multiplicityr,(ii) with multiplicityn m+r,(iii)12_ _42k +2_ fork = 1, . . . , mr.2Furthermore, if 1 t>2t+1 mr, thenthen +m eigenvalues of /in (3.13) are given by(i) zero with multiplicityr,(ii) with multiplicityn m+r,(iii)12_ _242k_ fork = t + 1, . . . , mr,(iv)12_ i_42k 2_ fork = 1, . . . , t.Proof. See Fischer et al. (1998, Section 2).This result shows that the eigenvalues of the symmetric indenite matrix/+(except for the multiple eigenvalues zero and ) always lie inNumericalsolutionofsaddlepointproblems 27twointervalssymmetricaboutthepoint /2. Changingonlyleadstoascalingofthesetwointervals. Ontheotherhand,thechoiceof hasasignicanteectontheeigenvaluesofthenonsymmetricmatrix / . Forexample, if > 21, then all eigenvalues of /are real, while for < 2mrall eigenvalues (except zero and) are purely imaginary. For intermediatevalues of , the eigenvalues (except zero and ) form a cross in the complexplane with midpoint/2. One is immediately tempted to determine whateigenvaluedistributionisthemostfavourableforthesolutionbyKrylovsubspacemethods; seeSidi(2003). WediscussthistopicinSection10.1,wherethematrices /arisenaturallyasaresultof blockdiagonal pre-conditioning of a symmetric saddle point matrix / withA positive deniteandC = O.3.5. Conditioning issuesSaddle point systems that arise in practice can be very poorly conditioned,and care must be taken when developing and applying solution algorithms.Itturnsoutthatinsomecasesthespecial structureof thesaddlepointmatrix / can be exploited to avoid or mitigate the eect of ill-conditioning.Moreover,thestructureoftheright-handsidebin(1.1)alsoplaysarole.Indeed, itisfrequentlythecasethateither f or gin(1.1)iszero. Forinstance, f =0instructural analysis(intheabsenceof dilation)andinmixedformulationsof Poissonsequation, whileg=0inincompressibleowproblemsandweightedleast-squares. Soif g(say)iszero, the(1, 2)and(2, 2) blocks in /1(see(3.4) and(3.8)) havenoinuenceonthesolutionu = /1b. In particular, any ill-conditioning that may be presentintheseblockswillnotaectthesolution, animportantfactthatshouldbe taken into account in the development of robust solution algorithms; seeDu (1994), Gansterer et al. (2003) and Vavasis (1994).Let us consider, for the sake of simplicity, a standard saddle point problemwhereA = ATis positive denite,B1 = B2 = Bhas full rank, andC = O.In this case / is symmetric and its spectral condition number is given by(/) =max [(/)[min [(/)[.From Theorem 3.5 one can see that the condition number of / grows un-boundedly as eithern = min(A) orm = min(B) goes to zero (assumingthat max(A)andmax(B)arekeptconstant). FormixedniteelementformulationsofellipticPDEs, bothnandmgotozeroash, themeshsizeparameter, goestozero, andtheconditionnumberof /growslikeO(hp)forsomepositivevalueof p; see, e.g., Maryskaetal.(1996), Wa-thenetal. (1995). Thisgrowthoftheconditionnumberof /meansthatthe rate of convergence of most iterative solvers (like Krylov subspace meth-ods) deteriorates as the problem size increases. As discussed in Section 10,28 M.Benzi,G.H.GolubandJ.Liesenpreconditioningmaybeusedtoreduceoreveneliminatethisdependencyonh in many cases. Similar considerations apply to nonsymmetric saddlepoint problems.A dierent type of ill-conditioning is encountered in saddle point systemsfrom interior point methods. Consider, for instance, the case of linear pro-gramming, where the (1, 1) blockA is diagonal. As the iterates generatedby the interior point algorithm approach the solution, many of the entriesofA tend to zero or innity, and thus / becomes very ill-conditioned (theconstraint matrix B remains constant throughout the iteration process). Inparticular, the norm of the inverse Schur complement S1= (BA1BT)1goes to innity. However, Stewart (1989) and Todd (1990) (see also Forsgren(1996)) have shownthat the normof the matrices X=S1BA1(aweighted pseudo-inverse of BT) and BTX(the associated oblique projectoronto the column space of BT) are bounded by numbers that are independentof A. Thisimportantobservationhasbeenexploitedina seriesofpapersby Vavasis and collaborators (Bobrovnikova and Vavasis 2001, Hough andVavasis 1997, Vavasis 1994, Vavasis 1996) to develop stable algorithms forcertain saddle point problems with a severely ill-conditioned (1, 1) blockA.Whenusingdirect methods basedontriangular factorization, Bj orck(1996, Sections2.5.3and4.4.2)hasnotedtheimportanceof scalingthe(1, 1) blockA by a positive scalar quantity. Suitable tuning of this scalingfactor can be interpreted as a form of preconditioning and has a dramaticimpact on the accuracy attainable by sparse direct solvers (Arioli, Du andDe Rijk 1989, Du 1994). Onthe otherhand,suchscaling seems to havelittle or no eect on the convergence behaviour of Krylov subspace methods(Fischer et al. 1998).Another possibleapproachfor dealingwithanill-conditionedor evensingular(1, 1)blockAistheaugmentedLagrangianmethod; seeFortinandGlowinski (1983), Glowinski andLeTallec(1989), Hestenes(1969),Powell (1969)andthemoregeneral treatmentinGolubandGreif(2003)and Greif etal. (2005). Here we assume thatA =AT(possibly singular),B1 =B2 =Bhas full rank, andC=O. The idea is to replace the saddlepoint system (1.3) with the equivalent one_A+BTWB BTB O_ _xy_ =_f +BTWgg_. (3.14)ThemmmatrixW, tobesuitablydetermined, issymmetricpositivesemidenite. Thesimplest choiceis totake W=I ( >0). Inthiscasethe(1, 1)blockin(3.14)isnonsingular, andindeedpositivedenite,provided thatA is positive denite on ker(B). The goal is to chooseWsothat system (3.14) is easier to solve than the original one, particularly whenusing iterative methods. WhenW= Iis used, the choice = |A|2/|B|22has been found to perform well in practice, in the sense that the conditionNumericalsolutionofsaddlepointproblems 29number of both the (1, 1) block and of the whole coecient matrix in (3.14)are approximately minimized. This choice also results in rapid convergenceof classical iterative schemes like the method of multipliers; see Golub andGreif (2003) and Section 8.2 below.The conditioning of equality-constrained and weighted least squares prob-lems has been studied in depth by several authors; see Gulliksson, Jin andWei (2002), Wei (1992b) and the references therein.Conditioning properties of quasidenite and saddle-point matrices arisingfrom interior-point methods in linear programming have also been investig-ated in George and Ikramov (2000) and Korzak (1999). Finally, we mentionthat a numerical validation method for verifying the accuracy of approxim-atesolutions of symmetricsaddlepoint problems has beenpresentedinChen and Hashimoto (2003).4. OverviewofsolutionalgorithmsBesides the usual (and somewhat simplistic) distinction between direct anditerative methods, solution algorithms for generalized saddle point problemscan be subdivided into two broad categories,which we will call segregatedandcoupled(orallatonce)methods. Segregatedmethodscomputethetwo unknown vectors,x andy, separately; in some cases it isx to be com-putedrst, inothersitis y. Thisapproachinvolvesthesolutionof twolinear systems of size smaller thann + m (called reduced systems), one foreach of x and y; in some cases a reduced system for an intermediate quantityissolved. Segregatedmethodscanbeeitherdirectoriterative,orinvolveacombinationofthetwo; forexample, oneofthereducedsystemscouldbesolvedbyadirectmethodandtheotheriteratively. Themainrepres-entativesofthesegregatedapproacharetheSchurcomplementreductionmethod, which is based on a block LU factorization of /, and the null spacemethod, which relies on a basis for the null space for the constraints.Coupledmethods, ontheotherhand, deal withthesystem(1.1)asawhole, computing x and y (or approximations to them) simultaneously andwithoutmakingexplicituseof reducedsystems. Thesemethodsincludeboth direct solvers based on triangular factorizations of the global matrix /,and iterative algorithms like Krylov subspace methods applied to the entiresystem (1.1), typically with some form of preconditioning. As we shall see,preconditioningtendstoblurthedistinctionbetweendirectanditerativesolvers, andalsothatbetweensegregatedandcoupledschemes. Thisisbecausedirectsolversmaybeusedtoconstructpreconditioners, andalsobecause preconditioners for coupled iterative schemes are frequently basedon segregated methods.In the next sections we review a number of solution methods, starting withdirect solvers and continuing with stationary iterative methods, Krylov sub-30 M.Benzi,G.H.GolubandJ.Liesenspace solvers, and preconditioners. We also include a brief discussion of mul-tilevel methods, including multigrid and Schwarz-type algorithms. Withineach group, we discuss segregated as well as coupled schemes and the inter-play between them. It is simply not possible to cover every method that hasbeen described in the literature; instead, we have striven to include, besidesall of the classical algorithms, those among the more recent methods thatappear to be the most widely applicable and eective.5. SchurcomplementreductionConsider the saddle point system (1.1), orAx +BT1 y = f, B2x Cy = g.We assume that bothA and / are nonsingular;by (3.1) this implies thatS= (C + B2A1BT1 )isalsononsingular. Pre-multiplyingbothsidesofthe rst equation byB2A1, we obtainB2x +B2A1BT1 y = B2A1f.UsingB2x = g +Cy and rearranging, we nd(B2A1BT1+C) y = B2A1f g, (5.1)a reduced system of order mfor y involving the (negative) Schur complementS = B2A1BT1+ C. Note that unlessf= 0, forming the right-hand sideof (5.1) requires solving a linear system of the formAv = f.Oncey has been computed from (5.1),x can be obtained by solvingAx = f BT1 y, (5.2)a reduced system of ordern forx involving the (1,1) block, A. Note thatthisis justblock Gaussianeliminationappliedto (1.1). Indeed,using theblock LU factorization (3.3) we get the transformed system_I OB2A1I_ _A BT1B2C_ _xy_ =_I OB2A1I_ _fg_,that is,_A BT1O S_ _xy_ =_fg B2A1f_.Solving this block upper triangular system by block backsubstitution leadstothetworeducedsystems(5.1)and(5.2)for yandx. Thesesystemscanbesolvedeitherdirectlyoriteratively. Intheimportantspecial casewhereAand Saresymmetricpositivedenite, highlyreliablemethodssuch as Cholesky factorization or the conjugate gradient (CG) method canbe applied.Numericalsolutionofsaddlepointproblems 31Thesolutionstrategyoutlinedaboveiscommonlyusedinthecomple-mentary energy formulation of structural mechanics,where it is known asthe displacementmethod, since the vector of nodal displacements y is com-puted rst; the reduction to the Schur complement system (5.1) is known asstatic condensation, and the Schur complement itself is called the assembledstiness matrix(McGuire and Gallagher 1979). In electrical engineering itis knownasthe nodal analysismethod,andinoptimizationas the range-space method(Vavasis 1994). Inall theseapplications, Ais symmetricpositive (semi)denite,B1 = B2, andC = O.This approach is attractive if the orderm of the reduced system (5.1) issmall and if linear systems with coecient matrix A can be solved eciently.The main disadvantages are the need forA to be nonsingular, and the factthat the Schur complement S = (BA1BT+C) may be completely full andtoo expensive to compute or to factor. Numerical instabilities may also be aconcern when forming S, especially when A is ill-conditioned (Vavasis 1994).Dense Schur complements occur in the case of Stokes and Oseen problems,whereAcorrespondstoa(vector)dierential operator. OtherexamplesincludeproblemsfromoptimizationwhenBcontainsoneormoredensecolumns. Note, however, that whenBcontains no dense columns andA1is sparse (e.g.,A is diagonal or block diagonal with small blocks), then S isusually quite sparse. In this case ecient (graph-based) algorithms can beused to form S, and it is sometimes possible to apply the Schur complementreduction recursively and in a way that preserves sparsity through severallevels, in the sense that the number of nonzeros to be stored remains nearlyconstant throughout the successive reduction steps; see Maryska, Rozloznkand Tuma (2000) for an example arising from the solution of groundwaterow problems.In cases where A is positive semidenite and singular, Schur complementreduction methods may still be applied by making use of augmented Lagran-gian techniques (3.14), which replace the original saddle point system withanequivalent one witha nonsingular(1,1) block. IfSis too expensive toform or factor, Schur complement reduction can still be applied by solving(5.1) by iterative methods that do not need access to individual entries ofS, but only needSin the form of matrix-vector productsp = Sy = (B2A1BT1+C)y.The action ofS ony can be computed by means of matrix-vector productswith BT1 , B2 and C and by solving a linear system with matrix A. If the lat-ter can be performed eciently and the iteration converges suciently fast,this is a viable option. The Schur complement system (5.1), however, mayberatherill-conditioned, inwhichcasepreconditioningwill berequired.Preconditioning the system (5.1) is nontrivial when S is not explicitly avail-32 M.Benzi,G.H.GolubandJ.Liesenable. SomeoptionsarediscussedinSection10.1below, inthecontextofblock preconditioners.6. NullspacemethodsIn this section we assume that B1 = B2 = B has full rank and C = O. Fur-thermore, we assume that ker(H)ker(B) = 0, where H is the symmetricpart ofA. The saddle point system is thenAx +BTy = f, Bx = g.The null space method assumes that the following are available:(1) a particular solution x ofBx = g;(2) a matrix Z Rn(nm)such that BZ = O, that is, range(Z) = ker(B)(the columns ofZspan the null space ofB).Then the solution set ofBx = g is described byx = Zv + x asv ranges inRnm. Substitutingx = Zv + x inAx +BTy = f, we obtainA(Zv + x) =f BTy. Pre-multiplying by the full-rank matrix ZTand using ZTBT= O,we getZTAZ v = ZT(f A x), (6.1)a reduced system of order nm for the auxiliary unknown v. This system isnonsingular under our assumptions. Once the solution v has been obtained,we setx = Zv + x; nally,y can be obtained by solvingBBTy = B(f Ax), (6.2)a reduced system of orderm with a symmetric positive denite coecientmatrixBBT. Of course, (6.2) is just the normal equations for the overde-termined systemBTy = f Ax, orminy|(f Ax) BTy|2,which could be solved, e.g., by LSQR (Paige and Saunders 1982) or a sparseQR factorization (Matstoms 1994). Just as the Schur complement reductionmethod can be related to expression (3.4) for /1, the null space method isrelated to the alternative expression (3.8). It is interesting to observe thatwhenAisinvertible,thenullspacemethodisjusttheSchurcomplementreduction method applied to the dual saddle point problem_A1ZZTO_ _wv_ =_ xZTf_.This strategy subsumes a whole family of null spacemethods, which dierprimarily in the way the matrixZ(often called a nullbasis) is computed;Numericalsolutionofsaddlepointproblems 33see the discussion below. Null space methods are quite popular in optimiz-ation,wheretheyareusuallyreferredtoasreducedHessianmethods; seeColeman(1984), Fletcher(1987), Gill et al. (1981), Nocedal andWright(1999) and Wolfe (1962). In this setting the matrixA is the (nn) Hes-sian of the function to be minimized subject to the constraintBx = g, andZTAZ is the reduced ((nm) (nm)) Hessian, obtained by eliminationof the constraints. WhenZhas orthonormal columns, the reduced system(6.1)canalsobeseenasaprojectionoftheproblemontotheconstraintset. The null space approach has been extensively used in structural mech-anicswhereitisknownunderthenameofforcemethod, becausex, thevector of internal forces,is computed rst;see,e.g.,Berry and Plemmons(1987), Heath, PlemmonsandWard(1984), Kaneko, LawoandThierauf(1982), KanekoandPlemmons(1984), PlemmonsandWhite(1990)andRobinson (1973). Other application areas where the null space approach isused include uid mechanics (under the somewhat misleading name of dualvariablemethod, see Amit, Hall and Porsching (1981), Arioli and Manzini(2002, 2003), Arioli, Maryska, Rozloznk and Tuma (2001), Gustafson andHartmann (1983), Hall (1985), Sarin and Sameh (1998)) and electrical en-gineering (under the name of loopanalysis; see Chua etal. (1987), Strang(1986), Tropper (1962), Vavasis (1994)).The null space method has the advantage of not requiringA1. In fact,the method is applicable even whenA is singular, as long as the conditionker(H) ker(B) = 0 is satised. The null space method is often used inapplications that require the solution of a sequence of saddle point systemsof the type_AkBTB O_ _xy_ =_fkgk_, k = 1, 2, . . . ,where theAksubmatrix changes withkwhileBremains xed. This situ-ations arises, for instance, in the solution of unsteady uid ow problems,andinthereanalysis of structuresincomputational mechanics; see, e.g.,Batt and Gellin (1985), Hall (1985) and Plemmons and White (1990). An-other example is the analysis of resistive networks with a xed connectivityand dierent values of the resistances. In all these cases the null basis matrixZneeds to be computed only once.Nullspacemethodsareespeciallyattractivewhenn missmall. If Aissymmetricandpositivesemidenite, thenZTAZissymmetricpositivedenite and ecient solvers can be used to solve the reduced system (6.1).IfZis sparse then it may be possible to form and factorZTAZexplicitly,otherwiseiterativemethodsmustbeused, suchasconjugategradientsorothers.Themethodislessattractiveif n mislarge, andcannotbeappliedif C,=O. Themaindiculty, however, isrepresentedbytheneedfor34 M.Benzi,G.H.GolubandJ.Liesenanull basis Zfor B. WenotethatcomputingaparticularsolutionforBx=gis usuallynot adicult problem, andit canbeobtainedas abyproduct of the computations necessary to obtain Z. In the case where g =0 (arising for instance from the divergence-free condition in incompressibleowproblems), thetrivial solution x=0will do. Hence, themainissueis the computation of a null basisZ. There are a number of methods thatone can use, at least in principle, to this end. In the large and sparse case,graph-based methods invariably play a major role.Let Pdenoteapermutationmatrixchosensothat BP = _BbBn,whereBbism mandnonsingular(thisisalwayspossible,sinceBisofrankm). Then it is straightforward to verify that the matrixZ = P_B1bBnI_, (6.3)whereIdenotestheidentitymatrixofordern m,isanullbasisforB.This approach goes back to Wolfe (1962); a basis of the form (6.3) is called afundamental basis. Quite often, the matrix B1bBn is not formed explicitly;rather, an LU factorization ofBbis computed and used to perform opera-tions involving B1b. For instance, if an iterative method like CG is used tosolve (6.1),then matrix-vector products withZTAZcan be performed bymeansofforwardandbacksubstitutionswiththetriangularfactorsof Bb,in addition to matrix-vector products withBn,BTn, andA.Since there are in principle many candidate submatricesBb, (i.e., manypermutation matricesP) it is natural to ask whether one can nd a matrixBb with certain desirable properties. Ideally, one would like Bb to be easy tofactor, well-conditioned, and to satisfy certain sparsity requirements (eitherinBb, or in its factors, or inB1bBn). Another desirable property could besomekindofdiagonal dominance. Intheliterature, thisisknownasthenicebasis problem. Thisisaverydicultproblemingeneral. Considerrst the sparsity requirement. Unfortunately, not all sparse matrices admitasparsenull basis. Toseethis, considerthematrixB= _I e, whereeis thecolumnvector all of whosecomponents areequal to1; clearly,there is no explicit sparse representation for its one-dimensional null space(Gilbert and Heath 1987). Moreover, even if a sparse null basis exists, theproblemof computinganull basisZ(fundamental ornot)withminimalnumberof nonzeroentrieshasbeenshowntobeNP-hard(ColemanandPothen 1986, Pothen 1984). In spite of this, there are important situationswhere a sparse null basis exists and can be explicitly obtained. As we shallsee, there may be no need to explicitly factor or invert any submatrix of B.An example of this is Kirchhos classical method for nding the currentsinaresistiveelectrical network(Kirchho1847). Ourdiscussioncloselyfollows Strang (1986). For this problem,Bis just the nodeedge incidenceNumericalsolutionofsaddlepointproblems 35matrix of the network, or directed graph, describing the connectivity of thenetwork. More precisely, if the network consists of m+1 nodes and n edges,let B0 be the (m+1) n matrix with entries bij given by 1 if edge j startsat node i and by +1 if edge j ends at node i; of course, bij = 0 if edge j doesnotmeetnodei. Hence, B0behavesasadiscretedivergenceoperatoronthe network: each column contains precisely two nonzero entries, one equalto+1andtheotherequalto 1. MatrixB0canbeshowntobeofrankm; note thatBT0 e = 0. A full-rank matrixBcan be obtained by droppingthe last row ofB0; that is, by grounding the last node in the network: seeStrang (1986, page 112).Anull spaceforBcanbefoundusingKirchhosVoltageLaw, whichimpliesthatthesumofthevoltagedropsaroundeachclosedloopinthenetworkmustbezero. Inotherwords, forcurrentowingaroundaloopthere is no buildup of charge. In matrix terms, each loop current is a solutionto By = 0. Since B has full rank, there are exactly nm independent loopcurrents, denoted byz1, z2, . . . znm. The loop currents can be determinedby a procedure due to Kirchho, which consists of the following steps.(1) Find a spanning treefor the network (graph); this is a connected sub-graph consisting of the m+1 nodes and just m edges, so that betweenany two nodes there is precisely one path, and there are no loops. Asshown by Kirchho,there are exactlyt = det BBTspanning trees inthe network (which is assumed to be connected).(2) Once a spanning tree has been picked, the remainingn m edges canbe used to construct then m loop currents by noticing that addingany of these edges to the spanning tree will create a loop. For each ofthese fundamentalloopswe construct the corresponding columnziofZby setting thejth entry equal to 1 if edgejbelongs to the loop,and equal to 0 otherwise; the choice of sign species the orientation ofthe edge.The resultingZ = [z1,z2,. . . ,znm] is then called an edgeloop matrix,andisabasisforthenullspaceof B. Asasimpleexample, considerthedirectedgraphofFigure6.1, withthespanningtreeontheright. Inthisexample, m=4andn=7. ThenodeedgeincidencematrixB0forthisgraph isB0 =__1 1 0 1 0 0 00 1 1 0 0 0 01 0 0 0 1 0 10 0 1 1 0 1 10 0 0 0 1 1 0__.Notethatrank(B0)=4; thematrixBobtainedfromB0bygroundingnode 5 (i.e., by dropping the last row ofB0) has full row rank, equal to 4.36 M.Benzi,G.H.GolubandJ.LiesenFigure 6.1. A directed graph with one of its spanning trees.Consider now the spanning tree on the right of Figure 6.1. By adding theremaining edges (numbered 2, 4 and 7) to the tree we obtain, respectively,the following loops:(1) 1 3 5 4 2 1, for the edge sequence (1, 5, 6, 3, 2);(2) 1 3 5 4 1, for the edge sequence (1, 5, 6, 4);(3) 3 5 4 3, for the edge sequence (5, 6, 7).Notethat anedgewas giventhenegativesignwhenever its orientationrequiredit. Itfollowsthattheedgeloopmatrixforthenetworkunderconsideration isZ =__1 1 01 0 01 0 00 1 01 1 11 1 10 0 1__.and it is straightforward to check thatBZ = O.It turns out that this elegant method is fairly general and can be appliedto other problems besides the analysis of resistive networks. An importantexample is uid dynamics, in particular the Darcy, Stokes and Oseen prob-lems, whereBrepresentsadiscretedivergenceoperator. Inthiscasethenull basisZiscalledasolenoidal basis, sincethecolumnsof Zspanthesubspace of all discrete solenoidal (i.e., divergence-free) functions. In otherwords, Ccan be regarded as a discrete curl operator (Chang, Giraldo andPerot2002). Inthecaseof nitedierencesonaregulargrid, Bisjustthe incidence matrix of a directed graph associated with the grid, and thecycles (loops) in this graph can be used to construct a sparseZ; see Amitet al. (1981), Burkardt, Hall and Porsching (1986), Chang et al. (2002), HallNumericalsolutionofsaddlepointproblems 37(1985), SamehandSarin(2002)andSarinandSameh(1998). Notethatin this context, solving the system (6.2) fory amounts to solving a Poissonequation. This methodology is not restricted to simple nite dierence dis-cretizations or to structured grids; see Alotto and Perugia (1999), Arioli andManzini (2002, 2003), Arioli et al. (2001), Hall, Cavendish and Frey (1991),Sarin(1997)andSarinandSameh(2003)forapplicationstoavarietyofdiscretization methods on possibly unstructured grids.We note that no oating-point arithmetic is needed to formZ. Further-more, thesparsitypatternofthematrixZTAZcanbeeasilydeterminedand the matrixZTAZassembled rather cheaply, as long as it is sucientlysparse. Thesparsitywill dependontheparticularspanningtreeusedtoformZ. Findingatreethatminimizesthenumberof nonzerosinZisequivalenttondingthetreeforwhichthesumof all thelengthsof thefundamental loops is minimal, which is an NP-hard problem. Nevertheless,manyecientheuristicshavebeendeveloped; seeTarjan(1983)forthefundamental concepts and algorithms. The relative size ofn,m andn mdepends on the discretization scheme used, and on whether the underlyinguid ow problem is posed in 2D or 3D. For lowest-order discretizations in2D,n m andm are comparable, whereasn m is much larger thanm in3D or for certain mixed nite element discretizations. If sparse direct solv-ers are used to solve for the dual variable in (6.1), this makes the null spaceapproach not viable in 3D. In this case iterative solvers must be used, andthe spectral properties of ZTAZ determine the convergence rate. When thematrixZTAZis not formed explicitly, nding appropriate preconditionersfor it requires some cleverness. Some work in this direction can be found inColemanandVerma(2001)andinNashandSofer(1996)forconstrainedoptimizationproblems. SeealsoSaint-Georges, NotayandWarzee(1998)for closely related work in the context of constrained nite element analyses,andBarlow, NicholsandPlemmons(1988), James(1992)andJamesandPlemmons (1990) for earlier work on the use of preconditioned CG methodsin the context of implicit null space algorithms i.e., null space algorithmsin which the matrixZis not formed explicitly.For many mixed nite element formulations of second-order elliptic prob-lems, A is symmetric positive denite and has condition number boundedindependently of the discretization parameterh. In this case, fast CG con-vergence can be obtained by using incomplete factorization preconditionersbased onZTZ: see Alotto and Perugia (1999). Point and block Jacobi pre-conditioners constructed without explicitly forming ZTAZ have been testedin the nite element solution of the potential uid ow problem (2.18)(2.20)in Arioli and Manzini (2003).Null space methods have been used for a long time in structural optimiz-ation. Some relevant references in this area include Cassell, Henderson andKaveh(1974), HendersonandMaunder(1969), Kaveh(1979, 1992, 2004)38 M.Benzi,G.H.GolubandJ.Liesenand Pothen (1989). In this case B is an equilibrium matrix associated withNewtons Third Law (i.e., action and reaction are equal and opposite). NowBis no longer an incidence matrix, but many of the above-described con-ceptscanbeextendedtothismoregeneral situation. Algorithmstonda null basis forBdeveloped by Coleman and Pothen (1987) consist of twophases, a combinatorial one and a numerical one. In the rst phase, a max-imummatchinginthebipartitegraphof Bisusedtolocatethenonzeroentriesinthenull basis. Inthesecondphase(notneededwhenBisanincidence matrix), the numerical values of the nonzero entries in the basisare computed by solving certain systems of equations.Whenadditional structure, suchas bandedness, is present inB, it isusuallypossibletoexploit it soas todevelopmoreecient algorithms.Banded equilibrium matrices often arise in structural engineering. The so-called turnbackalgorithm(Topcu 1979) can be used to compute a bandedZ; seealsoKanekoetal.(1982), whereaninterpretationoftheturnbackalgorithm in terms of matrix factorizations is given, and Gilbert and Heath(1987)foradditional methodsmotivatedbytheturnbackalgorithm. Wealso mention Plemmons and White (1990) for approaches based on dierentgraph-theoreticconceptswithafocusonparallelimplementationaspects,and Chow, Manteuel, Tong and Wallin (2003) for another example of howstructure in the constraints can be exploited to nd a null basis resulting ina sparse reduced matrixZTAZ.One problem that may occur in the null space method is that a computednull basis matrixZmay be very ill-conditioned, even numerically rank de-cient. One way to avoid this problem is to compute a Z with orthonormalcolumns, which would be optimally conditioned. An orthonormal null basisforBcan be computed by means of the QR factorization as follows. LetBT= Q_RO_,whereQ isn n orthogonal andR ismm, upper triangular and nonsin-gular. Thenthe rst mcolumns of Qformanorthonormal basis forrange(BT)andtheremainingn mcolumnsformanorthonormal basisforrange(BT)=ker(B). Therefore, if qidenotestheithcolumnof Q,thenZ = _qm+1qm+2. . . qnisthedesiredmatrix. Ofcourse, inthesparsecasespecialorderingtech-niques must be utilized in order to maintain sparsity inZ(Amestoy, Duand Puglisi 1996, Matstoms 1994). The fact that the columns of Z are ortho-normal is advantageous not only from the point of view of conditioning, butalso for other reasons. For example, in the computation of thin-plate splinesNumericalsolutionofsaddlepointproblems 39it is often required to solve saddle point systems of the form_A+I BTB O_ _xy_ =_f0_, (6.4)whereAissymmetric, >0isasmoothingparameter, andBhasfullrow rank; see Sibson and Stone (1991). Usually, problem (6.4) needs to besolved for several values of. IfZis a null basis matrix with orthonormalcolumns, the coecient matrix in the reduced system (6.1) is ZT(A+I)Z =ZTAZ + I. If n m is so small thata spectral decompositionZTAZ=UUTofZTAZcan be computed, then for any we haveZT(A+I)Z =U( +I)UTand the reduced linear systems can be solved eciently.One further advantage of having an orthonormal null basis is that the re-duced system (6.1) is guaranteed to be well-conditioned if A is. For example,if Aissymmetricpositivedeniteandhasconditionnumberboundedin-dependently of mesh size, the same is true ofZTAZand therefore the CGmethod applied to (6.1) converges in a number of iterations independent ofmesh size, even without preconditioning; see Arioli and Manzini (2002) foran example from groundwater ow computations. This property may fail tohold if Z does not have orthonormal columns, generally speaking; see Arioliand Manzini (2003) and Perugia and Simoncini (2000).Sparse orthogonal schemes have been developed by Berry, Heath, Kaneko,Lawo, Plemmons and Ward (1985), Gilbert and Heath (1987), Heath et al.(1984)andKanekoandPlemmons(1984)inthecontextofstructuralop-timization, and by Arioli (2000), Arioli and Manzini (2002) and Arioli et al.(2001) in the context of mixed-hybrid nite element formulations of the po-tential uidowproblem(2.18)(2.20). Aparallel orthogonal null spacescheme has been presented by Psiaki and Park (1995) for trajectory optim-ization problems in quadratic dynamic programming. One limitation of theQRfactorizationapproachisthatthenullbasesobtainedbythismethodare often rather dense compared to those obtained by other sparse schemes;indeed,thesparsestorthogonalnullbasismaybeconsiderablylesssparsethananarbitrarynullbasis: seeColemanandPothen(1987)andGilbertandHeath(1987). Hence, thereisatrade-obetweengoodconditioningproperties and sparsity.Erroranalysesof variousnull spacemethodshavebeencarriedoutbyCoxandHigham(1999a)fordenseproblems, andbyArioli andBaldini(2001)forthesparsecase. SeefurtherBarlow(1988),BarlowandHandy(1988), Bj orckandPaige(1994), CoxandHigham(1999b), FletcherandJohnson(1997), Gulliksson(1994), GullikssonandWedin(1992), Houghand Vavasis (1997), Sun (1999) and Vavasis (1994) for stable implementa-tionsandothernumerical stabilityaspectsofalgorithmsforsaddlepointproblems, in particular for equality-constrained and weighted least squaresproblems. Finally, appropriate stopping criteria for the CG method applied40 M.Benzi,G.H.GolubandJ.Liesento the reduced system (6.1) in a nite element context have been given byArioli and Manzini (2002).7. CoupleddirectsolversInthissectionwegiveabriefoverviewofdirectmethodsbasedontrian-gular factorizations of /. Our discussion is limited to the symmetric case(A=AT, B1=B2andC=CT, possiblyzero). Asfarasweknow, nospecialized direct solver exists for nonsymmetric saddle point problems. Al-thoughsuchproblemsareoftenstructurallysymmetric, inthesensethatthenonzeropatternof /issymmetric, someformof numerical pivotingisalmostcertainlygoingtobeneededforstabilityreasons; suchpivotingwould in turn destroy symmetry. See Du, Erisman and Reid (1986) for atreatment of direct methods for general sparse matrices.There are several ways to perform Gaussian elimination on a symmetric,possibly indenite matrix in a way that exploits (and preserves) symmetry.A factorization of the form/ = QTLTLTQ, (7.1)where Q is a permutation matrix, L is unit lower triangular, and T a blockdiagonal matrix with blocks of dimension 1 and 2 is usually referred to asan LDLTfactorization. The need for pivot blocks of size 2 is made clear bythe following simple example:/ =__0 1 11 0 11 1 0__,forwhichselectingpivotsfromthemaindiagonalisimpossible. Diagonalpivoting may also fail on matrices with a zero-free diagonal due to instabil-ities. The use of 2 2 pivot blocks dates back to Lagrange (1759). In 1965,W. Kahan(incorrespondencewithR. deMeersmanandL. Schotsmans)suggested that Lagranges method could be used to devise stable factoriza-tions for symmetric indenite matrices. The idea was developed by Bunchand Parlett (1971), resulting in a stable algorithm for factoring symmetricindenite matrices at a cost comparable to that of a Cholesky factorizationforpositivedeniteones. TheBunchParlettpivotingstrategyisakintocomplete pivoting; in subsequent papers (Bunch 1974, Bunch and Kaufman1977), alternative pivoting strategies requiring onlyO(n2) comparisons fora densenn matrix have been developed; see also Fletcher (1976b). TheBunchKaufmanpivotingstrategy(BunchandKaufman1977)iswidelyaccepted as the algorithm of choice for factoring dense symmetric inden-ite matrices. In the sparse case, the pivoting strategy is usually relaxed inNumericalsolutionofsaddlepointproblems 41order to maintain sparsity in L. Several sparse implementations are avail-able; see Du, Gould, Reid, Scott and Turner (1991), Du and Reid (1983,1995) and Liu (1987). In this case the permutation matrix Q is the result ofsymmetric row and column interchanges aimed at preserving sparsity in thefactors as well as numerical stability. While the BunchKaufman algorithmis normwise backward stable, the resulting factors can have unusual scaling,whichmayresultinadegradationoftheaccuracyofcomputedsolutions.AsreportedinAshcraft, GrimesandLewis(1998), suchdicultieshavebeen observed in the solution of saddle point systems arising in sparse non-linear optimization codes. We refer the reader to Ashcraft et al. (1998) forathoroughdiscussionof suchaccuracyissuesandwaystoaddresstheseproblems; see also Vavasis (1994).We note that whenA is positive denite andBhas full rank, the saddlepoint matrix / admits an LDLTfactorization with Tdiagonal and Q = I(i.e., nopivotingis needed). Indeed, since Ais positivedeniteit canbedecomposedas A=LADALTAwithLAunitlowertriangularandDAdiagonal (andpositivedenite); furthermoretheSchurcomplement S=(C +BA1BT) is negative denite and therefore it can be decomposed asS = LSDSLTS. Hence, we can write/ =_A BTB C_ =_LAOLBLS_ _DAOO DS_ _LTALTBO LTS_ = LTLT, (7.2)where LB=BLTAD1A; notethat BA1BT=LBDALTB. Inpractice,however, thefactorswill beratherdensewiththeoriginal ordering, andsymmetric permutations have to be used in order to preserve sparsity. Notethat LS and LB will be completely full if the Schur complement is. However,notall sparsity-preservingpermutationsareacceptable. Itcanbeshownthat there exist permutation matrices Q such that Q/QTdoes not have anLDLTfactorization with T diagonal. Furthermore, some permutations maylead to numerical instability problems.For manysymmetricindenitecodes thefactorizationconsists of twophases, asymbolicandanumericone. Inthesymbolicphase, aninitialll-reducing ordering is computed based on the structure of / only. This isoften some variant of minimum degree or nested dissection (Du et al. 1986).In the numeric phase,the actual factorization is computed. Frequently inthe course of this phase, the pivot order from the symbolic phase may haveto be altered for numerical stability reasons. There are, however, a few ex-ceptions to this rule. An important one is the quasidenite case discussedin Section 3.3, i.e., when C (as well as A) is symmetric positive denite. Inthis case Q/QTalways has an LDLTfactorization with T diagonal, regard-less of the choice of Q;see Vanderbei (1995). This is an important result:it suggests that the ll-reducing ordering computed in the symbolic phaseof the factorization will not need to be altered in the course of the numeric42 M.Benzi,G.H.GolubandJ.Liesenphase because of stability concerns. Since no pivoting is used in the numericphase, it is possible to exploit all the features of modern supernodal sparseCholesky factorization codes (Ng and Peyton 1993). The resulting algorithmis more ecient than performing a BunchParlett or BunchKaufman fac-torization. Numerical stability considerations in Vanderbei (1995) suggestthat the resulting factorization is usually suciently accurate. A stabilityanalysis was given in Gill et al. (1996), where the close relationship between/anditsnonsymmetricpositivedeniteform(3.10)wasusedtogetherwith results in Golub and Van Loan (1979) to derive stability conditions.Afurtherexception(withC=O) hasbeenidentiedbyTuma(2002).Foralargeclassof saddlepointmatricesarisingfrommixedandhybridnite element discretizations it is possible to prove the existence of static,ll-reducing pre-orderings Q such that the permuted matrix Q/QThas theLDLTfactorization with Tdiagonal. Such pre-orderings are characterizedin terms of conditions on the resulting elimination tree. The factorizationcan be carried out in three phases: a rst, symbolic phase in which an initialll-reducing ordering is computed and the corresponding elimination tree isbuilt; asecondphase, alsosymbolic, wheretheinitial orderingismodi-ed so that the permuted matrix satises the conditions that guarantee theexistence of the factorization;and a nal, numeric phase where the LDLTfactorization itself is computed. The numerical experiments in Tuma (2002)show that this is an eective approach. As in the quasidenite case, no nu-merical stability problems have appeared in practice; however, a formal erroranalysis has not yet been carried out. We mention that examples of saddlepointsystemsthatcausedicultiesforsymmetricindenitefactorizationalgorithms have been pointed out in Vavasis (1994).For the general case, sophisticated strategies for computing sparse LDLTfactorizations with 11 and 22 pivot blocks have been developed overmany years by Du and Reid together with several collaborators; see Du(1994), Du etal. (1986), Du etal. (1991), Du and Pralet (2004), Duand Reid (1983,1995,1996). This work has led to a series of widely usedcodesthatarepartoftheHSLlibrary; seeSection12forinformationonhow to access these codes. The rst is MA27, developed in the early 1980s;thesecondisMA47, acodegearedtowardssymmetricindenitesystemsinsaddlepointform(withC=O); latercameMA57and, recently, theMA67 code. All these codes, except for MA67, are multifrontal codes. Theneed for codes specically designed for saddle point systems (withC = O)is clear when one considers the presence of the zero block in position (2,2).Clearly,anyformofsymmetricpivotingmustberestrictedsothatpivotsarenotchosenfromthezeroblock. Failuretodosoduringthesymbolicphaseleadstoaverylargenumberof pivotorderalterationsduringthenumericfactorizationphase, dramaticallyslowingdownthecomputation.Furthermore, the structure of the matrix during the subsequent factorizationNumericalsolutionofsaddlepointproblems 43steps must also be taken into account in order to avoid excessive ll-in. ThecodeMA47hasbeendesignedwithbothofthesegoalsinmind; seeDu(1994) and Du and Reid (1996) for details. The code MA67, also gearedtowardssaddlepointsystems, isbasedondesignconceptsthatarequitedierent from those of the previous codes; as already mentioned it is not amultifrontal code, and furthermore it does not have separate symbolic andnumeric phases. Instead, the numerical values of the entries are taken intoaccountduringtheselectionof thepivots. AMarkowitz-typestrategyisused to balance sparsity and numerical stability needs. Unfortunately, theextensivecomparisonofHSLcodesperformedinGouldandScott(2004)indicates that MA67 is generally inferior to its predecessors.Othersparsedirectsolversforsymmetricindenitesystems, basedondierent design principles, exist; see for instance the recent reports (Mesharand Toledo 2005, Schenk and G artner 2004), and Section 12 below. Whilethese codes have not been developed specically for saddle point matrices,theymayworkquitewell onsuchproblems. For instance, SchenkandG artner(2004)reportthattheircodefactorsasaddlepointmatrixfromoptimization of order approximately 2 million (with 6 million nonzeros) inless than a minute on a 2.4 GHz Intel 32-bit processor, producing a factorwith about 1.4 108nonzeros.Althoughfairlyreliableinpractice, sparseLDLTfactorizationmethodsare not entirely foolproof. Besides the examples given in Vavasis (1994), afewfailurestocomputeacceptablyaccuratesolutionshavebeenreportedinGouldandScott(2004), evenwiththeuseofiterativerenement; seealsoSchenkandG artner(2004). Nevertheless, sparseLDLTmethodsarethesolversofchoiceinvarioussparseoptimizationcodes, wheretheyareoften preferred to methods based on Schur complement reduction (normalequationsmethods)forbothstabilityandsparsityreasons. Sparsedirectsolvers have been less popular in the numerical solution of PDE problemsbecauseoftheirintrinsicstorageandcomputationallimitations, althoughthesesolverscanbequitecompetitivefor2Dproblems; see, e.g., Perugiaetal. (1999). For saddle point systems arising from PDE problems on 3Dmeshes, it is necessary to turn to iterative methods.8. StationaryiterationsWebeginourdiscussionof iterativealgorithmswithstationaryschemes.Thesemethods havebeenpopular for years as standalone solvers, butnowadays they are most often used as preconditioners for Krylov subspacemethods(equivalently, theconvergenceof thesestationaryiterationscanbeacceleratedbyKrylovsubspacemethods.) Another commonuseforstationaryiterationsisassmoothersformultigridmethods; wereturntothis in Section 11.44 M.Benzi,G.H.GolubandJ.Liesen8.1. The ArrowHurwicz and Uzawa methodsTherstiterativeschemesforthesolutionof saddlepointproblemsof arather general type were the ones developed by the mathematical economistsArrow, Hurwicz and Uzawa (see Arrow et al. (1958)). The original papersaddressed the case of inequality constraints; see Polyak (1970) for an earlystudy of these methods in the context of the equality-constrained problem(1.4)(1.5).The ArrowHurwicz and Uzawa methods are stationary schemes consist-ingof simultaneousiterationsforbothxandy, andcanbeexpressedinterms of splittings of the matrix /. By elimination of one of the unknownvectors, theycanalsobeinterpretedasiterationsforthereduced(Schurcomplement)system. Hence, thesealgorithmsmayberegardedbothascoupled and as segregated solvers.We start with Uzawas method (Uzawa 1958), which enjoys considerablepopularity in uid dynamics, especially for solving the (steady) Stokes prob-lem(FortinandGlowinski1983,Glowinski1984,Glowinski2003,Temam1984, Turek1999). Forsimplicity, weassumeAisinvertibleandwede-scribe the algorithm in the caseB1 =B2 =BandC=O. GeneralizationtoproblemswithB1 ,=B2or C,=Oisstraightforward. Startingwithinitial guessesx0 andy0, Uzawas method consists of the following couplediteration:_Axk+1 = f BTyk,yk+1 = yk +(Bxk+1g),(8.1)where>0isarelaxationparameter. AsnotedinGolubandOverton(1988, page591)(seealsoSaad(2003, page258)), thisiterationcanbewrittenintermsofamatrixsplitting / = T Q, i.e.,asthexed-pointiterationTuk+1 = Quk +b,whereT =_A OB 1I_, Q =_O BTO 1I_, and uk =_xkyk_. (8.2)Note that the iteration matrix isT= T1Q =_O A1BTO I BA1BT_,and therefore the eigenvalues of T are all real (and at leastn of them areexactly zero).On the other hand, if we use the rst equation in (8.1) to eliminatexk+1from the second one we obtainyk+1 = yk + (BA1f g BA1BTyk), (8.3)Numericalsolutionofsaddlepointproblems 45showingthat Uzawas methodis equivalent toastationaryRichardsoniteration applied to the Schur complement systemBA1BTy = BA1f g. (8.4)If Ais symmetric and positive denite, so is BA1BT. Denoting the smallestand largest eigenvalues of BA1BTby min and max, respectively, it is wellknown that Richardsons iteration (8.3) converges for all such that0 < 0, one can in principle use a Cholesky factorization or the CG methodtocomputex(). Unfortunately, suchanapproachcannotberecommen-ded, since the condition number ofA+BTB grows like a (possibly large)multiple of ; see Glowinski (1984, pages 2223) and Van Loan (1985) for ananalysis of the penalty method applied to equality-constrained least squares48 M.Benzi,G.H.GolubandJ.Liesenproblems. In practice, for large values of the coecient matrix A+BTBisdominatedbythe(highlysingular)termBTB,andaccuratesolutionsof (8.10) are dicult to obtain.This drawback of the penalty method can be overcome in two ways. Oneway is to observe that (x(), y()) is the unique solution of the regularizedsaddle point system_A BTB I_ _xy_ =_fg_, = 1. (8.11)Stablesolutionof thislinearsystemisnowpossible, forinstanceusingasparsedirectsolver; thisapproachispopularinoptimization. However,using a direct solver is not always feasible.Theother optionis tomodifythepenaltymethodsoas toavoidill-conditioning. This leads to the method of multipliers, developed independ-entlybyArrowandSolow(1958, page172), Hestenes(1969)andPowell(1969). Afurtheradvantageof thismethod, whichcombinestheuseofpenaltywithLagrangemultipliers, isthatitproducestheexact solution(x, y) rather than an approximate one. The method of multipliers can bedescribed as follows. Select> 0 and consider the augmented LagrangianL(x, y) = J(x) + (Bx g)Ty +2 |Bx g|22. (8.12)Given an approximation yk for the Lagrange multiplier vector y, we computetheminimumxk+1ofthefunction(x) L(x, yk). Thisrequiressolvingthe linear system(A+BTB)x = f BTyk +BTg. (8.13)Now we use the computed solution xk+1 to obtain the new Lagrange multi-plier approximationyk+1 according toyk+1 = yk + (Bxk+1g),and so on. Clearly, the method of multipliers is precisely Uzawas iterationapplied to the saddle point system_A+BTB BTB O_ _xy_ =_f +BTgg_, (8.14)which has exactly the same solution (x, y) as the original one. Note thatthe parameter does double duty here, in that it appears both in the den-ition of the augmented Lagrangian and as the relaxation parameter for theUzawa iteration. As we know from our discussion of Uzawas method, theiteration converges for (0, 2/) where denotes the largest eigenvalue ofthe Schur complement B(A+BTB)1BT. This interval becomes unboun-ded, and the rate of convergence arbitrarily large, as . Again, takingtoolargeavalueof resultsinextremeill-conditioningofthecoecientNumericalsolutionofsaddlepointproblems 49matrixin(8.13). Itisnecessarytostrikeabalancebetweentherateofconvergence of the method and the conditioning properties of (8.13). Thechoice of and many other aspects of the multiplier method have been dis-cussed by a number of authors, including Bertsekas (1982), Chen and Zou(1999), FortinandGlowinski(1983), Greifetal.(2005), Hestenes(1975),Luenberger (1984), Zienkiewicz, Vilotte, Toyoshima and Nakazawa (1985).See further Awanou and Lai (2005) for a study of the nonsymmetric case.Another possibility is to combine the augmented Lagrangian method witha (preconditioned) ArrowHurwicz scheme; see Fortin and Glowinski (1983,page 26) and Kouhia and Menken (1995) for an application of this idea toproblems in structural mechanics.8.3. Other stationary iterationsIn addition to the foregoing algorithms, a number of other stationary itera-tions based on matrix splittings / = T Q can be found in the literature.Inparticular, SOR- andblock-SOR-typeschemeshavebeenproposedinStrikwerda(1984)fortheStokesproblem, inBarlowet al. (1988), Benzi(1993)andPlemmons(1986)forstructuralanalysiscomputations, andinChen(1998), Golub, WuandYuan(2001)andLi, Li, EvansandZhang(2003) for general saddlepoint systems. Someof theseschemes canbeinterpreted as preconditioned or inexact variants of the classical Uzawa al-gorithm. Alternating-direction iterative methods for saddle point problemshave been studied in Brown (1982),Douglas Jr.,Dur an and Pietra (1986,1987). OtherstationaryiterativemethodsforsaddlepointproblemshavebeenstudiedinBank, Welfert andYserentant (1990), Benzi andGolub(2004), Dyn and Ferguson (1983), Golub and Wathen (1998) and Tong andSameh (1998); since these methods are most often used as preconditionersfor Krylov subspace methods, we defer their description to Section 10.9. KrylovsubspacemethodsInthissectionwediscussKrylovsubspacemethodsforsolving(precon-ditioned) saddlepoint problems. Our goal is not tosurveyall existingmethodsandimplementations(morecompletesurveyscanbefound,e.g.,inthemonographsbyGreenbaum(1997), Saad(2003)andvanderVorst(2003)orinthepapersbyEiermannandErnst(2001)andFreundetal.(1992)), buttodescribethemainpropertiesof themostcommonlyusedmethods. We discuss the general theory, the main convergence results, andimplementation details. For simplicity, we describe the basics of Krylov sub-space methods for the unpreconditioned and nonsingular system (1.1)(1.2).Thelatersectionswill describethegeneral ideasofpreconditioning(Sec-tion10), anddierentpreconditioningtechniquesspecicallyconstructedfor (generalized) saddle point systems (Sections 10.110.4).50 M.Benzi,G.H.GolubandJ.Liesen9.1. General theorySuppose that u0 is an initial guess for the solution u of (1.1)(1.2), and denetheinitial residual r0=b /u0. Krylovsubspacemethodsareiterativemethods whosekth iterateuksatisesuk u0 +/k(/, r0), k = 1, 2, . . . , (9.1)where/k(/, r0) span r0, /r0, . . . , /k1r0 (9.2)denotes thekth Krylov subspace generated by / andr0. It is well knownthat the Krylov subspaces form a nested sequence that ends with dimensiond dim/n+m(/, r0) n +m, i.e.,/1(/, r0) /d(/, r0) == /n+m(/, r0).In particular, for eachk d, the Krylov subspace /k(/, r0) has dimensionk. Becauseofthekdegreesoffreedominthechoiceoftheiterateuk, kconstraintsarerequiredtomakeukunique. InKrylovsubspacemethodsthis is achieved by requiring that the kth residual rk = b/uk is orthogonalto ak-dimensional space (k, called the constraints space:rk = b /uk r0 +//k(/, r0), rk (k. (9.3)OrthogonalityhereismeantwithrespecttotheEuclideaninnerproduct.The relations (9.1)(9.3) show that Krylov subspace methods are based onageneral typeof projectionprocess thatcanbefoundinmanyareasofmathematics. Forexample, inthelanguageoftheniteelementmethod,we may consider /k(/, r0) the test and (kthe trial space for constructingtheapproximatesolutionuk. Inthissensetheprojectionprocess(9.1)(9.3)correspondstothePetrovGalerkinframework; see,e.g.,Quarteroniand Valli (1994, Chapter 5). The interpretation of Krylov subspace methodsas projection processes was popularized by Saad in a series of papers in theearly 1980s (Saad 1981, 1982). A survey of his approach can be found in hisbook, Saad (2003). For additional analyses of Krylov subspace methods intermsofprojectionssee,e.g.,BarthandManteuel(1994)andEiermannand Ernst (2001).Knowing the properties of the system matrix / it is possible to determineconstraint spaces (kthat lead to uniquely dened iteratesuk,k = 1, 2, . . . ,in (9.1)(9.3). Examples for such spaces are given in the following theorem.Theorem9.1. SupposethattheKrylovsubspace /k(/, r0)hasdimen-sionk. If(C) / is symmetric positive denite and (k = /k(/, r0), or(M) / is nonsingular and (k = //k(/, r0),Numericalsolutionofsaddlepointproblems 51thenthere exists a uniquely denediterateukofthe form (9.1) for whichthe residualrk = b /uksatises (9.3).Proof. See Saad (2003, Proposition 5.1).Items (C) and (M) in Theorem 9.1 represent mathematical characteriza-tions of the projection properties of well-known Krylov subspace methods.Item (C) characterizes the conjugate gradient (CG) method of Hestenes andStiefel for symmetric positive denite matrices (Hestenes and Stiefel 1952).Note that if / is not symmetric positive denite, an approximate solutionuksatisfying both (9.1)(9.2) and (9.3) with (k = /