1982 - algorithm 583 lsqr sparse linear

8/13/2019 1982 - Algorithm 583 LSQR Sparse Linear

1/15

ALGORITHM 583LSQR: Sparse Linear Equations and LeastSquares ProblemsCHRISTOPHER C. PAIGEMcGill University Canad aandMICHAEL A. SAUNDERSStanford UniversityCategor ie s and Sub jec t Descr ipto rs : G.1 .3 [N um er i ca l A na l ys i s ] Numer ica l L inea r A l g e b r a - - l i n e a rsys tem s d t rec t an d t te ra t tve me thods ) , G.3 [ Ma t h e m a t i c s o f C o m p u t i n g ] : P r o ba b il it y a nd S t a ti s -t m s - - s t a t z s h c a l c o m p u t t n g, s t a t t s t t c a l s o ft w a re , G. m [ M a t h e ma t i c s o f C o m p u t i n g ] : Mi s ce l la -n e o u s - - F O R T R A N p r og r a m u n tt sGenera l Te rm s Algor i thmsAddi t ional Key Words and Phrases : Analysis of var iance , conjugate-gradient method, leas t squares ,l inear equatmns, regress ion, sparse matr ix

1 . INTRODUCTIONL S Q R f i n d s a s o l u t i o n x t o t h e f o ll o w i ng p r o b l e m s :

U n s y m m e t r i c e q u a ti o n s : s o lv e A x = b (1.1)L in e a r l e a s t squa re s : min im iz e [[Ax - b [[2 (1.2)

w he re A i s a ma t r i x w i th m row s a nd n c o lumns , b i s a n m-ve c to r , ~ i s a s c a l a r ,a n d t h e g i v e n d a t a A , b , ) , a r e r e a l . T h e m a t r i x A w i l l n o r m a l l y b e l a r g e a n ds p a rs e . I t is d e em e d b y m e a n s o f a u s e r - w r i tt e n s u b r o u t i n e A P R O D , w h o s eReceived 4 June 1980; revised 23 September 1981, accepted 28 February 1982This work was suppor ted by N a tu ra l Sc iences and E ng inee ring Resea rch Counc i l o f Canada Gran tA8652, by the Ne w Zealand D epa r tm ent of Scient i fic and Industr ia l Research; and by U S. Nat ion alScience Foundat ion Grants MCS-7926009 and ECS-8012974, the D epar tme nt o f Energ y underContra ct AM03-76SF00326, PA No. DE-AT03-76ER72018, the Off ice of Nav al Researc h u nderContra ct N00014-75-C-0267, and the A rm y Research Off ice under Con tract DAA29-79-C-0U0,Auth ors ' addresses : C. C. Paige , School of Com puter Science , McG il l Univers i ty , Montreal , Quebec,Canada H3A 2K6; M. A Saundem, D epar tm ent of Operat ions Research, Stanford Univers i ty ,Stanford, CA 94305.Permmsion to copy withou t fee a ll or par t of th is mater ia l i s granted provided th at the copies are notmad e or d is t r ibuted for d i rec t comm ercia l advantage , the ACM copyright not ice and th e t i t le of thepubl ica t ion and i ts da te appear , and notme is g iven that copying is by permiss ion of the Associa t ionfor Computing Machinery . To copy otherwise , or to republ ish , requires a fee and/or specif icpermission. 1982 ACM 0098-3500/82/0600-0[95 $00 75

ACM Transactions on Mathem atical Software, Vol. 8, No. 2, Jun e 1982, Pages 195-209.


2/15

196 AlgorithmsTable I. Comparison of CGLS and LSQR

Storage Work per iterationCGLS, h = 0 2m 4- 2n 2m + 3nCGLS, )~ ~ 0 2m 4- 2n 2m + 5nLSQ R, any )~ m 4- 2n 3m 4- 5n

e s s e n ti a l f u n c t i o n is to c o m p u t e p r o d u c t s o f t h e f o r m A x a n d AWy f o r g i v e nv e c t o r s x a n d y .

P rob l e ms (1 . 1 ) a nd (1 . 2 ) a re t r e a t e d a s spe c i a l c a se s o f (1 . 3 ) , w h ic h w e sha l lw r i t e a s

A n e a r li e r su c c es s fu l m e t h o d f o r s u c h p r o b l e m s i s t h e c o n j u g a t e - g r a d i e n t m e t h o df o r l e a s t s q u a r e s s y s t e m s g i v e n b y H e s t e n e s a n d S t i e f e l [ 3 ] . ( T h i s m e t h o d i sd e s c r i b e d a s a l g o r i t h m C G L S i n [ 6 , s e c t . 7 . 1 ] . ) C G L S a n d L S Q R a r e i t e r a t i v em e t h o d s w i t h s i m il a r q u a li t a t iv e p ro p e r t i es . T h e i r c o m p u t a t i o n a l r e q u i r e m e n t sa r e s u m m a r i z e d i n T a b l e I . I n a d d i t i o n t h e y r e q u i r e a p r o d u c t A x a n d a p r o d u c tA T y e a c h i t e r a t i o n .

I n o r d e r t o a c h i e v e th e s t o r a g e s h o w n f o r L S Q R , w e a s k t h e u s e r t o i m p l e m e n tt h e m a t r i x - v e c to r p r o d u c t s i n th e f o r m

y , ,- - y + A x a n d x -- x + AT y, (1.5)w h e r e


3/15

Algorithms 197L S Q R u s e s a n a l g o r i t h m o f G o l u b a n d K a h a n t o r e d u c e A t o l o w e r b i d ia g o n a lf o r m . T h e q u a n t i t i e s p r o d u c e d f r o m A a n d b a f t e r k + 1 s t e p s o f t h e b i d i a g o n a l -

i za t ion (p rocedu re Bid iag 1 [6] ) a re

Uk+l ~ [UI~ U2~ -- . ~ Uk+l]~Vk+l = [V l , V2, . , Vk+l] , B k ~--

(XI~ 2 O~2fi3

~ k + l

(2.1)

T h e k t h a p p r o x i m a t i o n t o t h e s o l u t i o n x is t h e n d e f i n e d t o b e x k = V ~ y k , w h e r ey k s o lv e s t h e s u b p r o b l e m

(2.2)L e t t i n g t h e a s s o c i a t e d r e s i d u a l v e c t o r s b e

t k + l = f i l e 1 - B k y kr k = b - A x k (2.3)

w e f i n d t h a t t h e r e l a ti o n srk = Uk+l tk+ l (2.4)

A W r k = h 2 X k + Olk+lgk+lVk+l

wil l ho ld to m ac h in e a ccurac y , wh ere rk+~ i s the l as t c om po ne n t o f tk+~, an d w ether e fo r e c onc lu de th a t ( rk , xD wi l l be a n acc ep tab le so lu t ion o f (1.4 ) i f thec om pu te d va lue of e ith er II th+l II or I ak+lVk+l I is su ita bl y sma ll .

B j o r c k [ 1] h a s p r e v i o u s l y o b s e r v e d t h a t s u b p r o b l e m (2.2) i s t h e a p p r o p r i a t egenera l i za t ion o f m in l l B k y k - - f l ~e l II, wh en X ~ 0 . He a l so d i scusses met ho ds fo rco mp ut in g yk and xk e f f i c ien t ly fo r va r ious ~ a nd k .In L SQ R we as s um e th a t a s ing le va lue o f )~ i s g iven, and to s ave s to rag e an dwork , we d o no t c om pu te yk , rk, o r tk+~ . Th e o r tho gon a l f ac to r iza t ion

qk J(2.5)

is c o m p u t e d Q T Q k = I; R k upper b id iagona l , k x k ) and th i s wou ld g ive R k y k =DTDT V[ a n d f o r m x k D k f k .k , bu t ins te ad we so lve ~h k = =Th e fac to r iza t ion (2.5) i s fo rm ed s imi la r ly to th e case ) t = 0 in [6 ], excep t th a tt w o r o t a t i o n s a r e r e q u i r e d p e r s t e p i n s t e a d o f o n e . F o r k --- 2 , t h e f a c t o r i z a t i o n

ACM Transac$1ons on Ma them atica l Softw are, Vol. 8, No. 2, June 1982.


4/15

198 Algor i thmsproceeds accord ing to

2 0/2

XI pl 8= 61]

-'* f13

~ 2 O/2 j~2 ~ 2

p~ e2p2

pl

62

Note tha t the f i r s t ~ is ro ta ted in to the d iagona l e leme nt a~. Th is a l te rs the r ight -h a n d s id e f l ~ e ~ to pro duc e ~1, the f i rs t co mp on en t of qk. An al te rnat ive is to rota teinto f12 (and s im ilar ly for la ter h) , s ince th is does no t affect the r igh t-ha nd s ideand i t mor e c lose ly s imula tes the a lgor i thm tha t resu l t s whe n LSQ R is app l ied toand 5 direct ly . However , the rota t ions then have a greater effect on Bk, and inprac t ice th e f ir s t op t ion has p ro ved to g ive marg ina l ly more accura te resul t s.The es t imates requ i red to implement the s topp ing c r i te r ia a re

I I k U = = I l r k l l 2 X = l l x k l l = = I l q k l l = ,'] '4 T ~ * ] l f f i l ] A T r ~ - - X 2 X * l ' = ] a k ~ B k + ~ k ]o k

Th is is a s imple general izat ion of the case h ffi 0 . No addit ional s torage is need edfor qk, s ince only i ts norm is required. In short , a l though the presence of hcompl ica tes the a lgor i thm descr ip t ion , it adds essen t ia lly no th ing to the s to rageand work per i te ra t ion.3 . R E G U L A R I Z A T I O N A N D R E L A T E D W O R KIntr odu cing h as in (1.3) is just on e way of regular izing the solut ion x, in thesense tha t i t can reduce the s ize o f the com put ed so lu t ion and m ake i t s compo-nen ts less sens it ive to changes in the da ta . L SQR is app l icab le when a va lue o fi s know n a pr io r i. Th e va lue is en te red v ia the sub rou t ine pa ram ete r DAM P. Asecond me th od fo r regu la r izing x i s ava ilable th ro ugh LS QR's para me te r ACOND,whic h can cause i tera t ion s to term ina te before I[ xk [I beco mes large. A s imilarappro ach has recen t ly been descr ibed by Wold e t a l. [9] , who g ive an i l lumina t ingin te rpre ta t ion o f the b id iagona liza tion as a par t ia l l eas t squares p rocedure . The i rdescr ip t ion wi l l a l so be usefu l to those who pre fe r the no ta t ion o f mul t ip leregression.Methods for choosing X, and other approaches to regular izat ion, are given in[1, 2, 4, 8] and elsewhere. Fo r a ph iloso phic al discussion, see [7].4 . C O D I N G A P R O DT h e b e s t w a y to c o mp u te y + A x a n d x + ATy depends upon the o r ig in o f thema tr ix A. We shal l i l lustra te a case th at com mo nly ar ises , in whic h A is a sparsematr ix whose nonzero coeff ic ients are s tored by r o w s in a s imple l ist . Let A hav eACM Tra nsact ions on Mathem atm al Software, Voi. 8, No. 2, Jun e 1982


5/15

A l g o r i t h m s 1 9 9

M rows, N columns, and NZ nonzeros. Conceptua lly we need three arraysdimensioned as REA L RA NZ) and IN TEG ER JA NZ), NA M),.whereRA L) is the Lt h nonzero of A, counting across row 1, the n across row 2, andso on;JA L) is the column in which the Lt h nonzero of A lies;NA I) is the number o f nonzero coefficients in the Ith row of A.These quantities ma y be used in a straightforward way, as shown in Figure 1 aFORTRAN implementation). We assume that they are made available toAPROD through COMMON, and t hat the actual array dimensions are suitablylarge.

Blank or labeled COMMON will often be convenient for transmitt ing data toAPROD. Of course, some of the data could be local to APROD.) For greatergenerality, the parameter lists for LSQR and APROD include two workspacearrays IW, RW and their lengths LENIW, LENRW. LSQR does not use theseparameters directly; it just passes them to APROD.Figure 2 illus trates their use on the same example sparse A stored by rows).An auxiliary subroutine APROD1 is needed to make th e code readable. A similarscheme should be used to initialize the workspace parameters prior to callingLSQR.

Returning to the example itself, it may often be natural to store A by c o l u m n srather than rows, using analogous data structures. However, we note that insparse least squares applications, A may have many more rows than columnsM >> N). In such cases it is vital to store A by rows as shown, if the machine

being used has a paged virtual) memory. Random access is the n restricted toarrays of length N rather than M, and page faults will therefore be kept to aminimum.Note also that the ar rays RA, JA, NA are adequate for computing both A x and

A W y ; we do not need to store A by rows a n d by columns.Regardless of the application, it will be apparent when coding APROD for thetwo values of MODE that the matrix A is effectively being defined t w i c e . Great

care must be taken to avoid coding inconsistent expressions y + A l x and x +A T y , where either A1 or A2 is different from the desired A. If A1 ~ As, algori thmLSQR will not converge.) Parameters ANORM, ACOND, and CONLIM providea safeguard for such an event.5 . P R E C O N D I T I O N I N GIt is well known that conjugate-gradient metho ds can be accelerated if a nonsin-gular matrix M is available to approximate A in some useful sense. When A issquare and nonsingular, the system A x - - b is equivalent to bo th of the followingsystems:

M - 1 A ) x c where M c = b; 5.1)A M - I ) z - - b where M x = z . 5.2)

For least squares systems undamped), only the analogue of 5.2) is applicable:minl lAx - b]12 = minll AM-1)z - bH 2, where M x = z . 5.3)

ACM Transac~m ns on Math ema t ica l Sof tware , Vol. 8 , No 2 , Jun e 1982


6/15

2 0 0 Algorithms

Fig. 1 . Com putat ion of y + Ax,x + Ary, where A is a sparsemat r ix s to red compac t ly byrows. For convenience , thedata s t ructure for A is held inC O M M O N .

S U B R O U T I N E A P R O D ( M O D E , M , N , X , Y ,* L E N I W , L E N R W , I W , R W )I N T E G E R M O D E , M , N , L E N I W , L E N R WI N T E G E R I W L E N I W )R E A L X N ) , Y M ) , R W L E N R W )A P R O D P E R F O R M S T H E F O L L O W I N G F U N C T I O N S:

I F M O D E = I , S E T Y = Y + A * XI F M O D E = 2, S E T X " X + A ( T R A N S P O S E ) * Y

W H E R E A I S A M A T R I X S T O R E D B Y R O W S I NT H E A R R A Y S R A , J A , N A . I N T H I S E X A M P L E ,R A , J A , N A A R E S T O R E D I N C O M M O N .R E A L R AI N T E G E R J A , N AC O M M O N R A ( 9 0 0 0 ) , J A ( 9 0 0 0 ) , N A ( I 0 0 0 )I N T E G E R I , J , L , L I , L 2R E A L S U M, Y I , Z E R OZ E R O - 0 . 0L2 - 0I F ( M O D E . N E .I ) G O T O 4 0 0

MO DE - 1 -- SET Y = Y + A*X.DO 200 I - I, M

S U M - Z E R OL1 " L2 + 1L 2 " L 2 + N A ( I )DO 100 L " LI, L2

J .. JA (L )S U M - S U M + R A ( L ) * X ( J )

I 0 0 C O N T I N U EY ( 1 ) = Y ( 1 ) + S U M

2 0 0 C O N T I N U ER E T U R N

M O D E = 2 - - S E T X = X + A ( T R A N S P O S E ) * Y .400 DO 600 I " I, M

Y I = Y ( 1 )L1 " L2 + 1L 2 " L 2 + N A ( I )D O 5 0 0 L " L I , L 2

J " JA(L)X ( J ) " X ( J ) + R A ( L ) * Y I

5 0 0 C O N T I N U E6 0 0 C O N T I N U E

R E T U R NE N D O F A P R O DE N D

A C M Transactions n Mathematical oftware, ol 8, o. 2, June 1982.


7/15

S U B R O U T I N E A P R O D ( M O D E , H , N , X , Y ,* L E N I W , L E N R W , I W , R W )I N T E G E R M O D E , M , N , L E N I W , L E N R WI N T E G E R I W ( L E N I W )R E A L X ( N ) , Y ( M ) , R W ( L E N R W )

A P R O D P E R F O R M S T H E F O L L O W I N G FU N C T I ON S :IF MO D E = I, SE T Y ffi Y + A * XIF MO DE - 2, SE T X ffi X + A( TR AN SP OS E) *Y

W H E R E A I S A M A T R I X S T O R E D B Y R O W S I NT H E A R R A Y S R A , J A , N A . I N T H I S E X A M P L E ,A P R O D I S A N I N T E R F A C E B E T W E E N L S Q R A N DA N O T H E R U S E R R O U T I N E T H A T D O E S T H E W O R K.T H E W O R K S P A C E A R R A Y R W C O N T A I N S R A.T H E F I R S T M C O M P O N E N T S O F I W C O N T A I N N A ,A N D T H E R E M A I N D E R O F I W C O N T A I N S J A.T H E D I M E N S I O N S O F R W A N D I W A R E A S S U M E DT O B E S U F F I C I E N T L Y L A R G E .I N T E G E R L E N J A , L E N R A , L O C J AL O C J A ffi M + ILE NJ A ffi LE NI W - LOC JA + 1LE N RA ffi LE NR WC A LL A P R O D I ( M O D E , M , N , X , Y )

* L E N J A , L E N R A , I W , I W ( L O C J A ) , R W )R E T U R NE N D O F A P R O DE N D

Algorithms . 201

Fig . 2 S a me as F igu r e 1 w i t hthe data structure for A heldin the workspace parameters.

S U B R O U T I N E A P R O DI ( M O D E , M , N , X , Y ,* L E N J A , L E N R A , N A , J A , R A )I N T E G E RI N T E G E RR E A L

M O D E , M , N , L E N J A , L E N R AN A ( M ) , J A ( L E N J A )X ( N ) , Y ( M ) , R A ( L E N R A )

A P R O D I D O E S T H E W O R K F O R A P R O D .I N T E G E R I , J , L , L I , L 2R E A L S U M , Y I , Z E R O

< t h e s a m e c ode a s i n A P R O D i n F igur e 1 >

E N D O F A P R O D IE N D

ACM Transactions on Mathematical Software, Vol. 8, No. 2, June 1982


8/15

0'30I00

o

0o, I

I

r ~

Ir..r..1000

I|Q

00

II

Z0

0

, - 1 0II 0 IU r.r-10K O 0

0 II

0 0 000

II II

0

I

II I1

~.4 0~

00

0~'~ 0

0000 ,-i 0

0

0 0 0 0 0 0 ~ ~0 0 0 0 0 0 0 0 0 0 0 0 00 ~ ~ ~ ~0 ~ 0 0 ~ ~

I I I I I I 1 1 1 1 I I I

I I I I I I I I I I I l l~ 0 ~ ~ 0 ~ ~

0 0 0 0 0 0 0 0 0 0 0 0 0I I IO Q OO O 0 0 0 Q O O O Q Q Q OQ O O O O O O Q O 0 O 0 0

~ 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0I I I IO O O O O 0 0 0 0 0 Q O 0Q O O O O O Q O O 0 O O QO O 0 0 0 0 0 0 0 0 O O 0

~ ~ 0 0 ~ ~| 1 1


9/15

0

00

0

C ~r ~

v

0

0

0

0v

0

I.-.Iu ' l

I

-I

v

0Z

O 0

0 0 0

I I

0

O~ ~'~10 ~ - ~

0

0

0 0 0

c. , I r '~

0 0 o hZo0

0 0 0

O 0

~ 2

CO 0 , - ~O 0

0 0 0a O u ' ~0 0 u ' ~O C )txl r .- .

~ c O 0 00 u~ , .-4

Z

0


10/15

2 0 4 A l g o r i t h m s

W e n o t e o n l y t h a t subroutine LSQR may be appl ied wi thout change t o s y s t e m s( 5. 1) -( 5. 3) . T h e e f f e c t o f M i s l o c al i ze d t o t h e u s e r ' s o w n s u b r o u t i n e A P R O D . F o re x a m p l e , w h e n M O D E - - 1, A P R O D f o r t h e l a s t t w o s y s t e m s s h o u l d c o m p u t ey + AM -t )x b y f i r s t s o lv i n g Mw = x a n d t h e n c o m p u t i n g y + Aw. C l e a r l y i t m u s tb e p o s s i b l e t o s o l v e s y s t e m s i n v o l v i n g M a n d M T v e r y e f f i c ie n t l y .6 . O U T P U TS u b r o u t i n e L S Q R p r o d u c e s p ri n t ed o u t p u t o n f i le N O U T , i f t h e p a r a m e t e rN O U T i s p o s i ti v e . T h i s i s i l l u s t r a t e d i n F i g u r e 3 , i n w h i c h t h e l e a s t s q u a r e sp r o b l e m s o l v ed i s P (2 0 , 10, 1, 1) a s d e f i n ed i n [6 ], w i t h a s l i g h t g en e ra l i z a t i o n t oi n c l u d e a d a m p i n g p a r a m e t e r )~ = 1 0 -2. ( S in g le p r e c i s i o n w a s u s e d o n a n I B M3 7 0 /1 6 8 .) T h e i t e m s p r i n t e d ~ a t t h e k t h i t e r a t i o n a r e a s f o ll o w s.I T N T h e i t e ra t i o n n u m b e r k . R e s u l t s a r e a l w a y s p r i n t e d f o r t h ef i r st 1 0 a n d l a s t 1 0 i t e r a ti o n s . I n t e r m e d i a t e r e s u l t s a r e

p r i n t ed i f m _< 4 0 o r n ___ 4 0, o r i f o n e o f t h e co n v e rg en cec o n d i t i o n s i s n e a r l y s a ti s fi e d . O t h e r w i s e , i n f o r m a t i o n i sp r i n t e d e v e r y 1 0 t h i t e r a t i o n .

X ( 1 ) T h e v a l u e o f t h e f ir s t e l e m e n t o f t h e a p p r o x i m a t e s o l ut i onXk.

F U N C T I O N T h e v a l u e o f t h e f u n c t i o n b e i n g m i n i m i z e d , n a m e l y [] Fk [I - -( 1 1 rk I I 2 + x 2 I I x , 1 1 2 ) / 2C O M P A T I B L E A d i m e n s i o n le s s q u a n t i t y w h i c h s h o u l d c o n v e r g e t o z e ro i fand only i fAx = b i s c o m p a t i b l e . I t is a n e s t i m a t e o f I I ~ k I I /I I b I I , w h i c h d e c r e a s e s m o n o t o n i c a l l y .I N C O M P A T I B L E A d i m e n s i o n l e ss q u a n t i t y w h i c h s h o u l d c o n v e r g e t o z e ro i fand only i f t h e o p t i m u m I I ~ k I I i s n o n z e r o . I t i s a n e s t i m a t eo f I I i T ~ I I / (1 1 t i I I w I F k I I ) , w h i c h i s u s u a l l y not m o n o t o n i c .

N O R M ( A B A R ) A m o n o t o n i c a ll y i nc r ea s in g e s t i m a t e o f I I L i I I~.C O N D ( A B A R ) A m o n o t o n i c a l l y i n c r ea s i ng e s t i m a t e o f c on d( z~ ) =

I I A I I I I / i + I I , t h e c o n d i t i o n n u m b e r o f z{ .ACKNOWLEDGMENTT h e a u t h o r s a r e g r a te f u l t o R i c h a r d H a n s o n f o r s u g g es t io n s t h a t p r o m p t e ds e v er a l i m p r o v e m e n t s t o t h e i m p l e m e n t a t i o n o f L S Q R .REFERENCES

1. BJORCK, A. A bidlagonahzation algorithm for solving ill-posed syste ms of linear equatmns RepLITH-M AT-R-80-33, Dep. Mathem atics, Linkopmg Univ., Lmkoping, Sweden, 1980.2 ELDI~,N, L. Algorithms for the regularization of ill-conditioned lea st squares problems BIT 17(1977), 134-145.3. HESTENES, M.R, AND STIEFEL, E Methods of conjugate gradients for solving hne ar systems. JRes. N.B.S. 49 (1952), 409-436.4 LAWSON,C.L, AND HANSON, R.J Solvmg Least Squares Problems. Prentice-Hail, EnglewoodCliffs, N.J., 1974.5 LAWSON, C.L, HANSON, R J., KINCA1D, D.R , AND KROGH,F T Basic hnear algebra subprogramsfor Fortran usage ACM Trans Math Softw 5, 3 (Sept 1979), 308-323 and (Algorithm } 324-325.6. PAIGE, C.C., AND SAUNDERS, M A LSQR An algorithm for sparse linear equations and sparseleast squares ACM Trans Math Softw 8, 1 (March 1982), 43-71

ACM Tran sacUo ns on Math ematic al Software, Vol 8, No. 2, Jun e 1982


11/15

Algorithms 2057. SMITH, G., AND CAMPBELL, F. A critique of some ridge regression methods. J . Am. S tar . Assoc .

75, 369 (Ma rch 1980), 74-818 VARAH, J.M A practical examination of some numerical meth ods for linear discrete ill-posedproblems S I A M R e v. 2 1 (1979), 100-111.9. WOLD, S , WOLD, H., DUNN, W.J., AND RUHE, A. Th e collin earity pro blem in li near a ndnonlinear regression. The partial least squares (PLS) approach to generalized inverses. Rep.UMINF-83.80, Univ. Ume~, Ume&, Swe den, 1980.

A L G O R I T H M[ A p a r t o f t h e l i s t i n g i s p r i n t e d h e r e . T h e c o m p l e t e l i s t i n g i s a v a i l a b l e f r o m t h eA C M A l g o r i t h m s D i s t r i b u t i o n S e r v i c e ( s e e p a g e 2 27 f o r o r d e r f o r m ) . ]

S U B R O U T I N E L SQR( M , N , A P R O D , D A M P ,i L E N I W , L E N R W , I W , R W,2 U , V , W , X , S E ,3 A T O L , B T O L , C O N L I M , I T N L I M , N O UT ,4 I S T O P , A N O R M , A C O N D , R N O I ~ M , A R N O R M , X N O E M )E X T E R N A LI N T E G E RI N T E G E RR E A L

i

A P R O DM , N , L E N I W , L E N RW , I T N L I M , N O U T , I S T O PI W ( L E N I W )R W ( L E N R W ) , U ( M ) , V ( N ) , W ( N ) , X ( N ) , S E ( N ) ,A T O L , B T O L , C O N L IM , D A M P , A N O R M , A C O N D , R N O R M , A R N O R M , X N O R M

io2.3.4.5.6.7.8.9.

i.ii.12.

L S Q R F I N D S A S O L U T I O N X T O T H E F O L L O W I N G P R O B L E M S . . .i . U N S Y M M E T R I C E Q U A T I O N S - - S O L V E A * X = B2 . L I N E A R L E A S T S Q U A R E S - - S O L V E A * X = B

I N T H E L E A S T - S Q U A R E S S E N S E3. DA MP ED LE AS T SQUA RE S -- SOL VE ( A )* X ffi ( B )( D A M P * I ) ( ~b )

I N T H E L E A S T - S Q U A R E S S E N S EW H E R E A I S A M A T R I X W I T H M R O W S A N D N C O L U M N S , B I S A NM - V E C T O R , A N D D A M P I S A S C A L A R ( A L L Q U A N T I T I E S R E A L ) .T H E M A T R I X A I S I N T E N D E D T O B E L A R G E A N D S P A RS E . I T I S A C C E S S E DB Y M E A N S O F S U B R O U T I N E C A L L S O F T H E F O R M

C A L L A P R O D ( M O D E , M , N , X , Y , L E N I W , L E N R W , I W , R W )W H I C H M U S T P E R F O R M T H E F O L L O W I N G F U N C T I ON S . . .

IF MO D E ffi i, CO MP UT E Y = Y + A*X .IF MO DE ffi 2, COMP UTE X = X + A(TRA NSP OSE )*Y .

T H E V E C T O R S X A N D Y A R E I N P U T P A R A M E T E R S I N B O T H C A S E S .I F M O D E = i , Y S H O U L D B E A L T E R E D W I T H O U T C H A N G I N G X .I F M O D E = 2 , X S H O U L D B E A L T E R E D W I T H O U T C H A N G I N G Y .T H E P A R A M E T E R S L E N IW , L E N R W, I W, R W M A Y B E U S E D F O R W O R K S P A C EA S D E S C R I B E D B E L O W .T H E R H S V E C T O R B I S I N P U T V I A U , A N D S U B S E Q U E N T L Y O V E R W R I T T E N .

ACM Transactions on Mathematical Software, Vol. 8, No. 2, June

13.14.15.16.17.18.19.2~.21.22.23.24.25.26.27.28.29.3#.31.32.33.34.35.36.37.38.39.4.41.42.43.44.45.

1982


12/15

2 0 6 Algor i thmsNOTE. LSQR USES AN ITERATIVE METHOD TO APPROXIMA TE THE SOLUTION.THE NUMBE R OF ITERATIONS REQUIRED TO REACH A CERTAIN ACCURACYDEPENDS STRONGLY ON THE SCALING OF THE PROBLEM. POOR SCALING OFTHE ROWS OR COLUMNS OF A SHOULD THEREFORE BE AVOIDED WHEREPOSSIBLE.FOR EXAMPLE, IN PROBLE M i THE SOLUTION IS UNALTERED BYROW-SCALING. IF A ROW OF A IS VERY SMALL OR LARGE COMPARED TOTHE OTHER ROWS OF A, THE CORRESPONDING ROW OF ( A B ) SHOULDBE SC ALED UP OR DOWN.IN PROBLEMS i AND 2, THE SOLUTION X IS EASILY RECOVEREDFOLLOWING COLUMN-SCALING. IN THE ABSENCE OF BETTER INFORMATION,THE NONZERO COLUMNS OF A SHOULD BE SCALED SO THAT THEY ALL HAVETHE SAME EUCLIDEAN NORM ( E.G . i.@).

IN PROBLEM 3, THERE IS NO FREEDOM TO RE-SCALE IF DAMP ISNONZERO. HOWEVER, THE VALUE OF DAMP SHOULD BE ASSIGNED ONLYAFTER ATTENTION HAS BEEN PAID TO THE SCALING OF A.THE PARAMETER DAMP IS INTENDED TO HELP REGULARIZEILL-CONDITIONED SYSTEMS, BY PREVENTING THE TRUE SOLUTION FROMBEING VERY LARGE. ANOTHE R AID TO REGULARIZ ATION IS PROVIDED BYTHE PARAM ETER ACOND, WHICH MAY BE USED TO TERMINATE ITERATIONSBEFORE THE COMPUTE D S OLUTION BECOMES VERY LARGE.

N O T A T I O N

THE FOLLOWING QUANTITIES ARE USED IN DISCUSSING THE SUBROUTINEPARAMETERS...~AR = A ) , BBAR - B )D ~ * I ) ~ )R = B - A'X, REA R - BBA R - ABAR *XR N O E M - SQRT( NORM(R)**2 + DAMP**2 * NORM(X)**2

= N O R M ( R B A R )RELPR = T H E R E L A T I VE P R E C I S I O N O F F L O A T I N G -P O I N T A R I T H M E T I C

ON THE MACH INE BEIN G USED. FOR EXAMPLE, ON THE IBM 370,RELPR IS ABOUT i. @E-6 AND 1.~D-16 IN SINGLE AND DOUBLEPRECISION RESPECTIVELY.

LSQR MINIMIZES THE FUNCTION RNOEM WITH RESPECT TO X.

PARAMETERS

M INPUT THE NIIMBER OF ROWS IN A.N INPUT THE NIJM~ER OF COLUMNS IN A.APROD EXTERNAL SEE ABOVE.DAMP INPUT THE DAMPING PARAME TER FOR PROBLEM 3 ABOVE.

(DAMP SHOULD BE ~.~ FOR PROBLEMS i AND 2.)ACM Transactions on Mathematical Software, Vol 8, No 2, June 1982.

46.47.48.49.5.51.52.53.54.55.56.57.58.59.6.61.62.63.64.65.66.67.68.69.7.71.72.73.74.75.76.77.78.79.8 ~ .81.82.83.84.85.86.87.88.89.9.91.92.93.94.95.96.97.98.99.

z 2 .z03.


13/15

L E N I WL E N R WI WR W

U M )

V N )W N )X N)SE(N)

A T O L

B T O L

C O N L I M

Algorithms 207I F T H E S Y S T E M A * X = B I S I N C O M P A T I B L E , V A L U E S 1 @6 .O F D A M P I N T H E R A N G E @ T O S Q R T ( R E L P R ) * N O R M ( A ) 1 @7.W I L L P R O B A B L Y H A V E A N E G L I G I B L E E F F E C T . 1 @8 .L A R G E R V A L U E S O F D A M P W I L L T E N D T O D E C R E A S E 1 @9 .T H E N O R M O F X A N D T O R E D U C E T H E ~ E R O F I i @.I T E R A T I O N S R EQUI R E D B Y L SQR. I I i .

112.T H E W O R K P E R I T E R A T I O N A N D T H E S T O R A G E N E E D E D 1 1 3.B Y L SQR A R E T H E S A M E F O R A L L V A L U E S O l D A M P . 1 1 4 .

115.I N P U T T H E L E N G T H O F T H E W O R K S P A C E A R R A Y W . 1 16 .I N P U T T H E L E N G T H O F T H E W O R K S P A C E A R R A Y R W. 1 17.W O R K S P A C E A N I N T E G E R A R R A Y O F L E N G T H L E N I W. 1 1 8.W O R K S P A C E A R E A L A R R A Y O F L E N G T H L E N R W. 1 1 9.

[email protected] O T E. L S Q R D O E S N O T E X P L I C I T L Y U S E T H E P R E V I O U S F O U R 1 2 1.P A R A M E T E R S , B U T P A S S E S T H E M T O S U B R O U T I N E A P R O D F O R 1 2 2.P O S S I B L E U S E A S W O R K SP A C E . I F A P R O D D O E S N O T N E E D 1 2 3.I W O R R W , T H E V A L U E S L E N I W = i O R L E N R W ,, i S H O U L D 1 2 4.B E U S ED , A N D T H E A C T U A L P A R A M E T E R S C O R R E S P O N D I N G T O 1 2 5.I W O R R W M A Y B E A N Y C O N V E N I E N T A R R A Y O F S U I T A B L E T Y P E. 1 26 .

127.T H E R H S V E C T O R B . B E W A R E T H A T U I S 1 2 8 .O V E R - W R I T T E N B Y L SQR. 1 2 9 .

[email protected].

R E T U R N S T H E C O M P U T E D S O L U T I O N X . 1 34 .135.

R E T U R N S S T A N D A R D E R R O R E S T I M A T E S F O R T H E 1 36 .C O M P O N E N T S O F X . F O R E A C H I , S E ( 1 ) I S S E T 137.T O T H E V A L U E E N O R M * S Q R T ( S I G M A ( I , I ) / T ), 1 38 .W H E R E S I G M A ( I , I ) I S A N E S T I M A T E OF THE I-TH 139.D I A G O N A L O F T H E I N V E R S E O F A B A R ( T R A N S P O S E ) * A B A R 1 4 @.AN D T " i IF M .LE. N, 141.

T ffi M - N IF M .GT. N A N D DA MP ffi @, 14 2.T ,, M IF DA MP .NE. @. 143.

144.A N E S T I M A T E O F T H E R E L A T I V E E R R O R I N T H E D A T A 1 4 5.D E F I N I N G T H E M A T R I X A . F O R E X A M P L E , 146 .I F A I S A C C U R A T E T O A B O U T 6 D I G I T S , S E T 1 47.A T O L " I . @E - 6 . 1 4 8 .

149.A N E S T I M A T E O F T H E R E L A T I V E E R R O R I N T H E D A T A 1 5 @.D E F I N I N G T H E R H S V E C T O R B. F O R E X A M P L E, 1 5 1.I F B I S A C C U R A T E T O A B O U T 6 D I G I T S , S E T 1 5 2.B T OL ffi I. @E -6 153.

154.A N U P P E R L I M I T O N C ON D( 'A BA R) , T H E A P P A R E N T 1 5 5.C O N D I T I O N N U M B E R O F T H E M A T R I X A B A R . 1 56 .I T E R A T I O N S W I L L B E T E R M I N A T E D I F A C O M P U T E D 1 57.E S T I M A T E O F C O N D ( A B A R ) E X C E E D S C O N L I M . 1 5 8 .T H I S I S I N T E N D E D T O P R E V E N T C E R T A I N S M A L L O R 1 5 9.Z E R O S IN GU I2d~ V A L U E S O F A O R A B A R F R O M 1 6 @.C O M I N G I N T O E F F E C T A N D C A U S I N G U N W A N T E D G R O W T H 1 6 1.I N T H E C O M P U T E D S O L U T I O N. 1 6 2.

163.C O N L I M A N D D A M P M A Y B E U S E D S E P A R A T E L Y O R 1 6 4.T O G E T H E R T O R E G U L A R I Z E I L L - C O N D I T I O N E D S Y S T E M S . 1 6 5 .

ACM Transact ions on Mathematical Software, Vol. 8, No. 2, June 1982

I N P U T

W O R K S P A C EW O R K S P A C EOUTPUTOUTPUT

INPUT

INPUT

I N P U T

~ ; ~ L ~ I ~


14/15

2 0 8 Algor i thmsCCCCCCCCCCCCC ITNLIM INPUTCCCCC NOUT INPUTCCC ISTOP OUTPUTCC 0CCC 1CCCC 2CCCCC 3CCCCC 4CCCC 5CCCCC 6CCCCC 7CC A N O R M OUTPUTCCCCC

166.N O R M A L L Y , CO N L I M SHOUL D BE I N THE R A N G E 167.1 ~ T O 1/REAR. 1 6 8 .SUGGESTED VALb~ - - 169.CONL IM = 1/(100*RELPR ) FOR COMPA TIBLE SYSTEMS, 170.CON L I M - I/ 10*SQRT R E L PR ) ) FO R L E A S T SQUAR E S. 171.

172,N O T E. IF THE US E R I S N O T CO NCE R N E D ABOUT THE PARAMETERS 173.ATOL, BTOL AND CONLIM, ANY OR ALL OF T H E M M A Y B E S E T 174.T O ZERO. THE E F F E C T W I L L BE THE S A M E A S THE VALUES 175.RELPR, R E L P R A N D I/KELPR RESPECTIVELY . 176.

177A N UPP E R L I M I T O N T H E N U M B E R OF ITERAT IONS. 178.S U G G E S T E D VAL UE -- 179.ITNLI M " N/2 FOR WELL CONDI TION ED SYSTEMS, 180.ITNLI M - 4*N OTHERWISE. 181.

182.FILE N U M B E R F O R PRINT ER. IF POSITI VE, 183.A SUMMARY WILL BE PRIN TED ON FILE NOUT. 184.

185.AN INTEGER GIVING THE REASON FOR TERMINATION... 186.187.

X " 0 IS THE E X ACT S O LUT I O N. 188.N O I T E R A T I O N S W E R E P E R F O R M E D . 189.THE EQUAT I O N S A*X - B A R E PR OBABL YCO MPA T IBL E. N O R M A*X - B) IS SUFFI CIENT LYSMALL, G IVE N THE VA LUE S OF ATOL AND BTOL.T H E S Y S T E M A * X = B I S P R O BA B L Y N O TCOMPATIBLE. A L E A S T - S QUA R E S S O L U T I O N HA SBE E N OBT A I N E D WHICH I S SUFFICI E N T L Y ACCURA T E ,G IVE N T HE VA LUE OF ATOL.A N E S T I M A T E O F CO N D A BA R ) H A S E X C E E D E DCO N L I M. THE S Y S T E M A*X - B APPEA R S T O BEI L L - C O N D I T I O N E D . O T HE R W I S E , T H E R E C O UL D BE A NA N E R R O R I N S UBR O U T I N E A PR O D .T H E E QUA T I O N S A * X - B A R E P R O BA B L YCOMPATIBLE. NOR M(A *X - B) IS AS SMALL ASS E E M S R E A S O N A B L E O N T HI S M A CHI N E .THE S Y S T E M A*X - B I S PR OBABL Y N O TCO MPA T IBL E. A L E A S T - SQUAR E S S O LUT I O N HA SB E E N O B T A I N E D W H I CH I S A S A CCUR A T E A S S E E M SR E A S O N A B L E O N T H I S M A CHI N E .CO N D A BA R ) S E E M S T O B E S O L A R G E T HA T T HE R E I SN O T M UCH PO I N T I N D O I N G FU R T HE R I T E R A T I O N S ,G IVE N THE PR ECI S I O N OF THI S M ACHIN E.THE R E COUL D BE A N E R R O R I N SUBROUT I N E APR O D.T H E I T E R A T I O N LIMIT ITNLLM W A S R E ACHED.A N E S T I M A T E O F THE FR OBE N IUS N O R M OF ABA E.THI S I S THE SQUAR E - R O O T OF THE SUM OF SQUAR E SOF THE E L E M E N T S OF ABA R.IF D A M P I S S M A L L A N D IF THE CO LUM N S OF AR AVE A L L BE E N SCA L E D T O HAVE L E N G TH 1.0,A N O R M S H O UL D I N C R E A S E T O R O U G HL Y S QR T N ) .

190.191.192.193 194.195.196.197.198 199.200.2~1.202.203.204.205.206.207.208.209.21~.211.212.213.214.215.216.217.218.219.220.221.222.223.224.225.226.

ACMTransactions on Mathematical Software, Vol. 8, No. 2, June 1982.


15/15

CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCcCC

ACOND OUTPUT

RNORM OUTPUT

ARNORM OUTPUT

XNORM OUTPUT

A l g o r i t h m s

A RADICALLY DIFFERENT VALUE FOR A N O R M M A YI N D I CATE AN ERR O R IN SUBROUTINE APROD THEREMA Y BE AN INCONSIS T E NCY BETWEE N MOD E S 1 AND 2).AN ESTIMATE OF COND(ABAR), THE CONDITIONNUMB ER OF ABAR. A VERY HIGH VALUE OF ACONDMAY AGAIN INDICATE AN ERROR IN APROD.AN ESTIMATE OF THE FINAL VALUE OF NORM(KBAR),THE FUNCTION BEING MINIMIZED S E E N O T A T I O NABOVE). THIS WILL BE SMALL I F A*X " B HAftA SOLUTION.AN ESTIMATE OF THE FINAL VALUE OFNORM( ABAR(TRANSPOSE)*RBAR ) , T HE NOR M OFTHE RESIDUAL FOR THE USUAL NORMAL EQUATIONS.THIS SHOULD BE SMALL IN ALL CASES. (ARNOR MWILL OFTEN BE SMALLER THAN THE TRUE VALUECOMPUTED FR OM THE OUTPUT VECTOR X. )AN ESTIMATE OF THE NORM OF THE FINALSOLUTION VECTOR X.

SUBROUTINES AND FUNCTIONS USED

USER APRODLSQR NORMLZBLAS SCOFY,S NRM2,SS CAL (SEE LAWSON ET AL. BELOW)

(SNRM2 IS USED ON LY IN NORMLZ)FORTRAN ABS,MOD, SQRT

PRECISION

THE NUMBER OF ITERATIONS REQUIRED BY LSQR WILL USUALLY DECREASEIF THE COMPUTATI ON IS PERFORME D IN HIGH ER PRECISION. T O CONVERTLSQR AND NORMLZ BETWEEN SINGLE- AND DOUBLE-PRECISION, CHANGETHE WORDS

SCOPY, SNRM2, SSCALABS, REAL, SQRT

TO THE APPROPRIATE BLAS AND FORTRAN EQUIVALENTS.

REFERENCES

PAIGE, C,C, AND SAUNDERS, M,A. LSQR: AN A L G O R ~T ~ FOP, SPARSELINEAR EQUATIONS AND SPARSE LEAST SQUARES.ACM TRANSACT IONS ON MATHEM ATICAL SOFTWARE 8, 1 (MARCH 1982).

LAWSON, C.L., HANSON, R.J., KINCA ID, D.R. AN D KROGH, .T.BASIC LINEAR ALGEBRA SUBPROGRAMS FOR FORTRAN USAGE.ACM TRANSA CTIONS ON MATHEM ATICAL SOFTWARE 5, 3 (SEPT 1979),3@8-323 AND 324-325.

LSQR. THIS VERS ION DATE D 22 FEBR UARY 1982.

ACM Transac t ions on Mathem at ica l Sof tware, VoL 8, No. 2 , June

2 0 02 2 7 .2 2 8 .2 2 9 .23~.231.232.233.234.235.236.237.238.239.24@[email protected] 5 6 .257.2 5 8 .2 5 9 .2 6 ~ .261.262.263.264.2 6 5 .2 6 6 .2 6 7 .2 6 8 .2 6 9 .270.271.272.273.274.275.276.277.2 7 8 .2 7 9 [email protected] 8 4 .285.2 8 6 .287.1982.

1982 - algorithm 583 lsqr sparse linear

Documents