® backward error analysis and numerical software sven hammarling nag ltd, oxford sven@nag.co.uk
Post on 26-Dec-2015
222 Views
Preview:
TRANSCRIPT
®
Backward Error Analysis and Numerical Software
Sven Hammarling
NAG Ltd, Oxford
sven@nag.co.uk
Plan of Talk
• Introduction– Stability
– Error analysis
– Condition
• The Basic Linear Algebra Subprograms (BLAS)• LAPACK: state of the art Linear Algebra PACKage• ScaLAPACK: distributed memory version of
LAPACK (Scalable LAPACK)• Other Aspects• Summary
Stability
The stability of a method for solving a problem is concerned with the sensitivity of the method to (rounding) errors in the solution process
A method that guarantees as accurate a solution as the data warrants is said to be stable, otherwise the method is unstable
Example - Sample Variance
22
1
2
2 2
1 1
11
1
1 12
1
For 10000,10001,10002 using 8 figure arithmetic
1 gives: 1.0, 2 gives: 0.0
Are either of these reasonable? (Actually (1) is the
correc
n
n ii
n n
n i ii i
T
s x xn
s x xn n
x
t answer)
Error AnalysisError analysis is concerned with establishing whether or not an algorithm is stable for the problem in hand
A forward error analysis is concerned with how close the computed solution is to the exact solution
A backward error analysis is concerned with how well the computed solution satisfies the problem to be solved
Example
Problem: solve for
Let be the computed solution
Forward error:
Backward error: (residual: )
or: ( )
Ax b x
x
x x
Ax b r b Ax
A E x b
Example (cont’d),
99 98 1 1 , ,
100 99 1 1
2.97Consider
2.99
1.97 0.01 , but
1.99 0.01
1.01Now consider ,
0.99
0.01 , but
0.01
Ax b r b Ax
A b x
x
x x r
x
x x r
1.97
1.97
Componentwise and Normwise
ij
, computed solution
or , ( )
? or e = ? - componentwise
? or E = ? - normwise
j
Ax b x
x x Ax b A E x b
ConditionThe condition of a problem is concerned with the sensitivity of the problem to perturbations in the data
A problem is ill-conditioned if small changes in the data cause relatively large changes in the solution. Otherwise a problem is well-conditioned
®
Example
1 2
1 2
1 2
1 2
1 2
1 2
99 98 197
100 99 199
1
98.99 98 197
100 99 199
100, 99
x x
x x
x x
x x
x x
x x
Condition Number for Linear Equations
1
1
1
, ( )
( ) , so that
giving
( ) ,
where ( ) is the condition number of
with respect to matrix inversion
Ax b A E x b
A x x Ex x x A Ex
x x EA E A
x A
A A A
A
Condition and Error Analysis
forward error
condition number backward error
Example
1
1 1
1
2 41
99 98, 199
100 99
99 98, 199
100 99
199 4 10
A A
A A
A
Backward Error as Perturbations in the Data
Let be a matrix such that
(1)
Then
r Ax b
E
Ex r
A E x b
Backward Error as … (cont’d)
2
A particular that satisfies (1) is given by
and this can be shown to be the that minimizes
(and ). If
then
, ( )
This is an
T T
F
F F F F F
E
E rx x x
E
E E
r A x E A
r Ax b A E x b
a posteriori
bound
Example
99 98 1 2.97 , ,
100 99 1 2.99
0.01 .
0.01
Then
0.0297 0.0299 , 17.761
0.0297 0.0299
and
0.00167 0.00168
0.00167 0.00168
T T
T T
A b x
r E rx x x
rx x x
E
Backward Error Analysis
Let the computed solution satisfy the equation
.
Backward error analysis:
? (componentwise)
or
? (normwise)
ij
Ax b
x
A E x b
e
E
Backward Error and Perturbation Analysis
,
If we know how perturbations in affect the
solution , then we can estimate the accuracy of the
computed solution . That is, an estimate of the
backward error allows us to estim
Ax b A E x b
A
x
x
ate the forward error.
x x E
Ax A
The Purpose of Error Analysis“The clear identification of the factors determining the stability of an algorithm soon led to the development of better algorithms. The proper understanding of inverse iteration for eigenvectors and the development of the QR algorithm by Francis are the crowning achievements of this line of research.
“For me, then, the primary purpose of the rounding error analysis was insight.”
Wilkinson, 1986Bulletin of the IMA, Vol 22, p197
LAPACK and Error Bounds
“In Addition to providing faster routines than previously available, LAPACK provides more comprehensive and better error bounds. Our goal is to provide error bounds for most quantities computed by LAPACK.”
LAPACK Users’ Guide, Chapter 4 – Accuracy and Stability
The BLAS, LAPACK and ScaLAPACK
The Three Levels of BLAS The Level 1 BLAS are concerned with scalar and
vector operations, such as
the Level 2 BLAS with matrix-vector operations such as
and the Level 3 BLAS with matrix-matrix operations such as
2, ,Ty x y x y x
1, yAxyxTx
1,C AB C X T X
Why Higher Level BLAS?
Registers
L1 Cache
L2 Cache
Local Memory
Remote Memory
Secondary Memory
BLAS Mem Refs Ops Ratio
Level 1
Level 2
Level 3
3 : 2
1 : 2
2 : n
2n
2n2
2n3
3n
n2
4n2
y x y
y Ax y
C AB C
The so called ‘surface to volume’ effect
BLAS for Performance
IBM RS/6000-590 (66 MHz, 264 Mflop/s Peak)
0
50
100
150
200
250
10 100 200 300 400 500Order of vector/Matrices
Mfl
op
/s
Level 3 BLAS
Level 2 BLAS
Level 1 BLAS
Publication of the Specifications of the BLAS
[1] C L. Lawson, R. J. Hanson, D. Kincaid, and F. T. Krogh. Basic Linear Algebra Subprograms for FORTRAN usage. ACM Trans. Math. Software, 5:308-323, 1979.
[2] J. J. Dongarra, J. Du Croz, S. Hammarling, and R. J. Hanson. An extended set of FORTRAN Basic Linear Algebra Subprograms. ACM Trans. Math. Software, 14:1-32, 399, 1988. (Algorithm 656).
[3] J. J. Dongarra, J. Du Croz, I. S. Duff, and S. Hammarling. A set of Level 3 Basic Linear Algebra Subprograms. ACM Trans. Math. Software, 16:1-28, 1990. (Algorithm 679).
LAPACK
• Systems of linear equations
• Linear least squares problems
• Eigenvalue and singular value problems, including generalized problems
• Matrix factorizations
• Condition and error estimates
• The BLAS as a portability layer
Linear Algebra PACKage for high-performance computers
Dense and banded linear algebra for Shared Memory
Benefits of LAPACK
• Comprehensive and consistent set of routines for dense and banded linear algebra
• Efficient on a wide range of modern computers• Additional functionality compared to previous
packages• Improved numerical algorithms• Error and sensitivity estimates• Portability layer between users applications and
the machines
LAPACK
Users’
Guide
E. Anderson, Z. Bai, C. BischofS. Blackford, J. Demmel, J. Dongarra
J. Du Croz, A. Greenbaum, S. HammarlingA. McKenney, D. SorensenSIAM, 1999 (3rd Edition)
Proceeds to SIAM student travel fundhttp://www.netlib.org/lapack/lug/lapack_lug.html
L A P A C KL -A P -A C -KL A P A -C -KL -A P -A -C KL A -P -A C KL -A -P A C -K
1/4
1/L 1/L 1/L 1/L 1/A -1/A 1/A -1/A1/P 1/P -1/P -1/P1/A -1/A -1/A 1/A 1/C 1/C -1/C -1/C1/K -1/K -1/K 1/K
Performance Improvements
Efficiency in LAPACK has been principally achieved by restructuring algorithms so that as much computation as possible is performed by calls to:
Level 3 and Level 2 BLAS
In order to exploit the Level 3 BLAS we needed:
BLOCK-PARTITIONED ALGORITHMS
a11 a12 a13
a21 a22 a23
a31 a32 a33
A11 A12 A13
A21 A22 A23
A31 A32 A33
Error Bounds in LAPACK
The Users’ Guide gives details of error bounds and code fragments to compute those bounds. In many cases the routines return the bounds directly
Error Bounds in LAPACK: Example 1
DGESVX is an 'expert' driver for solving
Estimate of 1/
Estimated forward error for
Componentwise relat
j
AX B
A
X
DGESVX( ..., RCOND, FERR, BERR, ...)
RCOND -
FERR(j) -
BERR(j) - ive backward error for
(smallest relative change in any element of and
that makes an exact solution)
j
j
X
A B
X
Error Bounds in LAPACK: Example 2
DGEEVX is an 'expert' driver for computing
eigenvalues and eigenvectors of a matrix
Norm of balanced matrix
Reciprocal condition number for
A
DGEEVX( ..., ABNRM, RCONDE, RCONDV,...)
ABNRM -
RCONDE(i) - the
eigenvalue ( )
Reciprocal condition number for the
eigenvector ( )
thi
thi
i s
i sep
RCONDV(i) -
Error Bounds in LAPACK: Example 2 (cont’d)
2
2
Simple eigenvalue /
Eigenvector ( , ) / sep
Eigenvalue cluster /
Invariant subspace ( , ) / sep
i i i
i i iF
I I I
I I IF
E s
v v E
E s
S S E
Asymptotic error bounds for nonsymmetric eigenproblem
See Golub and Van Loan – Matrix Computations
Error Bounds in LAPACK: Example 2 (cont’d)
EPSMCH = DLAMCH(‘E’)
DO 10 I = 1, N
EERRBD(I) = EPSMCH*ABNRM/RCONDE(I)
VERRBD(I) = EPSMCH*ABNRM/RCONDV(I)
10 CONTINUE
EPSMCH is the machine precision
EERRBD( ), ( , ) VERRBD( )i i i ii v v i
Availability of LAPACK
LAPACK is freely available via netlib:
http://www.netlib.org/lapack/index.html
Much of LAPACK is included in the NAGFortran 77 Library and is the basis of the denselinear algebra in the NAG Fortran 90 Library
Tuned versions of some LAPACK routines are inthe NAG Fortran SMP Library
LAPACK - Summary
• LAPACK success from HP-48G to Cray T-90
• Portable Fortran 77 (+BLAS), 766K lines (354K library, 412K testing and timing)
• PCs, workstations, vector machines, and shared memory computers
• Initial release February 29th, 1992
• Now on the 3rd release
• Linear systems, least squares, eigenproblems
• C, Java and Fortran 95 interfaces
ScaLAPACK
• Scalability, performance and portability• Interfaces to stay as close to LAPACK as possible• Use LAPACK algorithms wherever possible• Principal additional consideration is the data distribution• The PBLAS (BLAS + BLACS) as a portabilility layer
Scalable Linear Algebra PACKage
Aim: To port LAPACK to distributed memory environments
As with LAPACK, ScaLAPACK is freely available:http://www.netlib.org/scalapack/index.html
Most of ScaLAPACK included in NAG Parallel Library
ScaLAPACK Structure
ScaLAPACK
BLAS
LAPACK BLACS
PVM/MPI/...
PBLASGlobal
Local
ScaLAPACK Summary
• Follow on to LAPACK - Designed for MPP’s • Object based design• Portability across wide range of architectures
– will work on a heterogeneous platform
• Software, examples, prebuilt libraries available from netlib
• Users’ Guide, including CD available from SIAM• Continuing development• HPF interface to some driver routines available
Hierarchy of Building Blocks
Users
Applications Packages
(Sca)LAPACK
(P)BLAS
Machines
Developments Since Wilkinson
• More problems solved, tighter bounds
• More emphasis on componentwise results – particularly for sparse and structured problems
• Use of standard precision iterative refinement
• Importance of scaling
• Iterative methods
• Importance of non-normality and pseudospectra
Diagonal Matrix Example
612
6
6
6
6
6 6
10 0, 10
0 10
1 010 0, 1
0 10 10
Beware turning a badly scaled problem into an
ill-conditioned problem
1 0 10 0A=
1 1 10 10
D D
J D J
D
Backward Errors in Other Areas
• For references to other areas see: Higham, Accuracy and Stability of Numerical Algorithms, Sections 1.20 & 1.21.
• Examples:– ODEs– Chaotic behaviour of iterations
What to do if Software does not Provide Estimates?
• Run the problem with perturbed data
• Better still, use a software tool such as PRECISE which allows you to perform a stochastic analysis
• Put pressure on developers to provide estimates
PRECISE
• Provides a module for statistical backward error analysis
• Provides a module for sensitivity analysis
• http://www.cerfacs.fr/algor/Softs/PRECISE/index.html
Quality of Reliable Software
Chaitin-Chatelin & Frayssé define the quality index
of a reliable algorithm at the computed solution as
( ) ( ) ,
where ( ) is the backward error and is the machine
precision.
(Lecture
x
B xJ x
B x
s on Finite Precision Computations)
References
[1] E. Anderson, Z. Bai, C. H. Bischof, J. Demmel, J. J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Blackford & D. C. Sorensen. LAPACK Users’ Guide, 3rd Edition, SIAM, 1999.
[2] F. Chaitin-Chatelin & V. Frayssé. Lectures on Finite Precision Computations, SIAM, 1996.
[3] G. H. Golub & C. F. Van Loan. Matrix Computations, 3rd Edition, The Johns Hopkins University Press, 1996.
[4] N. J. Higham. Accuracy and Stability of Numerical Algorithms, SIAM, 1996.
®
Closing Remarks• LAPACK and ScaLAPACK provide
efficient and reliable portable software for numerical linear algebra
• Emphasis has been placed on error and condition (sensitivity) estimates
• Research and development is ongoing• Available via the NAG Libraries• Expect error estimates from numerical
software• Use software tools
top related