numerics and scienti c computing -...

Numerics and Scientific Computing

Numerics and Scientific ComputingLecture 1, 26.02.2016

http://www.math.uzh.ch/binf2180

Tulin KamanInstitute of Mathematics, University of Zurich

E-mail: [email protected]

Lecture 1, 26.02.2016T. Kaman Numerics and Scientific Computing Lecture 1, 26.02.2016 1 / 30

http://www.math.uzh.ch/binf2180


Introduction

Introduction: Survey on lecture

1 Numerical algorithms (AG 1)

2 Roundoff errors (AG 2)

3 Nonlinear equations in one variable (AG 3)

4 Direct methods for linear system (AG 4.1-4.3/5)

5 Linear least squares problems (AG 6)

6 Iterative methods for linear system (AG 7)

7 Eigenvalues and singular values (AG 8)

8 (Piecewise) polynomial interpolation (AG 10.1-5/11.1-4)

9 Best approximation (AG 12)

10 Numerical integration (AG 15.1-5)

11 Ordinary differential equations (AG 16.1-6)

T. Kaman Numerics and Scientific Computing Lecture 1, 26.02.2016 2 / 30


Introduction

Literature

Uri Ascher, Chen Greif: A First Course in Numerical Methods. SIAM,2011. www.siam.org/books/cs07

Alfio Quarteroni, Fausto Saleri, Paola Gervasio: Scientific Computingwith MATLAB and Octave

Lloyd N. Trefethen, David Bau: Numerical Linear Algebra. SIAM,1997.


www.siam.org/books/cs07


Introduction

Organization of this course

13 lectures

Exercises

50% of the exercise must be solved correctlyto get hands-on experiencebased on Matlabwill be posted on the course website http://www.math.uzh.zh/binf2180

Examination

closed book on paperon 17.06.2016 from 8:00-10:002hour written examination


h


Introduction

Goals

Knowledge of the fundamental algorithms in numerical mathematics

Knowledge of the essential terms in numerical mathematics and thetechniques used for the analysis of numerical algorithms

Ability to choose the appropriate numerical method for concreteproblems

Ability to interpret numerical results

Ability to implement numerical algorithms efficiently in Matlab. Thefirst two weeks of the exercise lab will include a crash course inMatlab.

Learning by doing (exercises)



1. Numerical Algorithms

Scientific computing

Scientific computing is a discipline

concerned with the development and study

of numerical algorithms for solving

mathematical problems that arise in various

disciplines in science and engineering.





Applications





Problem solving environment

MATLAB: a high-level language and interactive environment fornumerical computation, visualization, and programming.

Many of the algorithms that we will learn are already implemented inMATLAB. So why learn them at all?

MATLAB help:

commands: help/doconline documentation: http://www.mathworks.com/help/matlab/

Other options: C++, Java, Python, or Fortran 90


h



Numerical algorithms and errors

The most fundamental feature of numerical computing is theinevitable presence of error.

The result of any interesting computation is typically onlyapproximate.

Our goal is to ensure that the resulting error is tolerably small.

Relative and absolute errors

Given a scalar quantity u and its approximation v :

The absolute error in v : |u − v |.The relative error (assuming u 6= 0) : |u−v ||u| .





Relative and absolute errors

for various calculations

u v Absolute Relativeerror error

1 0.99 0.01 0.011 1.01 0.01 0.01

-1.5 -1.2 0.3 0.2100 99.99 0.01 0.0001100 99 1 0.01

when |u| ≈ 1→ not much difference between absolute and relativeerror measureswhen |u| � 1→ the relative error is more meaningful

we expect the approximation in the last row to be similar in quality tothe one in the first row





Example

Stirling approximation is used to approximate u = n! = 1.2. · · · n for largen.

v = Sn =√

2πn(n

e

)nThe formula involves the constant e = exp(1) = 2.7182818...

The MATLAB script computes the values of u and v for n = 1, · · · , 10e=exp(1);

n=1:10; % array

Sn=sqrt(2*pi*n).*((n/e).^n); % the Stirling

fact_n=factorial(n);

abs_err=abs(Sn-fact_n); % absolute error

rel_err=abs_err./fact_n; % relative error

format short g

[n; fact_n; Sn; abs_err; rel_err]’ % print out values

1 1 0.92214 0.077863 0.077863

2 2 1.919 0.080996 0.040498

3 6 5.8362 0.16379 0.027298

4 24 23.506 0.49382 0.020576

5 120 118.02 1.9808 0.016507

6 720 710.08 9.9218 0.01378

7 5040 4980.4 59.604 0.011826

8 40320 39902 417.6 0.010357

9 3.6288e+05 3.5954e+05 3343.1 0.0092128

10 3.6288e+06 3.5987e+06 30104 0.008296

The absolute errors grow as n grows, but the relative errors stay wellbehaved and indicate that in fact the larger n is, the better the quality ofthe approximation is.





Error types

The types of error that may limit the accuracy of a numerical calculation:1 Errors in the problem to be solved.

Errors in the mathematical modelErrors in the input data

2 Approximation errors

Discretization errorsConvergence errorsDiscretization and convergence errors may be assessed by an analysis ofthe method used

3 Roundoff errors

Roundoff errors arise in any computation with real numbers because ofthe finite precision representation of real numbers on any computer





Discretization errors in action I

Problem

Approximating the derivative f ′(x0) of a given smooth function f (x) at thepoint x = x0.

Example

Let f (x) = sin(x) be defined on the real line −∞ < x <∞ and setx0 = 1.2 Thus, f (x0) = sin(1.2) ≈ 0.932....

Consider a situation where f (x) may be evaluated at any point x nearx0, but

f ′(x0) may not be directly availableor is computationally expensive to evaluate





Discretization errors in action II

⇒ approximate f ′(x0) by evaluating f at x near x0.

Taylor’s series gives us an algorithm to approximate f ′(x0): For somesmall, positive value h

f (x0 + h) = f (x0) + hf ′(x0) +h2

2!f ′′(x0) +

h3

3!f ′′′(x0) +

h4

4!f ′′′′(x0) + · · ·

f ′(x0) =f (x0 + h)− f (x0)

h−(

h

2f ′′(x0) +

h2

6f ′′′(x0) +

h3

24f ′′′′(x0) + ...

)The algorithm for approximating f ′(x0) is to calculate

f (x0 + h)− f (x0)

h





Discretization errors in action III

The approximation has the discretization error∣∣∣∣f ′(x0)− f (x0 + h)− f (x0)

h

∣∣∣∣ =

∣∣∣∣h2 f ′′(x0) +h2

6f ′′′(x0) +

h3

24f ′′′′(x0) + ...

∣∣∣∣Geometrically, the tangent f ′(x0) isapproximated by the chord (f (x0 +h)− f (x0))/h.

∣∣∣∣f ′(x0)− f (x0 + h)− f (x0)

h

∣∣∣∣ =h

2|f ′′(ξ)| , where x0 < ξ < x0 + h





Discretization errors in action IV

We expect the discretization error to decrease when h decrease:∣∣∣∣f ′(x0)− f (x0 + h)− f (x0)

h

∣∣∣∣ ≈ h

2|f ′′(x0)|

Big-O notation: We consider various computational errors depending ondiscretization step size h > 0 and ask how they decrease as h decreases.

Definition

For an error e depending on h we denote e = O(hq) if there are two positiveconstants q and C such that |e| ≤ Chq for all h > 0 small enough.

Using the Big-O notation:∣∣∣f ′(x0)− f (x0+h)−f (x0)

h

∣∣∣ = O(h)





For instance: f (x) = sin(x),the exact value f ′(x0) = cos(1.2) = 0.362357754476674...For h = 0.1,the approximation f ′(x0) ≈ (sin(1.3)− sin(1.2)) /0.1 = 0.315...,which is not very accurate.

h Absolute error

0.1 4.716676e-20.01 4.666196e-3

0.001 4.660799e-41.e-4 4.660256e-51.e-7 4.619326e-8

The error appears to decrease like h. Using the knowledge off ′′(x) = − sin(x)⇒ f ′′(x)/2 ≈ −0.466The quantity 0.466h provides an accurate estimate for the absolute errorvalues.





For very small h

Suppose we want

|cos(1.2)− sin(1.2 + h)− sin(1.20)

h| < 10−10

Can’t we just set h ≤ 10−10/0.466 ?

h Absolute error

1.e-8 4.361050e-101.e-9 5.594726e-8

1.e-10 1.669696e-71.e-11 7.938531e-61.e-13 4.250484e-41.e-15 8.173146e-21.e-16 3.623578e-1





The combined effect of discretization and roundoff errors.

The discretization error decreases as h decreases, and it dominates theroundoff error when h is relatively large.

When h gets below approximately 10−8 the discretization error becomes verysmall and roundoff error starts to dominate.

The roundoff error increases as h decreases.

This is one reason why we want it always dominated by the discretizationerror when solving problems involving numerical differentiation.




Algorithm properties

Performance features that may be expected from a good numericalalgorithm:

Accuracy refers to the closeness of a measured value to a standard orknown value. Precision refers to the closeness of two or moremeasurements to each other.

Efficiency depends on both CPU time and storage space requirement

How fast can we solve a certain problem?How much memory space do we need?

Rate of convergence. Floating point operations (flops)

Robustness Numerical software should run under all circumstances.Should yield correct results to within an acceptable error or should failgracefully if not successful.





Efficiency

Machine-independent estimate of the number of elementaryoperations required, (+ - * /), gives an idea of the algorithmsefficiency.

Operation counts do not give the complete picture regardingexecution speed, since they do not take into account the price (speed)of memory access which may vary considerably.

In modern computers, cache access, blocking and vectorizationfeatures, and other parameters are crucial in the determination ofexecution time.





Efficiency

Big-O notation: When estimating the efficiency of a particular algorithm,we are interested in a bound on the work estimate as a parameter nincreases unboundedly (e.g., n = 1/h).

Definition

For w = w(n) the expression w = O(nlogn) means that there is aconstant C > 0 such that w ≤ Cn log n as n→∞.

operation description #mul #add#div #sub

inner product (x ∈ Rn, x ∈ Rn) 7→ xTy n n-1 O(n)outer product (x ∈ Rm, x ∈ Rn) 7→ xyT mn 0 O(mn)

matrix product (A ∈ Rmxn,B ∈ Rnxk) 7→ AB mnk mk(n-1) O(mnk)





Problem conditioning and algorithm stability

The problem and the numerical algorithm both yield errors, question arises

regarding the appraisal of a given computed solution.

⇓

Problem sensitivity and algorithm stability play an important role.

The problem is ill-conditioned if a small perturbation in the data mayproduce a large difference in the result, then no algorithm may befound for that problem which would meet our requirement of solutionrobustness.

Otherwise the problem is well-conditioned.

The algorithm is stable if its output is the exact result of a slightlyperturbed input.





An unstable algorithm

An ill-conditioned problem of computing output values y given in terms ofinput values x by y = g(x): when the input x is slightly perturbed to x ,the result y = g(x) is far from y .For instance, the problem of numerical differentiation turns out to be ill-conditioned when extreme accuracy (translating to very

small values of h) is required.





A stable algorithm

An instance of a stable algorithm for computing y = g(x): the output y isthe exact result, y = g(x), for a slightly perturbed input, i.e. x which isclose to the input x.Thus, if the algorithm is stable and the problem is well-conditioned, thenthe computed result y is close to the exact y .





Example

The problem of evaluating the square root function g(x) =√

1 + x for anargument near the value 1 is well-conditioned. Show?

Suppose we fix x so that |x | � 1, and consider x = 0 as a smallperturbation of x . Then

y = g(x) = 1⇒ y − y =√

1 + x − 1

If we approximate√

x + 1 by the first 2-terms of its Taylor seriesexpansion: g(x) ≈ 1 + x

2

⇒ y − y ≈(

1 +x

2

)− 1 =

x

2=

1

2(x − x)

The conditioning of this problem is determined by g ′(0) = 1/2, because

g ′ ≈ g(x)−g(0)x−0 .

The problem is well-conditioned because this number is not large.T. Kaman Numerics and Scientific Computing Lecture 1, 26.02.2016 26 / 30




Example

The problem of evaluating the function g(x) = tan(x)

1 for x near zero is well-conditioned.

2 for x near π2 is ill-conditioned. Show?

2 Show that a small perturbation in the input data produces a largedifference in the result. Set x = π

2 − 0.001 and x = π2 − 0.002

⇒ |x − x | = 0.001 but | tan(x)− tan(x)| ≈ 500

The derivative g ′(x) = 1cos2(x)

for x near π2 explains why.





Error accumulation

Explicitly, if En measures the relative error at the nth operation of analgorithm, then

En ≈ c0nE0 for some constant c0 represents linear growth

En ≈ cn1 E0 for some constant c1 > 1 represents exponential growth.

An algorithm exhibiting relative exponential error growth is unstable. Suchalgorithms must be avoided!





Unstable algorithm

Consider the integrals yn =∫ 1

0xn

x+10 dx n = 1, 2, · · · 30 Observe thatanalytically, for n > 0,

yn + 10yn−1 =

∫ 1

0

xn + 10xn−1

x + 10dx =

∫ 1

0xn−1dx =

1

n

Also

y0 =

∫ 1

0

1

x + 10dx = ln(11)− ln(10)

An algorithm is

1 Evaluate y0 = ln(11)− ln(10)

2 For n = 1, · · · , 30 evaluate yn = 1n − 10yn−1

Applying the above recursion formula would give exact values if roundoff errors were not present.

However, this algorithm is unstable, as the magnitude of roundoff errors gets multiplied by 10 each time the recursion is applied.



Roundoff Errors

Next week:

We have much to say about the floating point representation of realnumbers and the accumulation of roundoff errors during a calculation.


numerics and scienti c computing -...

Documents