MA/CS 375 Fall 2002 1

MA/CS 375

Fall 2002

Lecture 31

MA/CS 375 Fall 2002 2

Root Finding

• Given a real valued function f of one variable (say x), the idea is to find an x such that:

f(x) = 0

MA/CS 375 Fall 2002 3

Newton’s Method cont.

• Repeat the formula to create an algorithm:

• If at each step the linear model is a good approximation to f then xn should get closer to a root of f as n increases.

1 'n

n nn

f xx x

f x

MA/CS 375 Fall 2002 4

Newton in Matlab

MA/CS 375 Fall 2002 5

Newton’s Method Without Knowing the Derivative

• Recall: we can approximate the derivative to a function with:

'

f x f xf x

MA/CS 375 Fall 2002 6Modification

MA/CS 375 Fall 2002 7

Team Exercise

• 10 minutes

• Modify your script to use the approximate derivative (note you will require an extra parameter delta)

• Test it with some function you do not know the derivative of.

MA/CS 375 Fall 2002 8

Convergence Rate For Newton’s Method

• Theorem 8 (van Loan p 285)– Suppose f(x) and f’(x) are defined on an interval

where and positive constants rho and delta exist such that

– If xc is in I, then is in I and

– That is x+ is at least half the distance to x* that xc was.Also, the rate of convergence is quadratic.

2

* * *

1

2 2c cx x x x x x

* *,x x *( ) 0, 0f x

' for all

'( ) '( ) for all ,

f x x

f x f y x y x y

( )

'( )c

cc

f xx x

f x

MA/CS 375 Fall 2002 9

Convergence Rate of Newton’s Method cont

• The proof of this theorem works by using the fundamental theorem of calculus.

• All of the restrictions are important – and can be fairly easily broken by a general function

• The restrictions amount to:

1) f’ does not change sign in a neighbourhood of the root x*

2) f is not too non-linear (Lipschitz condition)

3) the Newton’s iteration starts close enough to the root x* then convergence is guaranteed and the convergence rate is quadratic.

MA/CS 375 Fall 2002 10

Finding A Root Of A Two-dimensional Function of Two Variables

• Suppose:

• Specifically:

2 2: F

,

,

f x y

g x y

F

MA/CS 375 Fall 2002 11

Finding A Root Of A Two-dimensional Function of Two Variables

• Suppose:

• Specifically:

• We can construct a Taylor series:

2 2: F

,

,

f x y

g x y

F

2 2, ,, , , ,

f x y f x yf x y f x y O

x y

MA/CS 375 Fall 2002 12

Constructing A Newton Scheme

2 2

2 2

, ,, , , ,

, ,, , , ,

f x y f x yf x y f x y O

x y

g x y g x yg x y g x y O

x y

1 1 1 1

1 1 1 1

, ,, ,

, ,, ,

n n n n

n n n n n n n n

n n n n

n n n n n n n n

f x y f x yf x y f x y x x y y

x y

g x y g x yg x y g x y x x y y

x y

Create a sequence by using this linear approximation to update from (xn,yn) to (xn+1,yn+1)

MA/CS 375 Fall 2002 13

Constructing A Newton Scheme

1 1

1 1

, ,0 ,

, ,0 ,

n n n n

n n n n n n

n n n n

n n n n n n

f x y f x yf x y x x y y

x y

g x y g x yg x y x x y y

x y

1 1 1 1

1 1 1 1

, ,, ,

, ,, ,

n n n n

n n n n n n n n

n n n n

n n n n n n n n

f x y f x yf x y f x y x x y y

x y

g x y g x yg x y g x y x x y y

x y

We figure that if the linear approximationis good then the f(xn+1,yn+1) and g(xn+1,yn+1)should be small

MA/CS 375 Fall 2002 14

Two-dimensional Newton Method

1 1

1 1

, ,0 ,

, ,0 ,

n n n n

n n n n n n

n n n n

n n n n n n

f x y f x yf x y x x y y

x y

g x y g x yg x y x x y y

x y

1

1

, ,,

, , ,

n n n n

n n n n

n n n n n n n n

f x y f x yx x f x yx y

g x y g x y y y g x y

x y

MA/CS 375 Fall 2002 15

Jacobian Matrix

1

1

, ,

Define: , ,

,Then

,

n n n n

n n n n

n nn n

n n n n

f x y f x y

x y

g x y g x y

x y

f x yx x

y y g x y

J

J

1

1

, ,,

, , ,

n n n n

n n n n

n n n n n n n n

f x y f x yx x f x yx y

g x y g x y y y g x y

x y

MA/CS 375 Fall 2002 16

Jacobian Matrix

11

1

, ,

,, ,

,

,

n n n n

n n

n n n n

n nn n

n n n n

f x y f x y

x yx y

g x y g x y

x y

f x yx x

y y g x y

J

J

MA/CS 375 Fall 2002 17

Team Exercise (Part 1)• Code up the two-dimensional Newton solver. • Use the approximation for small delta:

• Use it to find x,y such that:

, , ,

, , ,

f x y f x y f x y

x

f x y f x y f x y

y

( , ) sin( ) 0

( , ) cos( )sin( ) 0

f x y xy

g x y x y

MA/CS 375 Fall 2002 18

Team Exercise (Part 2)

• Generalize your Newton Solver to solve:

• Construct a 3-vector function and find its roots.

• Plot the position of (xn,yn,zn) using sphere

3 3 where : F x 0 F

MA/CS 375 Fall 2002 19

Problems with Multi-D Newton

• The method relies on inverting the Jacobian matrix.

• Recall that matrix inversion is strongly dependent on the condition number of the matrix.

• Clearly, if the root is near a region where the gradient is small then we will run into slow convergence when the search nears the root.

MA/CS 375 Fall 2002 20

Team Exercise:Dodgy Convergence

• Try to find the (0,0) root of

• Compare the rate of convergence for the (0,0) root of:

• For each iteration plot the condition number of the Jacobian matrix. Try several different starting positions.

3 3

, 0

, 0

f x y x y

g x y x y

2 2

3 3

, 0

, 0

f x y x y x y

g x y x y x y

MA/CS 375 Fall 2002 21

Roots of a Polynomial

• Suppose we wish to find all the roots of a polynomial of order P

• Then there are going to be at most P roots!.

• We can use a variant of Newton’s method.

MA/CS 375 Fall 2002 22

Roots of a Polynomial cont.

• Suppose we have an initial guess for one of the roots of the polynomial function f

• Then we can use Newton’s method, starting at this guess to solve for f(x)=0

• Once we have found the first root x1 we apply polynomial deflation to remove this root and then repeat the process to find the next root.

MA/CS 375 Fall 2002 23

Algorithm1) We are seeking the roots x1,x2,..xP of a

polynomial f

2) We find x1 using Newton’s method.

3) We then use Newton’s method to find the next root of f(x)/(x-x1)

4) Then we find x3 as a root of

f(x)/((x-x1)(x-x2))

5) Repeat until all roots found

MA/CS 375 Fall 2002 24

Details of Newton’s Algorithm

• At the k’th step, we need to find a root of

For Newton’s we need:

1

1

i k

ii

f xg x

x x

g x

dg x

dx

MA/CS 375 Fall 2002 25

1

1

1

1 11

1 1

1 1

i k

ii

i k

i k i ki i

i ii i

dg x f xd

dx dx x x

df x f x

dx x xx x x x

Product rule fordifferentiation

MA/CS 375 Fall 2002 26

1

1 11

1 1

1

1

1 1

1

i k

i k i ki i

i ii i

i k

i i

dg x df x f x

dx dx x xx x x x

df x

dxg x g x

f x x x

1

1

1i k

i i

g x f x

dg x df xf x

dx dx x x

MA/CS 375 Fall 2002 27

Newton Scheme For Multiple Root Finding

1 2 P

1

1

Initiate guesses to the roots ,x ,..x

Loop over k=1:P

Iterate:

1

to find to a given tolerance

End loop

kk k i k

kk

i k i

k

x

f xx x

df xf x

dx x x

x

MA/CS 375 Fall 2002 28

Mul

tiple

Roo

t Fin

der

(app

lied

to fi

nd

ro

ots

of L

ege

ndre

po

lyn

om

ials

)

MA/CS 375 Fall 2002 29

Recall: recurrence relation of Legendre polynomials

MA/CS 375 Fall 2002 30

Recall: Recurrence Relation for Gradient of Legendre Polynomials

MA/CS 375 Fall 2002 31

Roots of the 10th Order Legendre Polynomial

Notice how they cluster at the end points

MA/CS 375 Fall 2002 32

Numerical Quadrature

• A numerical quadrature is a set of two vectors.

• The first vector is a list of x-coordinates for nodes where a function is to be evaluated.

• The second vector is a set of integration weights, used to calculate the integral of a function which is given at the nodes

MA/CS 375 Fall 2002 33

Example of Quadrature

• Say we wish to calculate an approximation to the integral of f over [-1,1] :

• Suppose we know the value of f at a set of N points then we would like to find a set of weights w1,w2,..,wN so that:

1

1

f x dx

1

11

i N

i ii

f x dx w f x

MA/CS 375 Fall 2002 34

Newton-Cotes Formula

• The first approach we are going to use is the well known Newton-Cotes quadrature.

• Suppose we are given a set of points x1,x2,..,xN. Then we require that the constant is exactly integrated:

11 10 0 0 0

1 1 2 2

1 11N N

xw x w x w x x dx

MA/CS 375 Fall 2002 35

11 10 0 0 0

1 1 2 2

1 1

11 21 1 1 1

1 1 2 2

1 1

111 1 1 1

1 1 2 2

1 1

1

2

N N

N N

NN N N N

N N

xw x w x w x x dx

xw x w x w x x dx

xw x w x w x x dx

N

Now we require that 1,x,x2,..,xN-1 are integrated exactly

MA/CS 375 Fall 2002 36

11

0 0 011 2 22

1 1 121 2

1 1 11 2

1 1

1

1 1

2

1 1

N

N

N N NNN NN

wx x x

wx x x

wx x x

N

In Matrix Notation:

Notice anything familiar?

MA/CS 375 Fall 2002 37

11

0 0 011 2 22

1 1 121 2

1 1 11 2

1 1

1

1 1

2

1 1

N

N

N N NNN NN

wx x x

wx x x

wx x x

N

tV w

It’s the transpose of the Vandermonde matrix

MA/CS 375 Fall 2002 38

Integration by Interpolation

• In essence this approach uses the unique (N-1)’th order interpolating polynomial If and integrates the area under the If instead of the area under f

• Clearly, we can estimate the approximation error using the estimates for the error in the interpolation we used before.

MA/CS 375 Fall 2002 39

Newton-Cotes Weights

11

1 22

2

1 1

1

1 1

2

1 1

t

N NN

w

w

w

N

1V

MA/CS 375 Fall 2002 40

Using Newton-Cotes Weights

1

11

i Nt

i ii

f x dx w f x

w f

MA/CS 375 Fall 2002 41

Using Newton-Cotes Weights(Interpretation)

1

11

1 21 21 1 1 1 1 1

1 2

i Nt

i ii

NN

f x dx w f x

N

1

w f

V f

i.e. we calculate the coefficients of the interpolating polynomial expansion using the Vandermonde, then since we know the integral of each term we can sum up the integral of each term to get the total.

MA/CS 375 Fall 2002 42

Matlab Function for Calculating Newton-Cotes Weights

MA/CS 375 Fall 2002 43

Demo: Matlab Function for Calculating Newton-Cotes Weights

1) set N=5 points2) build equispaced nodes3) calculate NC weights

4) evaluate F=X^3 at nodes5) evaluate integral

6) F is anti-symmetric on [-1,1] so its integral is 0

7) Answer correct

MA/CS 375 Fall 2002 44

Team Exercise

• Get the directory Lecture19m from the cd-rom

• make sure your matlab path points to the copy of this directory

• using a script figure out what order polynomial the weights can exactly integrate for a given set of N points (say N=6).

MA/CS 375 Fall 2002 45

Gauss Quadrature

• The construction of the Newton-Cotes weights does not utilize the ability to choose the distribution of nodes for greater accuracy.

• We can in fact choose the set of nodes to increase the order of polynomial that can be integrated exactly using just N points.

MA/CS 375 Fall 2002 46

2 1

1

where:

f 1,1

where 1,1

0 where s 1,1

1,1

p

pi i

pi

p

f x If x r x s x

If x f x If

s x

r

P

P

P

P

Suppose:

MA/CS 375 Fall 2002 47

2 1

1

where:

f 1,1

where 1,1

0 where s 1,1

1,1

p

pi i

pi

p

f x If x r x s x

If x f x If

s x

r

P

P

P

P

Suppose:Remainder term, whichmust have p roots locatedat the interpolating nodes

MA/CS 375 Fall 2002 48

1 1 1

1 1 1

1

1 1

i N

i ii

f x If x r x s x

f x dx If x dx r x s x dx

w f x r x s x dx

At this point we can choose the nodes {xi}. If we choose them so that they are the p+1 roots of the (p+1)’th order Legendre function then s(x) is in fact the N=(p+1)’th order Legendre function itself!.

MA/CS 375 Fall 2002 49

1 1 1

1 1 1

1

1 1

N

i N

i i Ni

f x dx If x dx L x r x dx

w f x L x r x dx

But we also know that if r is a lower order polynomial than (p+1)’th order, it can be expressedas a linear combination of Legendre polynomialsand is in fact orthogonal to Lp+1

MA/CS 375 Fall 2002 50

1

2 1

11

for all i N

Ni i

i

f x dx w f x f

P

i.e. the quadrature is exact for all polynomials of order up to p=(2N-1)

Hence:

MA/CS 375 Fall 2002 51

Summary of Gauss Quadrature

• We can use the multiple root finder to locate the roots of the N’th order Legendre polynomial.

• We can then use the Newton-Cotes formula with the roots of the N’th order Legendre polynomial to calculate a set of N weights.

• We now have a quadrature !!! which will integrate polynomials of order 2N-1 with N points

MA/CS 375 Fall 2002 52

Team Exercise

• Use the root finder and Newton-Cotes routines to build a quadrature for N points (N arbitrary).

• Test it on some functions you know the integral of (sin(x) or tan(x) or exp(x) or …) over the interval [-1,1]

MA/CS 375 Fall 2002 53

Summary

• We have looked at two ways to find the root of a single valued, single parameter function

• We considered a robust, but “slow” bisection method and then a “faster” but less robust Newton’s method which we generalized to vector functions of multiple arguments

• We discussed the theory of convergence for Newton’s method.