Lecture 3Introduction to Numerical Methods for Differential
and Differential Algebraic Equations
Dr. Abebe Geletu
Ilmenau University of TechnologyDepartment of Simulation and Optimal Processes (SOP)
Winter Semester 2011/12
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
TopicsNewton’s Law:mx = −F
l xmy = mgFl yConservation of mechanicalenergy: x2 + y2 = l2
(DAE) x1 = x3
x2 = x4
x3 = − F
m lx1
x4 = gF
lx2
0 = x2 + y2 − l2.1
2 Numerical Methods of Ordinary Differential Equations1 Initial Value Problems (IVPs)
Single Step MethodsMulti-step Methods
2 Boundary Value ProblemsFinite Difference MethodsSingle- and Multiple-Shooting Methods
3 Numerical Methods of Differential Algebraic Equations (DAEs)DAE models in engineering applicationsPeculiarities of DAEsIndex notions for DAEsBackward Difference Formula (BDF)Implicit Rung Kutta (IRK) MethodCollocation on finite elements
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Initial Value Problems
General form of an initial value problem (IVP) is
x(t) = f(t, x(t)), t0 ≤ t ≤ tf , (1)
x(t0) = x0 (given), (2)
where x and f can be vector functions:
x(t) =
x1(t)x2(t)
...xn(t)
, f(t, x(t)) =
f1(t, x(t))f2(t, x(t))
...fn(t, x(t))
.
If f is (explicitly) time independent; i.e. f(t, x(t)) = f(x(t)), thenthe system (1) - (2) is said to be autonomous.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
x(t0) = x0 is the initial condition (start time of the motion)
The expression x(t) represents the first order derivative of x(t)with respect to time; i.e. x(t) = d
dtx(t). Thus, the IVP (1) - (2)represents a system of first order differential equations.
If f(t, x(t)) = A(t)x(t), then we havex(t) = A(t)x(t), x(t0) = x0 is a first order linear IVP. In theautonomous case we have A(t) = A.
Example: Equation of motion of a simple pendulum:
θ = ω
ω = − gLsin(θ)
⇒(θω
)=
(ω
− gLsin(θ)
),θ(0) = π
4ω(0) = 0
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Commonly, numerical methods are developed for systems of firstorder differential equations.⇒ any higher order differential equation should be written as asystem of first order differential equations.
NB: The Matlab ODE Toolbox works only with systems of first orderdifferential equations.Strategy: Convert any n-th order initial value problem into a system
of first-order IVP by introduction new variables.
dnx(t)
dtn= f
(t,dx
dt,d2x
dt2, . . . ,
dn−1x
dtn−1
)x(t0) = a1, x(t0) = a2,
dx(t)
dt
∣∣∣∣t=t0
= a3, . . . ,dn−1x(t)
dtn−1
∣∣∣∣t=t0
= an.
Introduce new variables:x1 = x, x2 = x, x3 = d2x(t)
dt2, . . . , xn = dn−1x(t)
dtn−1 .
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Then the n-th order IVP reduces to
x1 = x2, x1(t0) = a1
x2 = x3, x2(t0) = a2...
xn = f(t, x1, x2, . . . , xn), xn(t0) = an.
Example: The equation of motion of a simple pendulum (neglectingair resistance) with mass m is a 2nd-order equation
mLθ +mg sin(θ(t)) = 0, (3)
θ(t0) = θ0, θ = ω0. (4)
Introduce ω = θ. Then
θ = ω, θ(t0) = θ0
ω = − gL
sin(θ), ω(t0) = ω0.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Objective:To determine a continuous function x∗(t) that satisfies the dynamicequation (1); i.e. x∗(t) = f(t, x∗(t)) and the initial condition (2); i.e.x∗(t0) = x0.
!!! In general, the solution function x∗(t) of the IVP is not easy to beanalytically determined.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Basic idea of numerical solution of differential equations:
Select a set of equally spaced time points t1, t2, . . . , tN from theinterval [t0, tf ] (tf freely chosen for IVPs). That is, t1 = t0 + h ,t2 = t1 +h, and so on; so that h = tk+1− tk, k = 0, 1, . . . , N − 1.
Approximate the continuous solution x∗(t) of (1) & (2) by valuesat the discrete time points t1, t2, . . . , tN ; i.e.x(t1) = x1, x(t2) = x2, . . . , x(tN ) = xN .
determine the values x1, x2, . . . , xN through the IVP (1) - (2).
Note:more discretization points from [t0, tf ] implies
better accuracy of the approximation; but
demands more computation effort,
depending on the algorithm used, it may be necessary to solve asystem of linear or nonlinear equations.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Euler’s MethodMotivation: for a single IVP x(t) = f(t, x(t)), x(t0) = x0; i.e.x(t) ∈ R, 0 ≤ t ≤ tf .
Given the point (t0, x0) the value x(t0) = dx(t)dt
∣∣∣t=t0
is the slope of
the tangent line to the graph of x(t) at the point (t0, x0).
Figure: Euler’s Method
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Since the function x(t) is not yet known, the derivative (slope) can beapproximated by
x(t0) ≈x(t0 + h)− x(t0)
hThus, using x(t0) = f(t0, x0), we have
x(t0 + h)− x(t0)
h= f(t0, x(t0))
This impliesx(t0 + h) = x(t0) + hf(t0, x(t0))
⇒x1 = x(t1) = x(t0 + h) = x(t0) + hf(t0, x(t0))
From this we obtain the value of x1 = x(t1). By the same procedurex2 can be also determined.In general, once xk is known, xk+1 can be found from
xk+1 = xk + hf(tk, xk), k = 0, 1, . . . ,m− 1.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Euler’s AlgorithmSelect N .Set h =
(tf−t0)N ; x0 = x(t0)
FOR k = 0 : (n− 1)xk+1 = xk + hf(tk, xk);tk+1 = tk + h;ENDFOR.
Matlab:function [t,x]=myEuler(@f,x0,t0,tf,N)
h=(tf-tn)/N;
t=[t0:h:tf]; %discrete time points between, and including, t0 and tf
x(1)=x0;
for k=2:N
x(k)=x(k-1) + h*feval(f,t(k-1),x(k-1));
end;
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Simple Pendulum with Euler’s Method
function time=MyPenEuler(N,tf)
t=cputime;
m=1;
L=9.81;
g=9.81;
beta=0;
t0=0;
theta0=(1/10)*pi;
omega0=0;
h=(tf-t0)/N;
tt=[t0:h:tf];
ttheta=zeros(size(tt));
oomega=zeros(size(tt));
ttheta(1)=theta0;
oomega(1)=omega0;Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Simple Pendulum with Euler’s Method...
for k=1:N
oomega(k+1)=oomega(k)+h*...
(-beta/m/L*oomega(k)-g/L*sin(ttheta(k)));
ttheta(k+1)=ttheta(k)+h*oomega(k);
end
time=cputime-t;
plot(tt,oomega,tt,ttheta)
title(’Simple pendulum using Euler’’s method’)
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Advantages/Disadvantages of Euler’s Method
Advantages:
Euler’s Method is simple and direct
Can be used for nonlinear IVPs
Disadvantages:
it is less accurate and numerically unstable.
Approximation error is proportional to the step size h. Hence,good approximation is obtained with a very small h. Thisrequires a large number of time discretization leading to a largercomputation time.
Usually applicable to explicit differential equations.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Improvements on Euler’s Method(A) The Mid-Point MethodLet k1 = hf(ti, xi).Note that: k1 = hf(ti, xi) = xi+1 − xi. But to compute k1, it is notnecessary to know the value of xi+1.
Figure: Use also the mid point
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Improvements on Euler’s Method...
• The point (ti + 12 ti, xi + 1
2k1) is the mid-point of the segment thatconnects the points (ti, xi) and (ti+1, xi+1).
• Now, instead of the single slope at (ti, xi), we use the slopes at(ti, xi) and at (ti + 1
2 ti, xi + 12k1) to approximate xi+1.
⇒ This leads to the Mid-Point Runge-Kutta Algorithm:
The mid-point Runge-Kutta Method
k1 = hf(ti, xi)
k2 = hf(ti +1
2h, xi +
1
2k1)
xi+1 = xi + k2.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Improvements on Euler’s Method...(B) Methods that use Intermediate PointsQuestion: Can we improve the mid-point RK method by freelychoosing any point from the segment connecting (ti, xi) and(ti+1, xi+1)?
Figure: Choose freely any intermediate point
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
• A weighted sum of f(ti, xi) and f(ti + ch, xi +mk1) providedsmore slope information than f(ti, xi) alone.Scheme:
(RK2) x(t+ h) = x(t) + hf(t, x)
where f(t, x) = [w1f(t, x) + w2f(t+ ch, x+mk1)] .How to determine w1, w2 , c and h?We use:(i) Taylor approximation of f(t+ ch, x+mk1):
f(t+ ch, x+mk1) = f(t, x) + chft(t, x) +mk1fx(t, x)x′ (∗)
Put this into (*) to obtain
x(t+h) = x(t)+h [w1f(t, x) + w2{f(t, x) + chft(t, x) +mk1fx(t, x)})] (∗)
⇒ (with k1 = hf(t, x))
x(t+h) = x(t)+h (w1 + w2) f(t, x)+w2ch2ft(t, x)+mw2h
2fx(t, x)f(t, x)
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Improvements on Euler’s Method...(ii) 2nd- order Taylor approximation of x(t+ h):
x(t+ h) = x(t) + hx′(t) +h2
2x′′(t) (∗∗)
x(t+ h) = x(t) + hx′(t) +h2
2x′′(t) (∗∗)
Recall x′(t) = x(t) = f(t, x)⇒ x′′(t) = ft(t, x) + fx(t, x)f(t, x).Putting these into (**) we obtain
x(t+ h) = x(t) + hf(x, t) +h2
2ft(t, x) +
h2
2fx(t, x)f(t, x)) (∗∗)
Comparing (*) and (**) we find w1 + w2 = 1, cw2 = 12 ,mw2 = 1
2 .
A system of three equations with four unknowns ⇒ This system hasinfinitely many solutions.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Improvements on Euler’s Method...• By fixing one of the unknowns w1, w2, c and m, we can solve thesystem of equations for the other three.⇒ Hence, we can construct several RK2 methods.
Example:• If we fix w1 = 0, then w2 = 1, c = 1
2 and m = 12 . Hence, (RK2)
reduces to the mid-point RK2 method.
w1 w2 c m Method
0 1 12
12 Modified Euler Method
12
12 1 1 Heun’s Method
13
23
34
34 Ralston’s Method
Heun’s Method:k1 = f(ti, xi)xi+1 = xi + h
[12f(ti, xi) + 1
2f(ti + h, xi + k1)]
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Runge-Kutta MethodsThe above schemes are known as second-order Runge-Kuttamethods, since x(t+ h) has been approximated by a second degreeTaylor polynomial.I For an accurate approximation use more intermediate pointsbetween (ti, xi) and (ti + h, xi + k1) and include more slopeinformation.I Requires higher degree Taylor approximations of x(t+ h).
Figure: Use several intermediate pointsLecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Runge-Kutta Methods...
General Scheme:
[RK(m)] xi+1 = xi + hf(ti, xi);
wheref(ti, xi) = w1f(ti, xi) + w2f(ti + α1h, xi + β1k1) + . . .+ wmf(ti + αmh, xi + βmkm).
• The values k1, k2, . . . , km are found at the intermediate points.• w1 + w2 + . . .+ wm = 1.• The relation between ω1, . . . , ωm, α1, . . . , αm, β1, . . . , βm isdetermined by comparing the m-th order Talyor approximation ofx(t+ h) and the scheme RK(m)
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
I Hence, the resulting method is called an m-th order Runge-Kuttamethod.I There are third-order Runge-Kutta (RK3), fourth-orderRunge-Kutta methods (RK4), etc.The classical 4-th order Runge-Kutta Method has the form:
(RK4s) k1 = f(ti, xi)
k2 = hf(ti +1
2h+ xi +
1
2k1)
k3 = hf(ti +1
2h+ xi +
1
2k2)
k4 = hf(ti + h, xi + k3)
xi+1 = xi +1
6k1 +
1
3k2 +
1
3k3 +
1
6k4
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
The classical 4th order Runge-Kutta Method
function myRK4(N,tf)
m=5; %Mass
L=9.81; %length of the string
g=9.81; %gravitaional acceleration
beta=0; %
t0=0;
theta0=(1/10)*pi;
omega0=0;
h=(tf-t0)/N
tt=[t0:h:tf];
ttheta=zeros(size(tt));
oomega=zeros(size(tt));
ttheta(1)=theta0;
oomega(1)=omega0;
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
%The classic 4th order Runge Kutta method
for k=1:N
k1=h*(-beta/m/L*oomega(k)-g/L*sin(ttheta(k)));
m1=h*oomega(k);
k2=h*(-beta/m/L*(oomega(k)+k1/2)-g/L*(sin(ttheta(k)+m1/2)));
m2=h*(oomega(k)+m1/2);
k3=h*(-beta/m/L*(oomega(k)+k2/2)-g/L*(sin(ttheta(k)+m2/2)));
m3=h*(oomega(k)+k2/2);
k4=h*(-beta/m/L*(oomega(k)+k3/2)-g/L*(sin(ttheta(k)+m3/2)));
m4=h*(oomega(k)+k3);
oomega(k+1)=oomega(k)+1/6*(k1+2*k2+2*k3+k4);
ttheta(k+1)=ttheta(k)+1/6*(m1+2*m2+2*m3+m4);
end
% Plot the solutions.
plot(tt,oomega,tt,ttheta)
title(’Simple pendulum solved with 4th order Runge Kutta’)
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
In general the 4-th order Runge-Kutta Methods take the form:
(RK4) k1 = f(ti, xi)
k2 = hf(ti + c2h, xi + a21k1)
k3 = hf(ti + c3h, xi + a31k1 + a32k2)
k4 = hf(ti + c4h, xi + a41k1 + a42k2 + a43k3)
xi+1 = xi + ω1k1 + ω2k2 + ω3k3 + ω4k4
where the unknown parameters are organized as
0c2 a21c3 a31 a32c4 a41 a42 a43
ω1 ω2 ω3 ω4
known as Butcher’s table. As a result it is possible to constructvarious 4th order Runge-Kutta methods.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Question: When do RK methods fail?Example:
x1 = −100x1 (5)
x2 = −0.1x2. (6)
The state x1(t) = x1(0)e−100t and x2(t) = x2(0) = e−0.1t. x2(t)decays faster than x1(t). So it is not reasonable to compute suchsystems with the same step length h.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Advantages and Disadvantages of Simple RKMethods
Advantages
they are easy to implement
the are stable
Disadvantages
they require relatively large computer time
error estimation are not easy to be done
the above simple RK methods do not work well for stiffdifferential equations (eg. linear differential equations with widelyspread eigenvalues)
In particular, they are not good for systems of differentialequations with a mix of fast and slow state dynamics.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Adaptive Runge-Kutta MethodsThe choice of a good step size h is crucial. In particular, this isimportant for stiff differential equations:• too large h may lead to inaccurate approximation of the solution;• too small h forces the algorithm use large computation resources.⇒ The selection of the step length h is a compromise betweennumerical accuracy and speed of computation.I Adaptive adjustment of the step size h reduces (local) truncationerror for approximating x(ti+1) by xi+1:
‖x(ti+1)− xi+1‖Recall the Taylor approximation:
x(ti + h) = x(ti) + hx(ti) +h2
2x(ti) +
h3
3!
...x (ti) +
h4
4!
....x (ti) + . . .
= xi + C1h+ C2h2 + C3h
3 + C4h4 + . . .
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Adaptive Runge-Kutta Methods...
• Adaptive strategies adjust the step-length from iteration to iterationby testing the accuracy iterates.• Usually this test of accuracy is done by comparing lower and higherorder RK algorithms.⇒ Hence, adaptive RK methods combine lower and higher orderRK methods to adjust the step-size h.
(RK3) k1 = f(ti, xi)
k2 = hf(ti +1
2h+ xi +
1
2k1)
k3 = hf(ti + h, xi − k1 + 2k2)
xRK3i+1 = xi +
h
6(k1 + 4k2 + k3) .
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
(RK4s) k1 = f(ti, xi)
k2 = hf(ti +1
2h+ xi +
1
2k1)
k3 = hf(ti +1
2h+ xi +
1
2k2)
k4 = hf(ti + h, xi + k3)
xRK4i+1 = xi +
h
6(k1 + 2k2 + 2k3 + k4)
• Both xRK4i+1 and xRK4
i+1 are approximations for x(ti+1).Thus the local truncation errors (using the Taylor approximation ofx(t+ h) with ti+1 = ti + h):
from (RK3) is x(ti+1)− xRK3i+1 (ti+1) = C3h
3 + . . .from (RK4) x(ti+1)− xRK4
i+1 = C4h4 + . . .
⇒ (subtracting the second from the first)
xRK4i+1 − xRK3
i+1 ≈ C3h3
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
I Hence, the truncation error can be estimated in two useful ways:(i) on the hand, the local truncation error is can be approximated by
ErrTrunc = |xRK4i+1 − xRK3
i+1 | ≈ ‖C1‖h3 (7)
In the next iteration step we would like to choose a new step-sizehnew so that the truncation error below a give tolerance:
ErrTruncnew ≈ ‖C1‖h3new < ErrTol. (8)
I The new discretization step-size hnew should be chosen in such away that the resulting truncation ErrTruncnew lies below a giventolerance level ErrTol.I The step-size should be adjusted relative to the truncation errorcommitted in the previous step and the given error tolerance for thenext step (taking ratios of terms in equations (7) and (8))
‖C1‖h3new‖C1‖h3
<ErrTol
ErrTrunc.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Adaptive Runge-Kutta Methods...⇒
hnew < h
(ErrTol
ErrTrunc
)1/3
In particular the new h can be computed adaptively by
hnew = γh
(ErrTol
ErrTrunc
)1/3
for a fixed γ such that 0 < γ < 1.
(ii) on the other hand, comparing the algorithms (RK3) and (EK4s)s,
ErrTrunc = |xRK4i+1 − xRK3
i+1 | =h
6[−2k2 + k3 + k4]
These lead to an adaptive 3-4 Runge-Kutta algorithm.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Adaptive Runge-Kutta Methods...an RK34
• Set t0,x0, initial step size h, ErrorTol, tf and γ,while t < tf do• Compute
k1 = f(t, x)k2 = hf(ti + 1
2h+ xi + 12k1)
k3 = hf(ti + h, xi + 12k2)
k4 = hf(ti + h, xi + k3)• Set ErrTrunc← h
6‖2k2 − k3 − k4‖.if ErrTrunc ≤ ErrorTol then• Set xi+1 ← xi + h
6 (k1 + 2k2 + 2k3 + k4)• Set t← t+ h
end if• New step-size: h← γh
(ErrorTolErrTrunc
)1/3end while
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Common adaptive methods are the Runge-Kutta-Fehlberg(RKF)formulas.• RKF23 (a combination of 2nd and 3rd order RKs), RKF45, etc.Matlab ode functions:I ode23 is an adaptive Runge-Kutta methods and a combination of a2nd and 3rd order Runge-Kutta methods.I ode45 is an adaptive RK known as the Dormand-Prince method.Example: Mass-springer dumper system:
Mx+ Cx+ kx = F
with mass M = 5kg,dumper valueC = 10Ns/m, springconstant k = 1000N/m,F = sin(20t)N withforcing frequency ω = 20.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Mass-springer dumper system...MatlabTo solve 5x+ 10x+ 100x = sin(20t), with x(0) = 0, x(0) = 0.⇒ x1 = x2, x2 = −2x2 − 20x1 + 0.2 ∗ sin(20t).
function springDumper(N,tf)
t0=0;
h=(tf-t0)/N;
x10=0;
x20=0;
tt=t0:h:tf;
[t,x]= ode23(@diffEq,tt,[x10,x20]);
subplot(2,1,1), plot(t,x(:,1));
title(’Result from Ode23’)
[t,x]= ode45(@diffEq,tt,[x10,x20]);
subplot(2,1,2), plot(t,x(:,1),’r’);
title(’Result from Ode45’)
end
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Mass-springer dumper system...Matlab...
function dxdt=diffEq(t,x)
dx1dt = x(2);
dx2dt = -2*x(2)-20*x(1)+0.2*sin(20*t);
dxdt=[dx1dt;dx2dt];
end
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Adaptive Runge-KuttaMethods...Advantages/Disadvantages
Advantages:• provide better precision in the numerical approximation statetrajectories;• step-sizes can be automatically adapted to the dynamics of thestates;⇒ process dynamics with stiff-differential equation models can beefficiently solved
Disadvantages:• such algorithms may take enormous computation time;• simulation of dynamic systems with several state equations can becomputationally demanding;
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Multi-step and Implicit MethodsEuler, Runge-Kutta and adaptive RK methods are single-step andexplicit methods.
• Multi-step methods: use a set of earlier iterates up to the currentiterate xi to compute the next iterate xi+1.⇒ Earlier iterates xi, xi−1, . . . , xi−m are used to compute xi+1.For example:
a two-step method uses: xi and xi−1 to compute xi+1
a three-step method uses: xi, xi−1 and xi−2 to compute xi+1
• Implicit methods:
xi+1 = f(ti+1, xi+1),
where xi+1 = x(t)|t=ti+1 . Hence, xi+1 appears implcitily inf(ti+1, xi+1).
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Multi-step and Implicit Methods...I BDF is a well known efficient multi-step method:BDF = Backward Differentiation FormulaI While IRK is a well known implicit method:IRK = Implicit Runge-Kutta MethodI Both BDF and IRK are widely used for the direct discretization ofcontinuous dynamic optimization problems.I Particularly, when the IVP involves a stiff-differential equations,BDF and IRK are methods of choice.Stiff differential equations: A differential equation
x = f(t, x)
is stiff if the (Jacobi) matrix ∂f∂x has eigenvalues with widely spread
negative real parts.⇒ some states change fast while others are slower.Examples: mass-spring systems with large spring constants,(chemical) process engineering models, etc.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Multi-step and Implicit Methods...BDFI Adaptive RK methods can solve some stiff problems, but they arein general not efficient for a system with several states.(A) BDF: Given xi, xi−1, . . . , xi+1−m to compute xi+1. Find an m-thdegree polynomial P that interpolates the m+ 1 points
(ti+1, xi+1), (ti, xi), (ti−1, xi−1), . . . , (ti+1−m, xi+1−m),
A good choice for the interpolating polynomials P is
P (t) =
m∑j=0
xi+1−jLj(t)
with the Lagrange polynomial
Lj(t) =
m∏l = 0l 6= j
[t− ti+1−l
ti+1−j − ti+1−l
], j = 0, 1, . . . ,m.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
BDF...Observe that
P (ti+1−j) = xi+1−j , j = 0, 1, . . . ,m.
In particular P (ti+1) = xi+1.Thus, replace xi+1 by P (ti+1) to obtain
P (ti+1) = f(ti+1, xi+1). (∗)
But
P (ti+1) =
m∑j=0
xi+1−jLj(ti+1) = xi+1L0(ti+1) +
m∑j=1
xi+1−jLj(t)
Putting this into (*) we get:
xi+1 = −m∑j=1
xi+1−jLj(ti+1)
L0(ti+1)+
1
L0(ti+1)f(ti+1, xi+1)
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Define
aj =Lj(ti+1)
L0(ti+1), bm =
1
hL0(ti+1).
An m-step BDF Algorithm (BDF m):Given a1, . . . , am,bm
xi+1 = −m∑j=1
ajxi+1−j + bmhf(ti+1, xi+1)
Example:(a) A BDF 2 algorithm
xi+1 =
(4
3xi −
1
3xi−1
)+
2
3hf(ti+1, xi+1)
(b) A BDF 3 algorithm
xi+1 =
(4
3xi −
9
11xi−1 +
2
11xi−2
)+
6
11f(ti+1, xi+1)
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
BDF...• If f is a nonlinear function, all BDF algorithms require the solutionof nonlinear equaitons.=⇒ BDF is used in conjuction with a Newton method.
I Matlab function: ode15s uses a BDF algorithm.
Advantages:• are suitable for the solution of stiff differential equations• can be easily generalized to solve DAEs• BDF 2 up to BDF 6 are known to be stable• widely used in circuit simulationDisadvantages:• may require the solution of nonlinear equations• faces difficulties if the step-size h is constant (i.e. requiresadaptation of the step size).
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Multi-step and Implicit Methods...IRK
(B) IRK:I If xi is an approximation of x(ti), how to determine theapproximation xi+1 of x(ti+1)?Idea:Approximate x(ti+1) by xi+1 = P (ti+1), where P is a polynomialsatisfying (the requirements):
P (ti) = xi (9)
P (ti + τlh) = f(ti + τlh, P (ti + τlh)), l = 1, . . . ,m, (10)
where τ1, τ2, . . . , τm ∈ (0, 1) and h = ti+1 − ti the steps size.• Note that since P is a polynomial, P is also a polynomial.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
IRK..Now define P as
P (ti + τh) =
m∑l=1
f(ti + τlh, P (ti + τlh))Ll(τ),
with
Ll(τ) =
m∏j = 1l 6= j
[τ − τjτl − τj
](11)
• This special polynomial satisfies equation (10).• Integrating equation (9) on both sides∫ τj
0P (ti + τh) =
m∑l=1
f(ti + τlh, P (ti + τlh))
∫ τj
0Ll(τ), j = 1, . . . ,m.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
IRK..It follows that
(i) P (ti+τjh) = xi+h
m∑l=1
f(ti+τlh, P (ti+τlh))
∫ τj
0Ll(τ), j = 1, . . . ,m.
and for τj=1
(ii) P (ti + h) = xi + h
m∑l=1
f(ti + τlh, P (ti + τlh))
∫ 1
0Ll(τ)dτ.
Now, define
ajl =
∫ τj
0Ll(τ) (12)
bl =
∫ 1
0Ll(τ) (13)
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
IRK..An m-stage IRK Algorithm:Given the points τ1, τ2, . . . , τm ∈ [0, 1):
kl = xi + h
m∑j=1
aljf(ti + τjh, kj), l = 1, . . . ,m (14)
xi+1 = xi + h
m∑l=1
blf(ti + τlh, kl) (15)
• If τ1, . . . , τm are known the vector b = (b1, . . . , bm) and the matrixA = (alj) are easy to determine from equations (14) and (15).Butcher Tabeleau:
τ A
b>
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
The difference in IRK methods lies:• on the selection of the parameters τ1, . . . , τm• on special adjustments to reduce the size of the matrix A.• IRK methods are also known as collocation methods.Some examples:(a) Gauss-Legendre τ1 = 1/2−
√15/10, τ2 = 1/2 and
τ3 = 1/2 +√
15/10 (are roots of the 3rd degree LegendrePolynomila):
1/2−√
15/10 536
29 −
√1515
536 −
√1530
1/2 536
√1524
29
536 −
√1524
1/2 +√
15/10 536 +
√1530
29 +
√1515
536
518
49
518
(b) Radau IIA(c) Radau 5(d) Lobatto IIIA(e) Diagonally implicit RK methods
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
IRK..I Matlab function: ode23tb is an implementation of an IRKalgorithm.
Example: A continually stirred homoiothermal reactor
A continuously stirred reactor (CSTR) with feed A and product B.Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
• Concentration of A in the reactor: CA• The feed concentration: CAf• Mixture temperature in the CSTR: T• Bulk volume of the mixture in the CSTR V• Density and specific heat constants: ρ, Cp• k0 reaction constant
Constants:k0 = 3.493x107h−1, E = 11843kcal/kmol,(−∆H) = 5960kcal/kmol, ρCp = 500kcal/m3/KUA = 150kcal/h/K,R = 1.987kcal/kmol/K, V =1m3, q = 1m3/h,CAi = 10kmol/m3, Ti = 298K,Tj = 298K
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
dCAdt
=q
V(CAi − CA) k0exp(−E/RT )CA
dT
dt=
q
V(Ti − T ) +
(−∆H)k0exp(−E/RT )CAρCp
+UAρCpV
(Tj − T )
Objective: Study the dynamics of the CSTR.
xo2 =[6,399];
df=@(t,x)cstr(x);
[t2,x2]=ode15s(df,[0,100],xo2,[]);
subplot(3,1,1);
[t2,x2]=ode23tb(df,[0,100],xo2,[]);
subplot(3,1,2);
plotyy(t2,x2(:,1),t2,x2(:,2));
subplot(3,1,3);
[t2,x2]=ode45(df,[0,100],xo2,[]);
plotyy(t2,x2(:,1),t2,x2(:,2));
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
IRK...Advantages/Disadvantages
Advantages:• can be used to solve stiff differential equations and DAEs• widely used for the discretization of optimal control problems
Disadvantages:• to guarantee stability of higher-order IRK some of the τl’s need tobe chosen outside [0, 1] (J. C. Butcher)• may require the solution of nonlinear equations implying intensivecomputations• adaptive selection of the step-size h can be resource intensive• multi-step IRK is still a research topic⇒ The method of ”orthogonal collocation on finite elements” is ageneralizations of IRK methods.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Boundary Value Problems
Note that: The solution x∗(t) of an IVP depends on the initial valuest0 and x0. If we change one or both of these values, the solution x∗(t)also changes. This dependence is indicated by writing x∗(t; t0, x0).
In general, a boundary value problems has the form
(BV P ) x = f(t, x), x(t0) = A, x(tf ) = B.
• The function f(t, x) is assumed continuous and differentiable.• The solution x(t) is required to satisfy the two condones x(t0) = Aand x(tf ) = B known as boundary conditions.
Objective: to compute a function x(t) that satisfies the BVP.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Boundary Value Problems (BVPs)Difficulties with BVPS:
A BVP (in contrast to IVPs) can have several solutions (i.e.,there can be several trajectories (paths) to arrive at x(tf ) = Bstarting from x(t0) = A.)
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Single-ShootingStrategy: BVPs are solved by solving a sequence of IVPsHence, any one of: Euler, Runge-Kuta, BDF, IRK, etc. can be used inthe solution strategy of BVPs.Single-ShootingIdea of single-shooting:
from the initial position x(t0) shoot with an angle to hit thetarget at x(tf ).
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Single-Shooting...
• Question: What is the shooting angle?Note that: the shooting (aiming) angle is related with the slopex(t0) of the trajectory x(t) (since tanα = x(t0)).• Nevertheless, x(t0) is not given (unknown). That is, x(t0) =?• So we make a guess and take
x(t0) = u.
(But, with this guess (trial shooting angle) we may not hit the target).• Now, drop the end-time condition x(tf ) = B and consider the initialvalue problem
x = f(t, x), x(t0) = A, x(t0) = u.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
• Use the transformation
x1 = x, x2 = x1 ⇒ x2 = x1 = x
to write the problem as a system of first-order IVPs
x1 = x2, x1(t0) = A (16)
x2 = ft(t, x1) + fx(t, x1)x2, x2(t0) = u. (17)
• Solve this IVP using any one of the methods for IVPs to obtain a
solution x∗(t) =
(x∗1(t)x∗2(t)
).
• If x∗1(t) satisfies the end-point boundary condition at t = tf ; i.e. if
x∗1(tf ) = B
then we are done: x∗1(t) solves the (BVP).
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Otherwise,x∗1(tf ) 6= B.
• Our earlier guess is wrong and we have to adjust our shooting(aiming) angle, i.e. take another guess for u.
• Question: How to make a good guess for u?
• We known that the solution x∗(t) depends on the initial values ofthe IVP, so we write x∗(t;u).• The solution of the system of equations
F (u) = x∗1(tf , u)−B = 0
provides a good guess for u. In general, this can be a system ofnonlinear equations.• We use the Newton’s method to solve such system of nonlinearequations.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
A Newton Algorithm
Initialize: Set u0 and tolerance εwhile ‖F (u(k))‖ > ε do• Compute the vector F (u(k)) and the Jacobian matrix F ′(u(k)).• Solve the system of linear equations
F ′(u(k))d = −F (u(k))
to obtain dk as a solution.• Set u(k+1) = u(k) + dk
end while
• To solve the system of nonlinear equations we need the values ofF (u(k)) and F ′(u(k)).• The value of F (u(k)) is found as F (uk) = x∗(tf , uk)−B.• It remains to compute the derivative F ′(u(k)).
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
We us the forward-difference numerical approximation for F ′(uk):
F ′(u(k)) =F (u(k) + ∆)− F (u(k))
∆
for a given ∆. Note that F (u(k) + ∆) = x∗1(tf , u(k) + ∆)−B.
• Since x∗1(tf , u(k) + ∆) is unknown, we need to re-solve the problem
(16) & (17) using x2(t0) = u(k) + ∆
x1 = x2, x1(t0) = A (18)
x2 = ft(t, x1) + fx(t, x1)x2, x2(t0) = u(k) + ∆. (19)
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Single Shooting ...A Single-Shooting Algorithm
Select u0, ∆ > 0 and tolerance ε > 0while ‖F (u(k))‖ > ε do• Compute inital value problems (16) & (17) with x2(tf ) = u(k)
and (18) & (19) with x2(tf ) = u(k) + ∆• Compute the vector F (u(k)) and the Jacobian matrix F ′(u(k)):
F ′(u(k)) =F (u(k) + ∆)− F (u(k))
ƥ Solve the system of linear equations
F ′(u(k))d = −F (u(k))
to obtain dk as a solution.• Set k ← k + 1• Set u(k) ← u(k) + dk
end whileLecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Single Shooting ... Advantages/Disadvantages
Advantages:• Simple to implement• effective when the interval [t0, tf ] is short.Disadvantages:• for some problems the algorithm can become un- stable• requires good initial guesses for the IVPs• the solution strategy is highly influenced by the length of theshooting (distance) interval [t0, tf ]. Hence, large numerber ofiterations may be needed.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Multiple-Shooting Method
• Consider again the problem
x = f(t, x), x(t0) = A, x(tf ) = B. (20)
• In the multiple-shooting method, the interval [t0, tf ] is divided intosubintervals [t0, t1], [t1, t2], . . . , [tm−1, tm] by using time-pointst0 = t1, t2, . . . , tm−1, tm = tf .• On each subinterval tj ≤ t ≤ tj+1, j = 0, 1, . . . ,m− 1, solve theinitial value problems
x = f(t, x), x(tj) = uj , tj ≤ t ≤ tj+1, j = 0, 1, . . . ,m− 1, (21)
to get a solution x(t; tj , uj) .
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Multiple-Shooting Method...
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Multiple-Shooting Method...
• Test if all of the following equations hold true at the end points ofeach sub-interval
x(t1; t0, u0) = u1 (22)
x(t2; t1, u1) = u2 (23)... (24)
x(tm−1; tm−2, um−2) = um−1 (25)
x(tm; tm−1, um−1) = um = B. (26)
• If yes, we are done. The solution obtained satisfies the (BVP).• Otherwise, make a new and a better guess of the u’s.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
• To find a better guess for the u′s, solve the system of (possiblynonlinear) equations:
F1(u) = x(t1; t0, A)− u1 = 0 (27)
F2(u) = x(t2; t1, u1)− u2 = 0 (28)... (29)
Fm−1(u) = x(tm−1; tm−2, um−2)− um−1 = 0 (30)
Fm(u) = x(tm; tm−1, um−1)−B = 0. (31)
• Put all the guess values into a variable vector u as
u =
u1u2...
um−1
• Note that, each of the uj ’s can be themselves vectors.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
• Hence, solve the system of equations
F (u) =
F1(u)F2(u)
...Fm(u)
= 0
using the Newton’s Algorithm to find a better guess for u.• For a given u(k), the vector F (u(k)) can be computed usingequations (27)-(31). Thus, we have
F (u(k)) =
F1(u
(k))
F2(u(k))
...
Fm(u(k))
=
x(t1; t0, A)− u(k)1
x(t2; t1, u(k)1 )− u(k)2...
x(tm−1; tm−2, u(k)m−2)− u
(k)m−1
x(tm; tm−1, u(k)m−1)−B
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
• The elements of the Jacobian matrix F ′(u(k)) can be computedusing finite differences:
∂Fj∂ul
=Fj(u
(k) + ∆el)− F (u(k))
∆
where el is the l-th unit vector in RK andK = length(u1) + . . .+ length(um).• The value of Fj(u
(k) + ∆el) is obtained by solve the IVP
x = f(t, x), x(tj) = u(k)j + ∆elj , tj ≤ t ≤ tj+1, j = 0, . . . ,m− 1.
• Once the values F (u(k)) and F ′(u(k)) are available, solve thesystem of linear equaitons
F ′(u(k))d = −F (u(k))
to obtain a solution d(k), and update the guess vector asu(k+1) = u(k) + d(k).
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
• In fact, the Jacobian F ′(u(k)) has the special structure
F ′(u(k)) =
−IX2 −I
X3 −I. . .
. . .
Xm−1 −IXm
,
where (the following are sensitivities w.r.t. the initial values)
X2(t2; t1, u1) =∂x(t2; t1, u1)
∂u
∣∣∣∣u=u(k)
(32)
... (33)
Xm−1(tm−1; tm−2, um−2) =∂x(tm−1; tm−2, um−2)
∂u
∣∣∣∣u=u(k)
(34)
Xm(tm; tm−1, um−1) =∂x(tm; tm−1, um−1)
∂u
∣∣∣∣u=u(k)
(35)Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
A Multiple-Shooting Algorithm
• Select shooting-points t0 < t1 < . . . , tm = tf , initial guess
(u(0)1 , u
(0)2 , . . . , u
(0)m ), ∆ > 0 and tolerance ε > 0
while ‖F (u(k))‖ > ε do
• Solve the IVPs in equation (21) twice to obtain x(tj+1; tj , u(k)j )
and X(tj+1; tj , u(k)j ), j = 0, 1, . . . ,m− 1
• Construct the vector the vector F (u(k)) and the Jacobianmatrix F ′(u(k)).• Solve the system of linear equations
F ′(u(k))d = −F (u(k))
to obtain dk as a solution.• Set k ← k + 1• Set u(k) ← u(k) + dk
end whileLecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Multiple-Shooting ... Remarks
Remark (Parallel Computation)
I The most time-consuming step in the multiple-shooting method isthe computation of the values of the vector F (u) and Jacobian matrixF ′(u) for each Newton iteration.I The special structure of the Jacobian F ′(u) (also of F (u))indicates that:• each j-th block row of the Jacobian can be obtained through thesolution of the IVP only on the j-th shooting interval.⇒ the computation of the Jacobian can be done by solving smallerIVPs in parallel, correspond to the block rows of the Jacobian.
Choosing the shooting points
I The optimal choice of the shooting points t1, t2, . . . , tm−1 can beproblem dependent. Hence, they can be considered as variables to bedetermined through some computational scheme.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
Multiple-Shooting ... Advantages/Disadvantages
Advantages:• it is efficient when the time-interval [t0, tf ] too long;• it is easy to parallelize and, hence, applicable to problems of largedimensions;• accuracy of computations can be improved by considering additionalintermediate points on the interval [tj , tj+1];• it is applicable to solve stiff differential equations and DAEsDisadvantage:• For large-number of sub-intervals it can be computationallyintensive;• The Newton-step may require the solution of a large set of linearequations requiring special linear-algebra strategies,• The choice of the shooting time points t1, t2, . . . , tm−1 has animpact on the quality of the solution.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
References and Toolboxes
References:• U. M. Ascherm, L. R. Petzold: Computer methods for ordinarydifferential equations and differential algebraic equations. SISM 1998.• L. F. Shampine and M. W. Reichelt: The Matlab ODE Suite, SIAMJ. Sci. Comput., V. 18, pp. 1-22, 1997.• R. Ashino, M. Nagase: Behind and beyond the Matlab ODEToolbox. Comput. Math. Appl. V. 40, pp. 491 – 512, 2000.• L. F. Shampine, J. Kierzenak:. Solving boundary value problems forordinary differential equations in Matlab bvp4c.• W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery:Numerical recipes in C++ (3r ed.). Cambridge University Press, 2007.
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau
References and Toolboxes...
Toolboxes:
The Matlab ODE-Toolbox found under the MATrix LABoratory(MATLAB) software - The Math Works.
SUite of Nonlinear and DIfferential/ALgebraic equation Solvers(Sundials) :https://computation.llnl.gov/casc/sundials/main.html
Modelica : https://modelica.org/
Scialb: http://www.scilab.org/
EMSO (Environment for Modelling, Simulation andOptimisation): http://www.vrtech.com.br/rps/emso.html
Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations
TU Ilmenau