nonlinear regression - math for...
TRANSCRIPT
3/19/2015 http://numericalmethods.eng.usf.edu 1
Nonlinear Regression
Major: All Engineering Majors
Authors: Autar Kaw, Luke Snyder
http://numericalmethods.eng.usf.eduTransforming Numerical Methods Education for STEM
Undergraduates
Nonlinear Regression
)( bxaey =
)( baxy =
+=
xbaxy
Some popular nonlinear regression models:
1. Exponential model:
2. Power model:
3. Saturation growth model:
4. Polynomial model: )( 10m
mxa...xaay +++=
http://numericalmethods.eng.usf.edu3
Nonlinear RegressionGiven n data points ),( , ... ),,(),,( 2211 nn yxyx yx best fit )(xfy =
to the data, where )(xf is a nonlinear function of x .
Figure. Nonlinear regression model for discrete y vs. x data
)(xfy =
),(nn
yx
),(11
yx
),(22
yx
),(ii
yx)(
iixfy −
http://numericalmethods.eng.usf.edu4
RegressionExponential Model
http://numericalmethods.eng.usf.edu5
Exponential Model),( , ... ),,(),,( 2211 nn yxyx yxGiven best fit bxaey = to the data.
Figure. Exponential model of nonlinear regression for y vs. x data
bxaey =
),(nn
yx
),(11
yx
),(22
yx
),(ii
yxibx
i aey −
http://numericalmethods.eng.usf.edu6
Finding Constants of Exponential Model
( )∑=
−=n
i
bx
ir iaeyS
1
2The sum of the square of the residuals is defined as
Differentiate with respect to a and b
( )( ) 021
=−−=∂∂ ∑
=
ii bxn
i
bxi
r eaeyaS
( )( ) 021
=−−=∂∂ ∑
=
ii bxi
n
i
bxi
r eaxaeybS
http://numericalmethods.eng.usf.edu7
Finding Constants of Exponential Model
Rewriting the equations, we obtain
01
2
1=∑+∑−
==
n
i
bxn
i
bxi
ii eaey
01
2
1=∑−∑
==
n
i
bxi
n
i
bxii
ii exaexy
http://numericalmethods.eng.usf.edu8
Finding constants of Exponential Model
Substituting a back into the previous equation
01
2
1
2
1
1=∑
∑
∑−∑
=
=
=
=
n
i
bxin
i
bx
bxn
ii
bxi
n
ii
i
i
i
i exe
eyexy
The constant b can be found through numerical methods such as bisection method.
∑
∑=
=
=n
i
bx
n
i
bxi
i
i
e
eya
1
2
1
Solving the first equation for a yields
http://numericalmethods.eng.usf.edu9
Example 1-Exponential Model
t(hrs) 0 1 3 5 7 91.000 0.891 0.708 0.562 0.447 0.355
Many patients get concerned when a test involves injection of a radioactive material. For example for scanning a gallbladder, a few drops of Technetium-99m isotope is used. Half of the Technetium-99m would be gone in about 6 hours. It, however, takes about 24 hours for the radiation levels to reach what we are exposed to in day-to-day activities. Below is given the relative intensity of radiation as a function of time.
Table. Relative intensity of radiation as a function of time.
γ
http://numericalmethods.eng.usf.edu10
Example 1-Exponential Model cont.
Find: a) The value of the regression constants A λand
b) The half-life of Technetium-99m
c) Radiation intensity after 24 hours
The relative intensity is related to time by the equationtAeλγ =
http://numericalmethods.eng.usf.edu11
Plot of data
http://numericalmethods.eng.usf.edu12
Constants of the Model
The value of λ is found by solving the nonlinear equation
( ) 01
2
1
2
1
1=∑
∑
∑−∑=
=
=
=
=
n
i
tin
i
t
n
i
ti
ti
n
ii
i
i
i
i ete
eetf λ
λ
λ
λγ
γλ
∑
∑
=
== n
i
t
n
i
ti
i
i
e
eA
1
2
1
λ
λγ
tAeλγ =
http://numericalmethods.eng.usf.edu13
Setting up the Equation in MATLAB
( ) 01
2
1
2
1
1=∑
∑
∑−∑=
=
=
=
=
n
i
tin
i
t
n
i
ti
ti
n
ii
i
i
i
i ete
eetf λ
λ
λ
λγ
γλ
t (hrs) 0 1 3 5 7 9γ 1.000 0.891 0.708 0.562 0.447 0.355
http://numericalmethods.eng.usf.edu14
Setting up the Equation in MATLAB
( ) 01
2
1
2
1
1=∑
∑
∑−∑=
=
=
=
=
n
i
tin
i
t
n
i
ti
ti
n
ii
i
i
i
i ete
eetf λ
λ
λ
λγ
γλ
t=[0 1 3 5 7 9]gamma=[1 0.891 0.708 0.562 0.447 0.355]syms lamdasum1=sum(gamma.*t.*exp(lamda*t));sum2=sum(gamma.*exp(lamda*t));sum3=sum(exp(2*lamda*t));sum4=sum(t.*exp(2*lamda*t));f=sum1-sum2/sum3*sum4;
1151.0−=λ
http://numericalmethods.eng.usf.edu15
Calculating the Other Constant
The value of A can now be calculated
∑
∑
=
== 6
1
2
6
1
i
t
i
ti
i
i
e
eA
λ
λγ9998.0=
The exponential regression model then iste 1151.0 9998.0 −=γ
http://numericalmethods.eng.usf.edu16
Plot of data and regression curve
te 1151.0 9998.0 −=γ
http://numericalmethods.eng.usf.edu17
Relative Intensity After 24 hrs
The relative intensity of radiation after 24 hours
( )241151.09998.0 −×= eγ2103160.6 −×=
This result implies that only
%317.61009998.0
10316.6 2
=×× −
radioactive intensity is left after 24 hours.
http://numericalmethods.eng.usf.edu18
Homework• What is the half-life of Technetium-99m
isotope?• Write a program in the language of your
choice to find the constants of the model.
• Compare the constants of this regression model with the one where the data is transformed.
• What if the model was ?
http://numericalmethods.eng.usf.edu19
teλγ =
THE END
http://numericalmethods.eng.usf.eduhttp://numericalmethods.eng.usf.edu20
Polynomial Model),( , ... ),,(),,( 2211 nn yxyx yxGiven best fit
mm
xa...xaay +++=10
)2( −≤ nm to a given data set.
Figure. Polynomial model for nonlinear regression of y vs. x data
mm
xaxaay +++=
10
),(nn
yx
),(11
yx
),(22
yx
),(ii
yx)(
iixfy −
http://numericalmethods.eng.usf.edu21
Polynomial Model cont.The residual at each data point is given by
mimiii xaxaayE −−−−= ...10
The sum of the square of the residuals then is
( )∑
∑
=
=
−−−−=
=
n
i
mimii
n
iir
xaxaay
ES
1
210
1
2
...
http://numericalmethods.eng.usf.edu22
Polynomial Model cont.To find the constants of the polynomial model, we set the derivatives with respect to ia where
( )
( )
( ) 0)(....2
0)(....2
0)1(....2
110
110
1
110
0
=−−−−−=∂∂
=−−−−−=∂∂
=−−−−−=∂∂
∑
∑
∑
=
=
=
mi
n
i
mimii
m
r
i
n
i
mimii
r
n
i
mimii
r
xxaxaayaS
xxaxaayaS
xaxaayaS
,,1 mi = equal to zero.
http://numericalmethods.eng.usf.edu23
Polynomial Model cont.These equations in matrix form are given by
=
∑
∑
∑
∑∑∑
∑∑∑
∑∑
=
=
=
==
+
=
=
+
==
==
n
ii
mi
n
iii
n
ii
mn
i
mi
n
i
mi
n
i
mi
n
i
mi
n
ii
n
ii
n
i
mi
n
ii
yx
yx
y
a
aa
xxx
xxx
xxn
1
1
1
1
0
1
2
1
1
1
1
1
1
2
1
11
......
...
...........
...
...
The above equations are then solved for maaa ,,, 10
http://numericalmethods.eng.usf.edu24
Example 2-Polynomial Model
Temperature, T(oF)
Coefficient of thermal
expansion, α (in/in/oF)
80 6.47×10−6
40 6.24×10−6
−40 5.72×10−6
−120 5.09×10−6
−200 4.30×10−6
−280 3.33×10−6
−340 2.45×10−6
Regress the thermal expansion coefficient vs. temperature data to a second order polynomial.
1.00E-06
2.00E-06
3.00E-06
4.00E-06
5.00E-06
6.00E-06
7.00E-06
-400 -300 -200 -100 0 100 200
Temperature, oF
Ther
mal
exp
ansi
on c
oeffi
cien
t, α
(in/in
/o F)
Table. Data points for temperature vs
Figure. Data points for thermal expansion coefficient vs temperature.
α
http://numericalmethods.eng.usf.edu25
Example 2-Polynomial Model cont.
2210 TaTaaα ++=
We are to fit the data to the polynomial regression model
=
∑
∑
∑
∑∑∑
∑∑∑
∑∑
=
=
=
===
===
==
n
iii
n
iii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
T
Taaa
TTT
TTT
TTn
1
2
1
1
2
1
0
1
4
1
3
1
2
1
3
1
2
1
1
2
1
α
α
α
The coefficients 210 , a,aa are found by differentiating the sum of thesquare of the residuals with respect to each variable and setting thevalues equal to zero to obtain
http://numericalmethods.eng.usf.edu26
Example 2-Polynomial Model cont.The necessary summations are as follows
Temperature, T(oF)
Coefficient of thermal expansion,
α (in/in/oF)
80 6.47×10−6
40 6.24×10−6
−40 5.72×10−6
−120 5.09×10−6
−200 4.30×10−6
−280 3.33×10−6
−340 2.45×10−6
Table. Data points for temperature vs. α 57
1
2 105580.2 ×=∑=i
iT
77
1
3 100472.7 ×−=∑=i
iT
107
1
4 101363.2∑=
×=i
iT
57
1103600.3 −
=
×=∑i
iα
37
1106978.2 −
=
×−=∑i
iiTα
17
1
2 105013.8 −
=
×=∑i
iiT α
http://numericalmethods.eng.usf.edu27
Example 2-Polynomial Model cont.
××−×
=
××−××−××−××−
−
−
−
1
3
5
2
1
0
1075
752
52
105013.8106978.2
103600.3
101363.2100472.7105800.2100472.7105800.210600.8
105800.2106000.80000.7
aaa
Using these summations, we can now calculate 210 , a,aa
Solving the above system of simultaneous linear equations we have
×−××
=
−
−
−
11
9
6
2
1
0
102218.1102782.6100217.6
aaa
The polynomial regression model is then
21196
2210
T101.2218T106.2782106.0217
α−−− ×−×+×=
++= TaTaa
http://numericalmethods.eng.usf.edu28
Transformation of DataTo find the constants of many nonlinear models, it results in solving simultaneous nonlinear equations. For mathematical convenience, some of the data for such models can be transformed. For example, the data for an exponential model can be transformed.
As shown in the previous example, many chemical and physical processes are governed by the equation,
bxaey =Taking the natural log of both sides yields,
bxay += lnlnLet yz ln= and aa ln0 =
(implying) oaea = with ba =1
We now have a linear regression model where xaaz 10 +=
http://numericalmethods.eng.usf.edu29
Transformation of data cont.Using linear model regression methods,
_
1
_
0
1
2
1
2
11 11
xaza
xxn
zxzxna
n
i
n
iii
n
ii
n
i
n
iiii
−=
−
−=
∑ ∑
∑∑ ∑
= =
== =
Once 1,aao are found, the original constants of the model are found as
0
1aea
ab
=
=
http://numericalmethods.eng.usf.edu30
THE END
http://numericalmethods.eng.usf.eduhttp://numericalmethods.eng.usf.edu31
Example 3-Transformation of data
t(hrs) 0 1 3 5 7 9
1.000 0.891 0.708 0.562 0.447 0.355
Many patients get concerned when a test involves injection of a radioactive material. For example for scanning a gallbladder, a few drops of Technetium-99m isotope is used. Half of the Technetium-99m would be gone in about 6 hours. It, however, takes about 24 hours for the radiation levels to reach what we are exposed to in day-to-day activities. Below is given the relative intensity of radiation as a function of time.
Table. Relative intensity of radiation as a function
γ
of time
0
0.5
1
0 5 10
Rela
tive
inte
nsity
of r
adia
tion,
γ
Time t, (hours)
Figure. Data points of relative radiation intensityvs. time
http://numericalmethods.eng.usf.edu32
Example 3-Transformation of data cont.
Find: a) The value of the regression constants A λand
b) The half-life of Technetium-99m
c) Radiation intensity after 24 hours
The relative intensity is related to time by the equationtAeλγ =
http://numericalmethods.eng.usf.edu33
Example 3-Transformation of data cont.
tAeλγ =Exponential model given as,
( ) ( ) tA λγ += lnlnAssuming γln=z , ( )Aao ln= and λ=1a we obtain
taaz10
+=This is a linear relationship between z and t
http://numericalmethods.eng.usf.edu34
Example 3-Transformation of data cont.
Using this linear relationship, we can calculate 10 ,aa
∑ ∑
∑∑ ∑
= =
== =
−
−=
n
i
n
ii
n
ii
n
i
n
iiii
ttn
ztztna
1
2
1
21
11 11
andtaza 10 −=
where
1a=λ
0aeA =
http://numericalmethods.eng.usf.edu35
Example 3-Transformation of Data cont.
123456
013579
10.8910.7080.5620.4470.355
0.00000−0.11541−0.34531−0.57625−0.80520−1.0356
0.0000−0.11541−1.0359−2.8813−5.6364−9.3207
0.00001.00009.000025.00049.00081.000
25.000 −2.8778 −18.990 165.00
Summations for data transformation are as follows
Table. Summation data for Transformation of data model
i it iγ iiz γln= ii
zt 2it
∑
With 6=n
000.256
1∑=
=i
it
∑=
−=6
18778.2
ii
z
∑=
−=6
1990.18
iii
zt
00.1656
1
2 =∑=i
it
http://numericalmethods.eng.usf.edu36
Example 3-Transformation of Data cont.
Calculating 10 ,aa( ) ( )( )
( ) ( )21 2500.16568778.225990.186
−−−−
=a 11505.0−=
( )62511505.0
68778.2
0 −−−
=a 4106150.2 −×−=
Since( )Aa ln0 =0aeA =
4106150.2 −×−= e 99974.0=
11505.01 −== aλalso
http://numericalmethods.eng.usf.edu37
Example 3-Transformation of Data cont.
Resulting model is te 11505.099974.0 −×=γ
0
0.5
1
0 5 10
Time, t (hrs)
Relative Intensity
of Radiation,
te 11505.099974.0 −×=γ
Figure. Relative intensity of radiation as a function of temperature using transformation of data model.
http://numericalmethods.eng.usf.edu38
Example 3-Transformation of Data cont.
The regression formula is thente 11505.099974.0 −×=γ
b) Half life of Technetium-99m is when02
1
=
=t
γγ
( ) ( )
( )hours.t
.t..e
e.e.
t.
.t.
0248650ln115050
50
99974021999740
115080
0115050115050
==−
=
=×
−
−−
http://numericalmethods.eng.usf.edu39
Example 3-Transformation of Data cont.
c) The relative intensity of radiation after 24 hours is then( )2411505.099974.0 −= eγ
063200.0=This implies that only %3216.6100
99983.0103200.6 2
=×× −
of the radioactive
material is left after 24 hours.
http://numericalmethods.eng.usf.edu40
Comparison Comparison of exponential model with and without data Transformation:
With data Transformation
(Example 3)
Without data Transformation
(Example 1)
A 0.99974 0.99983
λ −0.11505 −0.11508
Half-Life (hrs) 6.0248 6.0232
Relative intensity after 24 hrs. 6.3200×10−2 6.3160×10−2
Table. Comparison for exponential model with and without dataTransformation.
http://numericalmethods.eng.usf.edu41
Additional ResourcesFor all resources on this topic such as digital audiovisual lectures, primers, textbook chapters, multiple-choice tests, worksheets in MATLAB, MATHEMATICA, MathCad and MAPLE, blogs, related physical problems, please visit
http://numericalmethods.eng.usf.edu/topics/nonlinear_regression.html