Analisis_Numerik 1

Numerical AnalysisTopic 4:

Least Squares Curve Fitting

Read Chapter 17 of the textbook

Analisis_Numerik 2

Motivation

Given a set of experimental data:

x 1 2 3

y 5.1 5.9 6.3

• The relationship between

x and y may not be clear.

• Find a function f(x) that

best fit the data1 2 3

Analisis_Numerik 3

Motivation

In engineering, two types of applications are encountered:

Trend analysis: Predicting values of dependent variable, may include extrapolation beyond data points or interpolation between data points.

Hypothesis testing: Comparing existing mathematical model with measured data.

1. What is the best mathematical function f that represents the dataset?

2. What is the best criterion to assess the fitting of the function f to the data?

Analisis_Numerik 4

Curve Fitting

Given a set of tabulated data, find a curve or a function that best represents the data.

Given:

1.The tabulated data

2.The form of the function

3.The curve fitting criteria

Find the unknown coefficients

Analisis_Numerik 5

Least Squares Regression

Linear Regression

Fitting a straight line to a set of paired

observations:

(x1, y1), (x2, y2),…,(xn, yn).

y=a0+a1x+e

a1-slope.

a0-intercept.

e-error, or residual, between the model and the observations.

Analisis_Numerik 6

Selection of the Functions

known. are)(

)()(

)(

)(

)(

0

0

2

xg

xgaxfGeneral

xaxfPolynomial

cxbxaxfQuadratic

bxaxfLinear

k

m

k

kk

n

k

k

k

Analisis_Numerik 7

Decide on the Criterion

)(

:tion)(Interpola MatchingExact .2

))(( minimize

:n RegressioSquaresLeast .1

1

2

ii

n

i

ii

xfy

xfy

Chapter 17

Chapter 18

Analisis_Numerik 8

Least Squares Regression

xi x1 x2 …. xn

yi y1 y2 …. yn

Given:

The form of the function is assumed to be

known but the coefficients are unknown.

222))(())(( iiiii yxfxfye

The difference is assumed to be the result of

experimental error.

Analisis_Numerik 9

Determine the Unknowns

?),( :minimize to and obtain wedoHow

)(),(

:minimize to and find want toWe

1

2

baba

ybxaba

ba

n

i

ii

Analisis_Numerik 10

Determine the Unknowns

0),(

0),(

:minimum for the condition Necessary

b

ba

a

ba

Analisis_Numerik 11

Determining the Unknowns

02),(

02),(

1

1

i

n

i

ii

n

i

ii

xybxab

ba

ybxaa

ba

Analisis_Numerik 12

Normal Equations

n

i

ii

n

i

i

n

i

i

n

i

i

n

i

i

yxbxax

ybxan

11

2

1

11

Analisis_Numerik 13

Solving the Normal Equations

n

i

i

n

i

i

n

i

i

n

i

i

n

i

i

n

i

i

n

i

ii

xbyn

a

xxn

yxyxn

b

11

2

11

2

111

1

Analisis_Numerik 14

Example 1: Linear Regression

n

i

ii

n

i

i

n

i

i

n

i

i

n

i

i

yxbxax

ybxan

bxaf(x)

11

2

1

11

:Equations

:Assume

x 1 2 3

y 5.1 5.9 6.3

Analisis_Numerik 15

Example 1: Linear Regression

i 1 2 3 sum

xi 1 2 3 6

yi 5.1 5.9 6.3 17.3

xi2 1 4 9 14

xi yi 5.1 11.8 18.9 35.8

60.04.5667:

8.35146

3.1763

:Equations

baSolving

ba

ba

Analisis_Numerik 16

Multiple Linear Regression

Example:

Given the following data:

Determine a function of two variables:

f(x,t) = a + b x + c t

That best fits the data with the least sum of the square of errors.

t 0 1 2 3

x 0.1 0.4 0.2 0.2

y 3 2 1 2

Analisis_Numerik 17

Solution of Multiple Linear Regression

Construct , the sum

of the square of the

error and derive the

necessary conditions by

equating the partial

derivatives with respect

to the unknown

parameters to zero, then

solve the equations.

t 0 1 2 3

x 0.1 0.4 0.2 0.2

y 3 2 1 2

Analisis_Numerik 18

Solution of Multiple Linear Regression

02),,(

02),,(

02),,(

:conditions Necessary

),,( ,),(

1

1

1

1

2

i

n

i

iii

i

n

i

iii

n

i

iii

n

i

iii

tyctbxac

cba

xyctbxab

cba

yctbxaa

cba

yctbxacbactbxatxf

Analisis_Numerik 19

System of Equations

) ()() (

) () ()(

1

2

1 11

11 1

2

1

11 1

n

i

iii

n

i

n

i

ii

n

i

i

n

i

iiii

n

i

n

i

i

n

i

i

n

i

i

n

i

n

i

ii

yttctxbta

yxtxcxbxa

ytcxbna

Analisis_Numerik 20

Example 2: Multiple Linear Regression

i 1 2 3 4 Sum

ti 0 1 2 3 6

xi 0.1 0.4 0.2 0.2 0.9

yi 3 2 1 2 8

xi2 0.01 0.16 0.04 0.04 0.25

xi ti 0 0.4 0.4 0.6 1.4

xi yi 0.3 0.8 0.2 0.4 1.7

ti2 0 1 4 9 14

ti yi 0 2 2 6 10

Analisis_Numerik 21

Example 2: System of Equations

txctbxatxf

cba

Solving

cba

cba

cba

38298.0 7021.19574.2),(

38298.0 ,7021.1 ,9574.2

:

10144.16

7.14.125.09.0

869.0 4

Analisis_Numerik 22

Lecture 19Nonlinear Least Squares Problems

Examples of Nonlinear Least Squares

Solution of Inconsistent Equations

Continuous Least Square Problems

Polynomial Regression

The least squares method can be extended to fit the data to a higher-order polynomial

Analisis_Numerik 23

0),,(

,0),,(

,0),,(

:conditions Necessary

),,( Minimize

))(( ,)(

1

22

222

c

cba

b

cba

a

cba

ycxbxacba

yxfecxbxaxf

n

i

iii

ii

Equations for Quadratic Regression

Analisis_Numerik 24

0 2),,(

0 2),,(

02),,(

),,( Minimize

2

1

2

1

2

1

2

1

22

i

n

i

iii

i

n

i

iii

n

i

iii

n

i

iii

xycxbxac

cba

xycxbxab

cba

ycxbxaa

cba

ycxbxacba

Normal Equations

Analisis_Numerik 25

n

i

ii

n

i

i

n

i

i

n

i

i

n

i

ii

n

i

i

n

i

i

n

i

i

n

i

i

n

i

i

n

i

i

yxxcxbxa

yxxcxbxa

yxcxbna

1

2

1

4

1

3

1

2

11

3

1

2

1

11

2

1

Example 3: Polynomial Regression

Fit a second-order polynomial to the following data

Analisis_Numerik 26

xi 0 1 2 3 4 5 ∑=15

yi 2.1 7.7 13.6 27.2 40.9 61.1 ∑=152.6

xi2 0 1 4 9 16 25 ∑=55

xi3 0 1 8 27 64 125 225

xi4 0 1 16 81 256 625 ∑=979

xi yi 0 7.7 27.2 81.6 163.6 305.5 ∑=585.6

xi2 yi 0 7.7 54.4 244.8 654.4 1527.5 ∑=2488.8

Example 3: Equations and Solution

Analisis_Numerik 27

2 8607.1 3593.24786.2)(

8607.1 ,3593.2 ,4786.2

. . . Solving

8.2488 979 225 55

6.585 225 55 15

6.152 55 15 6

xxxf

cba

cba

cba

cba

Analisis_Numerik 28

How Do You Judge Functions?

best? eselect thyou do How

data, fit the tofunctions moreor Given two

best. theis errors) theof

squares theof sum(least smaller in resulting

function The one. eachfor compute then

function, eachfor parameters theDetermine

:

Answer

Example showing that Quadratic is

preferable than Linear Regression

Analisis_Numerik 29

x

y

Quadratic Regression

x

y

Linear Regression

Analisis_Numerik 30

Fitting with Nonlinear Functions

n

i

ii

x

yxfcba

ecxbxaxf

1

2))((),,(

data. fit the to

)cos()ln()(

:form theof functiona find torequired isIt

xi0.24 0.65 0.95 1.24 1.73 2.01 2.23 2.52

yi0.23 -0.23 -1.1 -0.45 0.27 0.1 -0.29 0.24

Analisis_Numerik 31

Fitting with Nonlinear Functions

EquationsNormal

c

cba

b

cba

a

cba

yecxbxacban

i

ix

iii

0),,(

0),,(

0),,(

:minimum for the condition Necessary

) )cos( )ln((),,(1

2

Analisis_Numerik 32

Normal Equations

equations. normal thesolve and sums theEvaluate

)()())((cos)()(ln

)(cos)()(cos)(cos)(cos)(ln

)(ln)()(ln)(cos)(ln)(ln

11

2

11

111

2

1

1111

2

iiii

i

i

xn

i

i

n

i

xn

i

xi

xn

i

i

n

i

iix

n

i

i

n

i

ii

n

i

i

n

i

iix

n

i

ii

n

i

i

n

i

i

eyecexbexa

xyexcxbxxa

xyexcxxbxa

Analisis_Numerik 33

Example 4: Evaluating Sums

xi 0.24 0.65 0.95 1.24 1.73 2.01 2.23 2.52 ∑=11.57

yi 0.23 -0.23 -1.1 -0.45 0.27 0.1 -0.29 0.24 ∑=-1.23

(ln xi)2 2.036 0.1856 0.0026 0.0463 0.3004 0.4874 0.6432 0.8543 ∑=4.556

ln(xi) cos(xi) -1.386 -0.3429 -0.0298 0.0699 -0.0869 -0.2969 -0.4912 -0.7514 ∑=-3.316

ln(xi) * exi -1.814 -0.8252 -0.1326 0.7433 3.0918 5.2104 7.4585 11.487 ∑=25.219

yi * ln(xi) -0.328 0.0991 0.0564 -0.0968 0.1480 0.0698 -0.2326 0.2218 ∑=-0.0625

cos(xi)2 0.943 0.6337 0.3384 0.1055 0.0251 0.1808 0.3751 0.6609 ∑=3.26307

cos(xi) * exi 1.235 1.5249 1.5041 1.1224 -0.8942 -3.1735 -5.696 -10.104 ∑=-14.481

yi*cos(xi) 0.223 -0.1831 -0.6399 -0.1462 -0.0428 -0.0425 0.1776 -0.1951 ∑=-0.8485

(exi)2 1.616 3.6693 6.6859 11.941 31.817 55.701 86.488 154.47 ∑=352.39

yi * exi 0.2924 -0.4406 -2.844 -1.555 1.523 0.7463 -2.697 2.9829 ∑=-1.9923

Analisis_Numerik 34

Example 4: Equations & Solution

xexxxf

, c , b a

cba

cba

cba

012398.0)cos( 1074.1)ln( 88815.0)(

Therefore,

012398.01074.188815.0

:equations above theSolving

992283.1 388.352 4815.14 2192.25

848514.0 4815.14 26307.3 31547.3

062486.0 2192.25 31547.3 55643.4

Analisis_Numerik 35

data. thefitsbest that functiona Find bxaef(x)

02

02

:using obtained are s EquationNormal

),(

1

1

1

2

ii

ii

i

bxi

n

i

ibx

bxn

i

ibx

n

i

ibx

exayaeb

eyaea

yaeba

Example 5

Given:xi 1 2 3

yi 2.4 5 9

Difficult to Solve

Analisis_Numerik 36

xbaxfxg

aef(x) bx

)ln())(ln()( Define

data. thefitsbest that functiona Find

solve) Easier to(),( :Minimize

),( :minimizing of Instead

)ln()ln(Let

)ln()ln( Define

1

2

1

2

n

i

ii

n

i

ibx

ii

iii

zbxb

yaeba

yzanda

bxayz

i

Linearization Method

Analisis_Numerik 37

n

i

ii

n

i

i

n

i

i

n

i

i

n

i

i

i

n

i

ii

n

i

ii

n

i

ii

zxxbxzxbn

xzxbb

zxb

zxbb

11

2

111

1

1

1

2

) ( and

0 2

0 2

:using obtained are s EquationNormal

),(

Example 5: Equations

Analisis_Numerik 38

66087.0 ,23897.0

: EquationsSolving

686.10 14 6

68213.4 6 3

:Equations

b

b

b

Evaluating Sums and Solving

xi 1 2 3 ∑=6

yi 2.4 5 9

zi=ln(yi) 0.875469 1.609438 2.197225 ∑=4.68213

xi2 1 4 9 ∑=14

xi zi 0.875469 3.218876 6.591674 ∑=10.6860

xbx eaexf

ea

eaa

66087.0

23897.0

26994.1)(

26994.1

),ln(