lecture 3: regression analysis & model fitting · lecture 3: regression analysis & model...
TRANSCRIPT
Lecture3:
Regressionanalysis&modelfitting
ShaneElipotTheRosenstielSchoolofMarineandAtmosphericScience,
UniversityofMiami
Createdwith{Remark.js}using{Markdown}+{MathJax}
Loading[MathJax]/jax/output/HTML-CSS/jax.js
References
[1]Bendat,J.S.,&Piersol,A.G.(2011).Randomdata:analysisandmeasurementprocedures(Vol.729).JohnWiley&Sons.
[2]Thomson,R.E.,&Emery,W.J.(2014).Dataanalysismethodsinphysicaloceanography.Newnes.dx.doi.org/10.1016/B978-0-12-387782-6.00003-X
[3]Press,W.H.etal.(2007).Numericalrecipes3rdedition:Theartofscientificcomputing.Cambridgeuniversitypress.
[4]vonStorch,H.andZwiers,F.W.(1999).StatisticalAnalysisinClimateResearch,CambridgeUniversityPress
[5]Rawlings,J.,Pantula,S.G.,andDickey,D.A.(1998)AppliedRegressionAnalysis,Aresearchtool,secondedition,Springer
[6]Wunsch,C.(2006).Discreteinverseandstateestimationproblems:withgeophysicalfluidapplications.CambridgeUniversityPress.
[7]Fan,J.andGijbels,I.(1996),LocalPolynomialModellingandItsApplications,CRCPress
Lecture3:Outline1. Introduction2. Linearregression3. Polynomialinterpolation4. LocalPolynomialmodeling5. Anoteonnonlinearmodeling
1.Introduction
Introduction
Regressionanalysisconsistsinusingmathematicalexpressions(thatismodeling,ormodellingintheU.K.)todescribetosomeextentthebehaviorofarandomvariable(r.v.)ofinterest.Thisvariableiscalledadependentvariable.Thevariablesthatarethoughttoprovideinformationaboutthedependentvariableandareincorporatedinthemodelarecalledindependentvariables.Themodelsusedinregressionanalysistypicallyinvolveunknownconstants,calledparameters,whicharetobeestimatedfromthedata.
Introduction
Regressionanalysisconsistsinusingmathematicalexpressions(thatismodeling,ormodellingintheU.K.)todescribetosomeextentthebehaviorofarandomvariable(r.v.)ofinterest.Thisvariableiscalledadependentvariable.Thevariablesthatarethoughttoprovideinformationaboutthedependentvariableandareincorporatedinthemodelarecalledindependentvariables.Themodelsusedinregressionanalysistypicallyinvolveunknownconstants,calledparameters,whicharetobeestimatedfromthedata.
Themathematicalcomplexityofthemodel,andthedegreetowhichitisrealistic,dependonhowmuchisknownabouttheprocessandthepurposeoftheregressionanalysis(andtheabilityandknowledgeofthescientist).
Mostregressionmodelsthatwewillencounterarelinearintheirparameters.Iftheyarenotlinear,theycanoftenbelinearized.
Criticalthinkingshouldbeemployed,asanymodelcanbefittedto(orregressedagainst)anydata.
Example
DailyatmosphericCO measuredatMaunaLoainHawaiiatanaltitudeof3400m.DatafromDr.PieterTans,NOAA/ESRL(www.esrl.noaa.gov/gmd/ccgg/trends/)andDr.RalphKeeling,ScrippsInstitutionofOceanography(scrippsco2.ucsd.edu).
2
Example
Determiningthelineartrendofthistimeseriesisanexampleoflinearregression.Furthermodelingcouldincludeestimatingtheseasonalcycleofthetimeseriesetc.
2.Linearregression
Simplelinearregression
Wearegoingtoreviewthesimplestlinearmodelinvolvingoneindependentvariable andonedependentvariable .Inparallelwewillalsopresenttheequationsforamoregeneralmodelrelatingto dependentvariables .Matlabusesnotationthatressemblethematrixformulasforthegeneral(multivariate)linearmodel.
x yy
p , ,… ,x1 x2 xp
Simplelinearregression
Wearegoingtoreviewthesimplestlinearmodelinvolvingoneindependentvariable andonedependentvariable .Inparallelwewillalsopresenttheequationsforamoregeneralmodelrelatingto dependentvariables .Matlabusesnotationthatressemblethematrixformulasforthegeneral(multivariate)linearmodel.
Asanexample,wewillseethattheleastsquaressolutionofthelinearmodelis
whichinMatlabcanbewritten
B = (X'*X)^-1*X'*Y;
butisbettercodedas
B = (X'*X)\X'*Y;
x yy
p , ,… ,x1 x2 xp
= ( X Yβ XT )−1XT
Simplelinearregression
Previously,whenexaminingasetofobservations of ,weassumedthattheexpectation,ortruemean,wasconstant,i.e.
.
Yi y
E[ ] =Yi μy
Simplelinearregression
Previously,whenexaminingasetofobservations of ,weassumedthattheexpectation,ortruemean,wasconstant,i.e.
.
Weknowconsiderthecasewhenthemeanisafunctionofanothervariable,asanexampletime.Linearregressionscanconsistinestimatingthetrendandtheseasonalcycleofyourtimeseries.
Yi y
E[ ] =Yi μy
InthecaseoftheCO record,themeanisclearlynotaconstant,increasingeveryyear,butalsooscillatingwithineachyear.
Simplelinearregression
Previously,whenexaminingasetofobservations of ,weassumedthattheexpectation,ortruemean,wasconstant,i.e.
.
Weknowconsiderthecasewhenthemeanisafunctionofanothervariable,asanexampletime.Linearregressionscanconsistinestimatingthetrendandtheseasonalcycleofyourtimeseries.
Yi y
E[ ] =Yi μy
2
Thismodelisapplicableasanexampleforestimatingalineartrendofatimeseries,oranylinearrelationshipbetweentwor.vs.
Simplelinearregression
Thesimplestmodelisthatthetruemeanorexpectationofchangesataconstantrateasthevalueof decreasesorincreases:
where and aretheparameterstoestimate.
yx
E[ ] = + , i = 1,…,nYi β0 β1Xi
β0 β1
Simplelinearregression
Theobservationsofthedependentvariable arelookedatasindividualrealizationsofther.vs. withpopulationsmeans .Thedeviationof from istakenintoaccountbyincorporatingarandomerror inthelinearmodel
yYi E[ ]Yi
Yi E[ ]Yi
ϵi
= + +Yi β0 β1Xi ϵi
The areassumednormallyindependentidenticallydistributed(i.i.d.)r.vs.as .Sinceand areconstant, .
Incontrast,theobservedvaluesof aresupposedtobefreeoferrors,treatedasconstants.
Simplelinearregression
Theobservationsofthedependentvariable arelookedatasindividualrealizationsofther.vs. withpopulationsmeans .Thedeviationof from istakenintoaccountbyincorporatingarandomerror inthelinearmodel
yYi E[ ]Yi
Yi E[ ]Yi
ϵi
= + +Yi β0 β1Xi ϵi
ϵi
∼ N (0,σ) ,β0 β1Xi ∼ N (E[ ],σ)Yi Yi
X
Generallinearmodel(ormultipleregression)
Thegenerallinearmodelwith independentvariablesforobservation is
Thereare parameterstoestimate:aconstant andfactors .Inmatrixnotation,for observations,weobtainthelinearsystem
pi
= + + +⋯+ +Yi β0 β1Xi1 β2Xi2 βpXip ϵi
= p+ 1p′ ( )β0 p,… ,β1 βp n
Y = Xβ+ ϵ
⎡
⎣⎢⎢⎢⎢Y1
Y2
⋮Yn
⎤
⎦⎥⎥⎥⎥
(n× 1)
=
⎡
⎣⎢⎢⎢⎢⎢11
⋮1
X11
X21
⋮Xn1
X12
X22
⋮Xn2
⋯⋯
⋯
X1p
X2p
⋮Xnp
⎤
⎦⎥⎥⎥⎥⎥
(n× )p′
+
⎡
⎣⎢⎢⎢⎢β0
β1
⋮βp
⎤
⎦⎥⎥⎥⎥
( × 1)p′
⎡
⎣⎢⎢⎢⎢ϵ1
ϵ2
⋮ϵn
⎤
⎦⎥⎥⎥⎥
(n× 1)
Generallinearmodel
Eachelement isapartialregressioncoefficientthatquantifiesthechangeinthedependentvariable perunitchangeintheindependentvariable ,assumingallotherindependentvariablesareheldconstant.
β =
⎡
⎣⎢⎢⎢⎢β0
β1
⋮βp
⎤
⎦⎥⎥⎥⎥
βj
Yi
Xij
Simplelinearmodel
Ifthe were andthemodelwereabsolutelytrue,anytwopairsofobservations wouldbeenoughtosolveforthetwounkownparameters and .
= + +Yi β0 β1Xi ϵi
ϵi 0( , )Xi Yi
β0 β1
Simplelinearmodel
Ifthe were andthemodelwereabsolutelytrue,anytwopairsofobservations wouldbeenoughtosolveforthetwounkownparameters and .
Yet,becauseoferrors,anothermethodisused,calledleastsquaresestimation,whichgivesasolution,orestimate thatleadstothesmallestpossiblesumofsquareddeviationsoftheobservations fromtheestimates oftheirtruemeans .
= + +Yi β0 β1Xi ϵi
ϵi 0( , )Xi Yi
β0 β1
( , )β0 β1
Yi E[ ]Yi E[ ]Yi
Simplelinearmodel:LSsolution
Let providetheestimateofthetruemean
suchthatthesumofsquaresofdeviationsfromthemean
isminimized.
iscalledthe -thobservedresidual.
( , )β0 β1
= + ≡E[ ]Yi β0 β1Xi Yi
SS(Res) = ( − =∑i=1
n
Yi Yi )2 e2i
ei i
Generallinearmodel
Forthegeneralmodel,inmatrixnotation,
andtheresidualsarefoundinthe vector
andthesumofsquaresofresidualsis
whichisaminimumbecauseof .
Howtofind ?
≡ XY β
(n× 1)
e = Y− = Y−XY β
SS(Res) = e = (Y− (Y− ) = (Y−X (Y−X )eT Y)T Y β)T β
β
β
Simplelinearmodel
Themethodtofindthevalues thatminimize isclassic.Youtakethederivativesof withrespecttoeachofthe parametersandequatetheresultstozero.Youobtainasystemof equationswith unkowns.Forthesimplelinearmodelyouobtainthenormalequations
whichsolutionis
( , )β0 β1 SS(Res)SS(Res)
p+ 1p+ 1 p+ 1
nβ0
β0 ∑i
Xi
+
+
β1 ∑i
Xi
β1 ∑i
X2i
=
=
∑i
Yi
∑i
XiYi
β1
β0
=
=
= =( − )( − )∑i Xi X
¯ ¯¯Yi Y
¯ ¯¯
( −∑i Xi X¯ ¯¯ )2
sxy
sxx
sxy
s2x
−Y¯ ¯¯
β1X¯ ¯¯
Simplelinearmodel
Thepredictedvaluesfromthesolutionofthelinearmodelare
canbeinterpretedasbeingboththeestimateofthepopulationmeanof foragivenvalueof ,andthepredictedvaluevalueofforafuturevalueof whichis .
= + =Yi β0 β1Xi E[ ]Yi
Yi
y x yx Xi
Generallinearmodel
Forthemultipleregressionmodel,thenormalequationsareobtained
andtheleastsquares(LS)solutionis
Thepredictedvaluesof are
with calledtheprojectionmatrix.Thislastexpressionsshowsthattheestimated arelinearfunctionofalltheobservedvalues .
= 0 → X = Y∂SS(Res)
∂βXT β XT
= ( X Yβ XT )−1XT
y
Y ==
X = X( X Yβ XT )−1XT
PY
P = X( XXT )−1XT
Yi
Yi
Simplelinearmodel
Theobservationsof cannowbewrittenasthesumoftheestimatedpopulationmeanforagivenvalueof andaresidual
Thesumofthesquaresoftheobservationsare
sinceitcanbeshownthat .
Thesumofthesquaresoftheobservationsisthesumofthesquares"accountedfor"bythemodelplusthesumofthesquaresof"unaccountedfor".
yx
= +Yi Yi ei
∑i
Y 2i =
=
( +∑i
Yi ei)2
+∑i
Yi
2 ∑i
e2i
2 = 0∑i Yi
2ei
Simplelinearmodel
Using ,thedecompositionofthesumofthesquarescanbeusedasfollows
= (1/n)Y¯ ¯¯ ∑i Yi
− n∑i
Y 2i Y
¯ ¯¯ 2
( −∑i
Yi Y¯ ¯¯ )2
=
=
− n∑i
Yi
2Y¯ ¯¯ 2
( −β12∑
i
Xi X¯ ¯¯ )2
+
+
∑i
e2i
∑i
e2i
Simplelinearmodel
Using ,thedecompositionofthesumofthesquarescanbeusedasfollows
Whatdoesthissay?[Uptoafactor ]
= (1/n)Y¯ ¯¯ ∑i Yi
− n∑i
Y 2i Y
¯ ¯¯ 2
( −∑i
Yi Y¯ ¯¯ )2
=
=
− n∑i
Yi
2Y¯ ¯¯ 2
( −β12∑
i
Xi X¯ ¯¯ )2
+
+
∑i
e2i
∑i
e2i
1/(n− 1)
Simplelinearmodel
Using ,thedecompositionofthesumofthesquarescanbeusedasfollows
Whatdoesthissay?[Uptoafactor ]
Itapproximatelysaysthat:
"Thetotalvariancefromobservations"="variancefromtheregression"+"varianceoftheresiduals"
Inthemodel ,theregressionpartis .
iscalledtheregressioncoefficient.
= (1/n)Y¯ ¯¯ ∑i Yi
− n∑i
Y 2i Y
¯ ¯¯ 2
( −∑i
Yi Y¯ ¯¯ )2
=
=
− n∑i
Yi
2Y¯ ¯¯ 2
( −β12∑
i
Xi X¯ ¯¯ )2
+
+
∑i
e2i
∑i
e2i
1/(n− 1)
= +Yi β1Xi β0 β1Xi
β1
Coefficientofdetermination
Fromthelinearmodel,weareinterestedinaquantitycalledthecoefficientofdetermination
Forthesimple(univariate)linearmodel,
isthusthesquareofthePearson'scorrelationcoefficientbetween and .
= = 1 −R2( − −∑i Yi Y
¯ ¯¯ )2 ∑i e2i
( −∑i Yi Y¯ ¯¯ )2
∑i e2i
( −∑i Yi Y¯ ¯¯ )2
R2 =
=
=( −β1
2∑i Xi X
¯ ¯¯ )2
( −∑i Yi Y¯ ¯¯ )2
( )sxy
s2x
2s2x
s2y
= =s2xy
s2xs2y
⎛⎝⎜
sxy
s2xs2y
− −−−√⎞⎠⎟2
r2xy
R2
x y
Simplelinearmodel
Atraditionalinterpretationof isthatitisameasureofthefractionofvarianceofthedependentvariable explainedbytheindependentvariable .
Thisiswhythe(squareofthe)Pearsoncorrelationcoefficientisveryquicklyinterpretedasbeingameasureoftheamountofvarianceexplainedbetweentwovariables.
Asanexampleif youwilloftenreadsometinglike"xisabletoexplain49%ofthevarianceofy".(Since )
R2
R2
yx
= 0.7rxy= 0.490.72
Simplelinearmodel:uncertainties
Inthemodel whereweassumedthat,wedidnotknowthevariance .= + +Yi β0 β1Xi ϵi
∼ N (0,σ)ϵi σ2
Simplelinearmodel:uncertainties
Inthemodel whereweassumedthat,wedidnotknowthevariance .Anunbiased
estimateof isgivenbytheresidualmeansquare:
This"mean"valueisobtainedbydividingthe bythenumberofdegreesoffreedomfortheresidualswhichisthenumberofdatapoints minusthenumberofparametersofthemodel
.
= + +Yi β0 β1Xi ϵi∼ N (0,σ)ϵi σ2
σ2
= ≡σ2
s2∑i e
2i
n− (p+ 1)
SS(Res)
(n)(p+ 1)
Simplelinearmodel:uncertainties
AnumberofformulasforthevarianceoftheestimatescanbederivedandusedforcalculatingCIs:
Var[ ]β1
Var[ ]β0
Var[ ]Yi
Var[ ]Y0
=
=
=
=
s2
( −∑i Xi X¯ ¯¯ )2
[ + ]1n
X¯ ¯¯ 2
( −∑i Xi X¯ ¯¯ )2
s2
[ + ]1n
( −Xi X¯ ¯¯ )2
( −∑i Xi X¯ ¯¯ )2
s2
[1 + + ]1n
( −X0 X¯ ¯¯ )2
( −∑i Xi X¯ ¯¯ )2
s2
Simplelinearmodel:uncertainties
AnumberofformulasforthevarianceoftheestimatescanbederivedandusedforcalculatingCIs:
Var[ ]β1
Var[ ]β0
Var[ ]Yi
Var[ ]Y0
=
=
=
=
s2
( −∑i Xi X¯ ¯¯ )2
[ + ]1n
X¯ ¯¯ 2
( −∑i Xi X¯ ¯¯ )2
s2
[ + ]1n
( −Xi X¯ ¯¯ )2
( −∑i Xi X¯ ¯¯ )2
s2
[1 + + ]1n
( −X0 X¯ ¯¯ )2
( −∑i Xi X¯ ¯¯ )2
s2
: , = , , ∼ t(0,n− p−H0 β 1,0 Yi m1,0,Yi
−β1 m1
Var[ ]β1
− −−−−−√−β0 m0
Var[ ]β0
− −−−−−√−Yi mYi
Var[ ]Yi
− −−−−−√
Generallinearmodel:uncertainties
Forthegenerallinearmodel,theformulasare
P
Var[ ]β
Var[ ]Y
Var[ ]Y0
Var[e]=σ
2s2
=
=
=
=
=
=
X( XXT )−1XT
( XXT )−1σ2
Pσ2
[I+ ( X ]X0 XT )−1X0T σ2
(I−P)σ2
e/(n− p− 1)eT
Generallinearmodel:uncertainties
Forthegenerallinearmodel,theformulasare
Bewarethattheexpressionabovesarematrices.Asanexampleforthesimplelinearmodelforwhich :
thisimpliesthattheparameterestimatescovary.
P
Var[ ]β
Var[ ]Y
Var[ ]Y0
Var[e]=σ
2s2
=
=
=
=
=
=
X( XXT )−1XT
( XXT )−1σ2
Pσ2
[I+ ( X ]X0 XT )−1X0T σ2
(I−P)σ2
e/(n− p− 1)eT
p+ 1 = 2
Var[ ] = [ ]βVar( )β0
Cov( , )β1 β0
Cov( , )β0 β1
Var( )β1
NotehowtheCIsfortheprediction forfuturevalues
of arelargerthantheCIsforthepredictionofthemeanof
.Thevarianceofthepredictionisthevarianceofestimatingthemeanplusthevarianceofthequantityestimated.
TheCIsarethesmallestfor.
Inthiscase,IgenerateddataandIhadprescribed
Simplelinearmodel:uncertainties
Y0X0 x
Yi
=X0 X¯ ¯¯
= 0, = 0.8, = 0.04β0 β1 σ2
Linearmodelbyleastsquares
Themethodofleastsquarestofindasolutiontothegenerallinearmodelisappropriatewhenfourassumptionsarevalid:(1)therandomerrors arenormallydistributed,(2)independent,(3)withzeromeanandconstantvariance ,and(4)the areobservationsofthe independentvariablesmeasuredwithouterrors.
= + + +⋯+ +Yi β0 β1Xi1 β2Xi2 βpXip ϵi
ϵiσ2 Xij
p
Linearmodelbyleastsquares
Themethodofleastsquarestofindasolutiontothegenerallinearmodelisappropriatewhenfourassumptionsarevalid:(1)therandomerrors arenormallydistributed,(2)independent,(3)withzeromeanandconstantvariance ,and(4)the areobservationsofthe independentvariablesmeasuredwithouterrors.
Ifwerelyonalargenumber ofdata,thenormalassumptionmaybeinvokedbecauseoftheCLT.Otherwise,MaximumLikelihoodmethodscanbeused.AsanexampleseeElipotetal.(2016).
= + + +⋯+ +Yi β0 β1Xi1 β2Xi2 βpXip ϵi
ϵiσ2 Xij
p
n
Linearmodelbyleastsquares
Themethodofleastsquarestofindasolutiontothegenerallinearmodelisappropriatewhenfourassumptionsarevalid:(1)therandomerrors arenormallydistributed,(2)independent,(3)withzeromeanandconstantvariance ,and(4)the areobservationsofthe independentvariablesmeasuredwithouterrors.
Ifwerelyonalargenumber ofdata,thenormalassumptionmaybeinvokedbecauseoftheCLT.Otherwise,MaximumLikelihoodmethodscanbeused.AsanexampleseeElipotetal.(2016).
Whenthedependentvariableobservationsarenormallydistributedbutdonothavethesamevariances,orerrors,themethodofweightedleastsquarescanbeimplemented.
= + + +⋯+ +Yi β0 β1Xi1 β2Xi2 βpXip ϵi
ϵiσ2 Xij
p
n
Linearmodelbyleastsquares
Themethodofleastsquarestofindasolutiontothegenerallinearmodelisappropriatewhenfourassumptionsarevalid:(1)therandomerrors arenormallydistributed,(2)independent,(3)withzeromeanandconstantvariance ,and(4)the areobservationsofthe independentvariablesmeasuredwithouterrors.
Ifwerelyonalargenumber ofdata,thenormalassumptionmaybeinvokedbecauseoftheCLT.Otherwise,MaximumLikelihoodmethodscanbeused.AsanexampleseeElipotetal.(2016).
Whenthedependentvariableobservationsarenormallydistributedbutdonothavethesamevariances,orerrors,themethodofweightedleastsquarescanbeimplemented.
Whentheindependentvariablesareactuallynotindependent(becausetheyaremaybecorrelated),themethodofgeneralleastsquarescanbeimplemented.Seereferences[5]and[6].
= + + +⋯+ +Yi β0 β1Xi1 β2Xi2 βpXip ϵi
ϵiσ2 Xij
p
n
Linearmodelbyweightedleastsquares
Let'sassumethatthevarianceofeach (andthusofeach )iswhere isaconstant.Asanexample,someobservationsmay
havebetteraccuracythanothers.
ϵi Yi
a2i σ2 σ
Linearmodelbyweightedleastsquares
Wecanconsiderthefollowingrescaledmodel,dividingby :
or
Becausethevarianceofthe is ,thevarianceofthebecomes .Wecannowuseleastsquarestoregress onthe
.
ai
= + + +⋯+ +Yi
ai
1ai
β0 β1Xi1
aiβ2
Xi2
aiβp
Xip
ai
ϵiai
= + + +⋯+ +Y ∗i X∗
i0β0 β1X∗i1 β2X
∗i2 βpX
∗ip ϵ∗i
ϵi a2i σ2 ϵ∗i
σ2 Y ∗i
X∗ij
Linearmodelbyweightedleastsquares
Theprinciplehereistoassigntheleastamountofweighttotheobservationswiththelargestvariance,orerror.Theweightingmatrixis
Considerthegenerallinearmodelequationleft-multipliedby
whichcanberewrittenas
with ,etc.
W =
⎡
⎣⎢⎢⎢⎢⎢1/a10
⋮0
01/a2
⋮0
⋯0
⋱⋯
0
⋮1/an
⎤
⎦⎥⎥⎥⎥⎥
W
WY = WXβ+Wϵ
= β+Y∗ X∗ ϵ∗
= WYY∗
Linearmodelbyweightedleastsquares
Theweightedleastsquaresolutionis
with
β
Var[ ]β
Var[ ]YVar[e]
=
=
==
( X YX′V−1 )−1X′V−1
( XX′V−1 )−1σ2
X( XX′V−1 )−1X′σ2
[V−X( X ]X′V−1 )−1X′ σ2
= W =V−1 W′
⎡
⎣⎢⎢⎢⎢⎢a21
0
⋮0
0
a22
⋮0
⋯0
⋱⋯
0
⋮a2n
⎤
⎦⎥⎥⎥⎥⎥
Linearmodelbyweightedleastsquares
Theweightedleastsquaresolutionis
with
Infact,theweightingmatrixcanhavewhatevercoefficientyouwant!Hereitisaspecialcasethatsimplifiestheformofthesolution.Seesection2.4ofreference[6].
β
Var[ ]β
Var[ ]YVar[e]
=
=
==
( X YX′V−1 )−1X′V−1
( XX′V−1 )−1σ2
X( XX′V−1 )−1X′σ2
[V−X( X ]X′V−1 )−1X′ σ2
= W =V−1 W′
⎡
⎣⎢⎢⎢⎢⎢a21
0
⋮0
0
a22
⋮0
⋯0
⋱⋯
0
⋮a2n
⎤
⎦⎥⎥⎥⎥⎥
3.Polynomialinterpolation
Polynomialfitting
Fittingapolynomialfunctionofanindependentvariable toadependentvariable isalinearregressionproblemwhichconsistsinestimatingthecoefficientsofthepolynomial
xy
y = + x+ +…+ +…β0 β1 β2x2 βkx
k
Polynomialfitting
Fittingapolynomialfunctionofanindependentvariable toadependentvariable isalinearregressionproblemwhichconsistsinestimatingthecoefficientsofthepolynomial
Wewillreviewtwogeneralcases.Thefirstcaseisglobalpolynomialfittingwhereyouarefittingapolynomialfunctionthatexactlyestimateyourdata,maybepiecewise,inseparateintervals.Thispolynomialisofmaximumorderofthenumberofobservationsminusoneandiscalledaninterpolatingpolynomial.
xy
y = + x+ +…+ +…β0 β1 β2x2 βkx
k
Polynomialfitting
Fittingapolynomialfunctionofanindependentvariable toadependentvariable isalinearregressionproblemwhichconsistsinestimatingthecoefficientsofthepolynomial
Wewillreviewtwogeneralcases.Thefirstcaseisglobalpolynomialfittingwhereyouarefittingapolynomialfunctionthatexactlyestimateyourdata,maybepiecewise,inseparateintervals.Thispolynomialisofmaximumorderofthenumberofobservationsminusoneandiscalledaninterpolatingpolynomial.
Thesecondcaseiscalledlocalpolynomialestimationwhenyouarefittingapolynomialinthevicinity,thatiswithinawindow,ofagivenvalueof .Thispolynomialofarbitraryorderapproximateyourdatalocallyandthesolutionistypicallyobtainedbyweightedleastsquares.
xy
y = + x+ +…+ +…β0 β1 β2x2 βkx
k
x
Polynomialfitting
Fittingapolynomialfunctionofanindependentvariable toadependentvariable isalinearregressionproblemwhichconsistsinestimatingthecoefficientsofthepolynomial
Wewillreviewtwogeneralcases.Thefirstcaseisglobalpolynomialfittingwhereyouarefittingapolynomialfunctionthatexactlyestimateyourdata,maybepiecewise,inseparateintervals.Thispolynomialisofmaximumorderofthenumberofobservationsminusoneandiscalledaninterpolatingpolynomial.
Thesecondcaseiscalledlocalpolynomialestimationwhenyouarefittingapolynomialinthevicinity,thatiswithinawindow,ofagivenvalueof .Thispolynomialofarbitraryorderapproximateyourdatalocallyandthesolutionistypicallyobtainedbyweightedleastsquares.
Thesetwotypesofmethodscanbeusedingeneraltoprocessyourdatatoeitherinterpolateorgridyourdata.
xy
y = + x+ +…+ +…β0 β1 β2x2 βkx
k
x
Interpolatingpolynomial
Assumeyouhave pairsofobservations andwouldliketointerpolate forgivenvalueof .
Thereexistsaninterpolatingpolynomialoforder givenbythefollowingLagrangeformula
whichpassesthroughyourdatapoints,i.e.
N ( , )Xi Yi
y x
N − 1
(x) =PN−1 ∑k=1
N⎛⎝⎜⎜ ∏
j=1j≠k
N x−Xj
−Xk Xj
⎞⎠⎟⎟ Yk
( ) =PN−1 Xi Yi
Interpolatingpolynomial
ExamplewithN = 5
Polynomialfitting
Alternativelyyoucanuseleastsquarestofitapolynomialofanyorderequaltoorlessthan withthemodelN − 1
= + + +…Yi β0 β1Xi β2X2i βN−1X
N−1i
Interpolatingpolynomial
Bewarethatinterpolatingpolynomialcanquicklygenerateverylargeoscillations!
Sameexampleasbeforeexceptthattheoriginaldatapointwasmovedto .
= 2Xi
2.9
Piecewiselinearinterpolation
Apiecewiselinearinterpolation,orsimplylinearinterpolationconsistincalculatingtheinterpolatingpolynomialoforder1overaninterval ,i.ewith points.TheLagrangeformulagives
whichcanberearrangedtogivethelinearinterpolant
InMatlabitisimplementedby
yi = interp1(x,y,xi);
[ , ]Xk Xk+1 N = 2
(x) = ( ) + ( )P1x−Xk+1
−Xk Xk+1Yk
x− Xk
−Xk+1 XkYk+1
(x) = + (x− )L1 Yk Xk−Yk+1 Yk
−Xk+1 Xk
Piecewiselinearinterpolation:errors
TheLagrangeformulagivesyouaneasywaytoestimatetheinterpolationerror.If and aretheerrorsoruncertaintiesfor
and ,sinceδk δk+1
Yk Yk+1
(x) = +P1 akYk ak+1Yk+1
δ (x) =P1 +a2kδ2k
a2k+1δ
2k+1
− −−−−−−−−−−−−√
Hermitepolynomialinterpolation
TheissueoflargeoscillationsininterpolatingcanbereinedinbyusingHermitepolynomialswhichsatisfyadditionalconditionsonitsderivativesatthedatapoints,
where istobespecified. isthe -thderivativeof .
( ) = , ( ) =Pn Xk Yk P(1)n Xk dk
dk P(ν)n ν Pn
Hermitepolynomialinterpolation
TheissueoflargeoscillationsininterpolatingcanbereinedinbyusingHermitepolynomialswhichsatisfyadditionalconditionsonitsderivativesatthedatapoints,
where istobespecified. isthe -thderivativeof .
ApopularHermitepolynomialistheshape-preservingpiecewisecubicHermiteinterpolatingpolynomialorshape-preservingpchip,implementedinMatlabby
yi = pchip(x,y,xi);
( ) = , ( ) =Pn Xk Yk P(1)n Xk dk
dk P(ν)n ν Pn
pchipexample
Apchippolynomialiscubic(order3)anditsderivatives ,orslopes,ateachdatapointarezeroortheharmonicmeansofconsecutiveslopes:
dk
= + with =1dk
1δk
1δk−1
δk−Yk+1 Yk
−Xk+1 Xk
Splineinterpolation
Anotherpopularinterpolationmethodusescubicsplineswhicharepiecewisecubicinterpolatingpolynomialswithconstraintsonthesecondderivativetobeacontinuous.ItisimplementedinMatlabas
yi = spline(x,y,xi);
Cubicinterpolation
Anothermethodusingpiecewisepolynomialsoforder3iscalledcubicconvolutionandisdescribedindetailinKeys1981.ThismethodisaccessibleinoneorhigherdimensionsinMatlabas
yi = interpn(x,y,xi,'cubic');
Somecomments
Interpolatingpolynomialsandsplinesaregreatsetsoftoolthatallowyoutoquicklyinterpolateyourdata.Splinesarenotnecessarilypolynomialoforder3andcanbeofgreaterorder.Thereexistsaverylargebodyoflitteraturedealingwithsplines.
Somecomments
Interpolatingpolynomialsandsplinesaregreatsetsoftoolthatallowyoutoquicklyinterpolateyourdata.Splinesarenotnecessarilypolynomialoforder3andcanbeofgreaterorder.Thereexistsaverylargebodyoflitteraturedealingwithsplines.
Wehavedealtsofarwithmethodsofinterpolationinonedimensionbutthesecanbeeasilyexpandedintwoormoredimensions,notablythelinearandcubicmethods.
Somecomments
Interpolatingpolynomialsandsplinesaregreatsetsoftoolthatallowyoutoquicklyinterpolateyourdata.Splinesarenotnecessarilypolynomialoforder3andcanbeofgreaterorder.Thereexistsaverylargebodyoflitteraturedealingwithsplines.
Wehavedealtsofarwithmethodsofinterpolationinonedimensionbutthesecanbeeasilyexpandedintwoormoredimensions,notablythelinearandcubicmethods.
Polynomialinterpolationimpliesthatyouareexactlyrecoveringyourdata,i.e .Thisimpliesthatyourdataareeffectivelyerrorfree.Wenowrelaxthisconditionandreviewsomeprinciplesoflocalpolynomialmodeling.
P ( ) =Xi Yi
4.LocalPolynomialModeling
Polynomialbyleastsquares
Wesawearlierthatwecanuseleastsquarestofitapolynomialofanyorderequaltoorlessthanyour datapoints:N − 1
= + + +…Yi β0 β1Xi β2X2i βN−1X
N−1i
Localpolynomialfitting
Onceagainweattempttoestimatethevalueofadependentvariablegivenavalueoftheindependentvalue .Herewefollowclosely
reference[7].Theideaistoestimateanarbitraryfunctionanditsderivativenoted withthemodel
where areobservationsand isthevarianceof at.
y xm(x)
(x), (x),… , (x)m(1) m(1) m(p)
= m( ) + σ( )Yi Xi Xi
( , )Xi Yi σ( )Xi Yi
x = Xi
Localpolynomialfitting
Onceagainweattempttoestimatethevalueofadependentvariablegivenavalueoftheindependentvalue .Herewefollowclosely
reference[7].Theideaistoestimateanarbitraryfunctionanditsderivativenoted withthemodel
where areobservationsand isthevarianceof at.
Thefunction isapproximatedlocallybyapolynomialoforderbyconsideringaTaylorexpansionintheneighborhoodof as
y xm(x)
(x), (x),… , (x)m(1) m(1) m(p)
= m( ) + σ( )Yi Xi Xi
( , )Xi Yi σ( )Xi Yi
x = Xi
m(x)p x0
m(x) ≈ m( )x0
=
+ ( )(x− ) + (x− +…+ (x−m′ x0 x0( )m′′ x0
2!x0)2
m(p)
p!+ (x− ) + (x− +…+ (x−β0 β1 x0 β2 x0)2 βp x0)p
Localpolynomialfitting
Thefunction ismodeledlocallyas
and .Theestimatesof ofthispolynomialareobtainedforeachlocationofinterest byleastsquaresfitting,minimizingthefollowingexpression
where
iscalledakernelfunction,actingoverahalf-bandwidth .
m(x)
m(x) = (x− .∑j=0
p
βj x0)j
( ) = j!m(j) x0 βj βj
x0
{ − ( − ( − )∑i=1
n
Yi ∑j=0
p
βj Xi x0)j}2Kh Xi x0
(x) = K ( )Kh
1h
x
h
K h
Localpolynomialfitting
Inthisexample,theunknownfunctiongiving isestimatedat usinganorderonepolynomial,usingdatapointswithintheorangewindow.
E[ ] = m( )Yi Xi
X = x0
Localpolynomialfitting
Forfittingapolynomialtoyourdata,anumberofaspectsneedtobeconsidered,allcoveredinmanydetailsasanexampleinreference[7]:
1. Whichorderpolynomialdoyouneed?Areyoutryingtoestimatethevalueofyourunknownfunctiononly,orareyoutryingtoestimatethe -thderivativeaswell?Inthiscase,itisrecommendedthat beanoddnumber.
2. Whatbandwidth doyouneed?Itwilldependsonthedensityofyourdata,aswellastheorderofthechosenpolynomial.Thechoiceofthebandwidthisacompromisebetweenbiasandvarianceofyourestimate.Sinceyouaretryingtoestimateparametersbyleastsquaresyoushouldhaveatleastthatnumberofpointsinyourwindow.
3. Whatshapeshouldthekernelfunctionhave?Shoulditbeuniform?Gaussian?Quadratic?AquadratickernelcalledtheEpanechnikovkernelisoftenrecommended
(seepracticalthisafternoon!)
νp− ν
h
p+ 1
Localpolynomialfitting
Thisfigureshowsanexampleoffittingaknownfunctionembeddedinnoisewithaknownvariance.Itshowstheimpactofthebandwidthandpolynomialorderonthebiasandvarianceoftheestimates.
Localpolynomialfitting
Asimplersmootherconsistsinestimatingthefunction asapolynomialoforder .TheequivalentiscalledtheNadaraya-Watsonkernelestimatordefinedas
ThetypicalkernelfunctionsusedaretheGaussiankernel
andthesymmetricBetafamily
where isacomplicatedfunctionofnointeresthere.
m(x)0
(x) ≡mh
( − x)∑j=1
n
Kh Xj Yj
( − x)∑j=1
n
Kh Xj
K(z) = ( exp(− /2)2π−−
√ )−1 z2
K(z) = (1 − , γ = 0, 1,… ,1
B(1/2,γ+ 1)t2)γ+
B(z,w)
5.Anoteonnonlinearfitting
Nonlinearfitting
Whatdoesonedowhenthefunctionyouaretryingtofittoyourdataisnonlinearinyourparameter?
Nonlinearfitting
Whatdoesonedowhenthefunctionyouaretryingtofittoyourdataisnonlinearinyourparameter?Asanexample,youexpectthatasinusoidfunctionisagoodmodeltodescribethedependencyofyourdependentvariable ontheindependentvariable ,i.e.
where istheamplitudeand isthephase.
y x
y(x) = a cos(x+ ϕ)
a ϕ
Nonlinearfitting
Whatdoesonedowhenthefunctionyouaretryingtofittoyourdataisnonlinearinyourparameter?Asanexample,youexpectthatasinusoidfunctionisagoodmodeltodescribethedependencyofyourdependentvariable ontheindependentvariable ,i.e.
where istheamplitudeand isthephase.Inthiscaseyou'reinluckbecauseyoucanusetrigonometricidentitiesandwrite
y x
y(x) = a cos(x+ ϕ)
a ϕ
y(x) = a cos(ϕ) cos(x) − a sin(ϕ) sin(x)
Nonlinearfitting
Whatdoesonedowhenthefunctionyouaretryingtofittoyourdataisnonlinearinyourparameter?Asanexample,youexpectthatasinusoidfunctionisagoodmodeltodescribethedependencyofyourdependentvariable ontheindependentvariable ,i.e.
where istheamplitudeand isthephase.Inthiscaseyou'reinluckbecauseyoucanusetrigonometricidentitiesandwrite
Youhavelinearizedyourproblem,andyouarenowfacedwithamultiplelinearregressionproblem,estimating and
asafunctionofobservationsof and (seepracticalthisafternoon).
y x
y(x) = a cos(x+ ϕ)
a ϕ
y(x) = a cos(ϕ) cos(x) − a sin(ϕ) sin(x)
a cos(ϕ)a sin(ϕ) y, cos(x) sin(x)
Nonlinearfitting
Whatifyoureallycannotlinearizeyourproblem?
Nonlinearfitting
Whatifyoureallycannotlinearizeyourproblem?Asanexample,itisoftenusefultomodelthelaggedcorrelationfunction ,asinBealetal.2015
ρ(τ)
Nonlinearfitting
Inthisparticularcase,weassumedthatthelaggedcorrelationfunctionforthealong-shorecomponentofvelocity,asafunctionofseparationdistance(lag)wasgivenby
Thegoalisheretofitthedataforthevalueoftheparameter ,aspatiallengthscale.Weapplythesameprincipleofminimization,tryingtofindthevalue minimizing
(r) = cos( )ρh e−(r/rh)2 πr
2rh
rh
rh
SS(Res) = ( ( ) −∑i=1
N
ρh ri ρi)2
Nonlinearfitting
Inthisparticularcase,weassumedthatthelaggedcorrelationfunctionforthealong-shorecomponentofvelocity,asafunctionofseparationdistance(lag)wasgivenby
Thegoalisheretofitthedataforthevalueoftheparameter ,aspatiallengthscale.Weapplythesameprincipleofminimization,tryingtofindthevalue minimizing
Sincetheproblemcannotbeputinlinearform,theleastsquaremethodisnotavailable.Insteadyoumustrelyonnonlinearoptimizationroutines.Asanexample,Matlabcanapplycommonalgorithmsbythefunction .
(r) = cos( )ρh e−(r/rh)2 πr
2rh
rh
rh
SS(Res) = ( ( ) −∑i=1
N
ρh ri ρi)2
fminsearch
Practicalsession
Pleasedownloaddataatthefollowinglink:
PleasedownloadtheMatlabcodeatthefollowinglink:
MakesureyouhaveinstalledandtestedthefreejLabMatlabtoolboxfromJonathanLillyatwww.jmlilly.net/jmlsoft.html