Chapter 12

Multiple Regression

Multiple Regression

• Numeric Response variable (y)• k Numeric predictor variables (k < n)• Model:

Y = 0 + 1x1 + + kxk +

• Partial Regression Coefficients: i effect (on the mean response) of increasing the ith predictor variable by 1 unit, holding all other predictors constant

• Model Assumptions (Involving Error terms )– Normally distributed with mean 0

– Constant Variance 2

– Independent (Problematic when data are series in time/space)

Example - Effect of Birth weight on Body Size in Early Adolescence

• Response: Height at Early adolescence (n =250 cases)

• Predictors (k=6 explanatory variables)

• Adolescent Age (x1, in years -- 11-14)

• Tanner stage (x2, units not given)

• Gender (x3=1 if male, 0 if female)

• Gestational age (x4, in weeks at birth)

• Birth length (x5, units not given)

• Birthweight Group (x6=1,...,6 <1500g (1), 1500-1999g(2), 2000-2499g(3), 2500-2999g(4), 3000-3499g(5), >3500g(6))Source: Falkner, et al (2004)

Least Squares Estimation

• Population Model for mean response:

kk xxYE 110)(

• Least Squares Fitted (predicted) equation, minimizing SSE:

2^^

11

^

0

^^

YYSSExxY kk

• All statistical software packages/spreadsheets can compute least squares estimates and their standard errors

Analysis of Variance

• Direct extension to ANOVA based on simple linear regression

• Only adjustments are to degrees of freedom:– DFR = k DFE = n-(k+1)

Source ofVariation

Sum ofSquares

Degrees ofFreedom

MeanSquare F

Model SSR k MSR = SSR/k F = MSR/MSEError SSE n-(k+1) MSE = SSE/(n-(k+1))Total TSS n-1

TSS

SSR

TSS

SSETSSR

2

Testing for the Overall Model - F-test

• Tests whether any of the explanatory variables are associated with the response

• H0: 1==k=0 (None of the xs associated with y)

• HA: Not all i = 0

)(:

:..

))1(/()1(

/:..

)1(,,

2

2

obs

knkobs

obs

FFPvalP

FFRR

knR

kR

MSE

MSRFST

Example - Effect of Birth weight on Body Size in Early Adolescence

• Authors did not print ANOVA, but did provide following:

• n=250 k=6 R2=0.26• H0: 1==6=0 HA: Not all i = 0

)2.14(:

13.2:..

2.140030.

0433.

))16(250/()26.01(

6/26.0

))1(/()1(

/:..

243,6,

2

2

FPvalP

FFRR

knR

kR

MSE

MSRFST

obs

obs

Testing Individual Partial Coefficients - t-tests

• Wish to determine whether the response is associated with a single explanatory variable, after controlling for the others

• H0: i = 0 HA: i 0 (2-sided alternative)

|)|(2:

||:..

:..

)1(,2/

^

^

obs

knobs

b

iobs

ttPvalP

ttRR

stST

i

Example - Effect of Birth weight on Body Size in Early Adolescence

Variable b SEb t=b/SEb P-val (z)

Adolescent Age 2.86 0.99 2.89 .0038

Tanner Stage 3.41 0.89 3.83 <.001

Male 0.08 1.26 0.06 .9522

Gestational Age -0.11 0.21 -0.52 .6030

Birth Length 0.44 0.19 2.32 .0204

Birth Wt Grp -0.78 0.64 -1.22 .2224

Controlling for all other predictors, adolescent age, Tanner stage, and Birth length are associated with adolescent height measurement

Comparing Regression Models• Conflicting Goals: Explaining variation in Y while keeping

model as simple as possible (parsimony)• We can test whether a subset of k-g predictors (including

possibly cross-product terms) can be dropped from a model that contains the remaining g predictors. H0: g+1=…=k =0

– Complete Model: Contains all k predictors

– Reduced Model: Eliminates the predictors from H0

– Fit both models, obtaining sums of squares for each (or R2 from each):

• Complete: SSRc , SSEc (Rc2)

• Reduced: SSRr , SSEr (Rr2)

Comparing Regression Models

• H0: g+1=…=k = 0 (After removing the effects of X1,…,Xg, none of other predictors are associated with Y)

• Ha: H0 is false

)(

:

)]1([1

)(

)]1(/[

)/()( :TS

))1((,,

2

22

obs

kngkobs

c

rc

c

rcobs

FFPP

FFRR

knR

gkRR

knSSE

gkSSRSSRF

P-value based on F-distribution with k-g and n-(k+1) d.f.

Models with Dummy Variables

• Some models have both numeric and categorical explanatory variables (Recall gender in example)

• If a categorical variable has m levels, need to create m-1 dummy variables that take on the values 1 if the level of interest is present, 0 otherwise.

• The baseline level of the categorical variable is the one for which all m-1 dummy variables are set to 0

• The regression coefficient corresponding to a dummy variable is the difference between the mean for that level and the mean for baseline group, controlling for all numeric predictors

Example - Deep Cervical Infections• Subjects - Patients with deep neck infections

• Response (Y) - Length of Stay in hospital

• Predictors: (One numeric, 11 Dichotomous)– Age (x1)

– Gender (x2=1 if female, 0 if male)

– Fever (x3=1 if Body Temp > 38C, 0 if not)

– Neck swelling (x4=1 if Present, 0 if absent)

– Neck Pain (x5=1 if Present, 0 if absent)

– Trismus (x6=1 if Present, 0 if absent)

– Underlying Disease (x7=1 if Present, 0 if absent)

– Respiration Difficulty (x8=1 if Present, 0 if absent)

– Complication (x9=1 if Present, 0 if absent)

– WBC > 15000/mm3 (x10=1 if Present, 0 if absent)

– CRP > 100g/ml (x11=1 if Present, 0 if absent)

Source: Wang, et al (2003)

Example - Weather and Spinal Patients• Subjects - Visitors to National Spinal Network in 23 cities

Completing SF-36 Form• Response - Physical Function subscale (1 of 10 reported)• Predictors:

– Patient’s age (x1)

– Gender (x2=1 if female, 0 if male)

– High temperature on day of visit (x3)

– Low temperature on day of visit (x4)

– Dew point (x5)

– Wet bulb (x6)

– Total precipitation (x7)

– Barometric Pressure (x7)

– Length of sunlight (x8)

– Moon Phase (new, wax crescent, 1st Qtr, wax gibbous, full moon, wan gibbous, last Qtr, wan crescent, presumably had 8-1=7 dummy variables)

Source: Glaser, et al (2004)

Modeling Interactions• Statistical Interaction: When the effect of one

predictor (on the response) depends on the level of other predictors.

• Can be modeled (and thus tested) with cross-product terms (case of 2 predictors):– E(Y) = 1X1 + 2X2 + 3X1X2

– X2=0 E(Y) = 1X1

– X2=10 E(Y) = 1X1 + 102 + 103X1

= (+ 102) + (1 + 103)X1

• The effect of increasing X1 by 1 on E(Y) depends on level of X2, unless 3=0 (t-test)

Logistic Regression

• Logistic Regression - Binary Response variable and numeric and/or categorical explanatory variable(s)– Goal: Model the probability of a particular

outcome as a function of the predictor variable(s)

– Problem: Probabilities are bounded between 0 and 1

Logistic Regression with 1 Predictor

• Response - Presence/Absence of characteristic

• Predictor - Numeric variable observed for each case

• Model - (x) Probability of presence at predictor level x

x

x

e

ex

10

10

1)(

• = 0 P(Presence) is the same at each level of x

• > 0 P(Presence) increases as x increases

• < 0 P(Presence) decreases as x increases

Logistic Regression with 1 Predictor

01 are unknown parameters and must be estimated using statistical software such as SPSS, SAS, or STATA

· Primary interest in estimating and testing hypotheses regarding 1

· Large-Sample test (Wald Test): (Some software runs z-test)

· H0: 1= 0 HA: 1 0

)(:value

:..

:..

22

21,

2

2

1

^

2

1

^

obs

obs

obs

XPP

XRR

sXST

Example - Rizatriptan for Migraine

• Response - Complete Pain Relief at 2 hours (Yes/No)• Predictor - Dose (mg): Placebo (0),2.5,5,10

Dose # Patients # Relieved % Relieved0 67 2 3.0

2.5 75 7 9.35 130 29 22.310 145 40 27.6

Source: Gijsmant, et al (1997)

Example - Rizatriptan for Migraine (SPSS)

Variables in the Equation

.165 .037 19.819 1 .000 1.180

-2.490 .285 76.456 1 .000 .083

DOSE

Constant

Step1

a

B S.E. Wald df Sig. Exp(B)

Variable(s) entered on step 1: DOSE.a.

x

x

e

ex

165.0490.2

165.0490.2^

1)(

000.:

84.3:

819.19037.0

165.0:..

0:0:

21,05.

2

22

110

valP

XRR

XST

HH

obs

obs

A

Odds Ratio

• Interpretation of Regression Coefficient (1):– In linear regression, the slope coefficient is the change

in the mean response as x increases by 1 unit

– In logistic regression, we can show that:

)(1

)()(

)(

)1(1

x

xxoddse

xodds

xodds

• Thus e1represents the change in the odds of the outcome

(multiplicatively) by increasing x by 1 unit

• If 1=0, the odds (and probability) are equal at all x levels (e1=1)

• If 1>0 , the odds (and probability) increase as x increases (e1>1)

• If 1< 0 , the odds (and probability) decrease as x increases (e1<1)

95% Confidence Interval for Odds Ratio

• Step 1: Construct a 95% CI for :

1

^

1

^

1

^ 96.1,96.196.1 1

^

1

^

1

^

sss

• Step 2: Raise e = 2.718 to the lower and upper bounds of the CI:

1

^1

^

1^1

^96.196.1

, ss

ee

• If entire interval is above 1, conclude positive association

• If entire interval is below 1, conclude negative association

• If interval contains 1, cannot conclude there is an association

Example - Rizatriptan for Migraine

)2375.0,0925.0()037.0(96.1165.0:%95

037.0165.01

^1

^

CI

s

• 95% CI for 1:

• 95% CI for population odds ratio:

)27.1,10.1(, 2375.00925.0 ee

• Conclude positive association between dose and probability of complete relief

Multiple Logistic Regression

• Extension to more than one predictor variable (either numeric or dummy variables).

• With p predictors, the model is written:

kk

kk

xx

xx

e

e

110

110

1

• Adjusted Odds ratio for raising xi by 1 unit, holding all other predictors constant:

ieORi

• Inferences on i and ORi are conducted as was described above for the case with a single predictor

Example - ED in Older Dutch Men

• Response: Presence/Absence of ED (n=1688)

• Predictors: (k=12)– Age stratum (50-54*, 55-59, 60-64, 65-69, 70-78)– Smoking status (Nonsmoker*, Smoker)– BMI stratum (<25*, 25-30, >30)– Lower urinary tract symptoms (None*, Mild,

Moderate, Severe)– Under treatment for cardiac symptoms (No*, Yes)– Under treatment for COPD (No*, Yes)

* Baseline group for dummy variablesSource: Blanker, et al (2001)

Example - ED in Older Dutch MenPredictor b SEb Adjusted OR (95% CI)Age 55-59 (vs 50-54) 0.83 0.42 2.3 (1.0 – 5.2)Age 60-64 (vs 50-54) 1.53 0.40 4.6 (2.1 – 10.1)Age 65-69 (vs 50-54) 2.19 0.40 8.9 (4.1 – 19.5)Age 70-78 (vs 50-54) 2.66 0.41 14.3 (6.4 – 32.1)Smoker (vs nonsmoker) 0.47 0.19 1.6 (1.1 – 2.3)BMI 25-30 (vs <25) 0.41 0.21 1.5 (1.0 – 2.3)BMI >30 (vs <25) 1.10 0.29 3.0 (1.7 – 5.4)LUTS Mild (vs None) 0.59 0.41 1.8 (0.8 – 4.3)LUTS Moderate (vs None) 1.22 0.45 3.4 (1.4 – 8.4)LUTS Severe (vs None) 2.01 0.56 7.5 (2.5 – 22.5)Cardiac symptoms (Yes vs No) 0.92 0.26 2.5 (1.5 – 4.3)COPD (Yes vs No) 0.64 0.28 1.9 (1.1 – 3.6)

•Interpretations: Risk of ED appears to be:

• Increasing with age, BMI, and LUTS strata

• Higher among smokers

• Higher among men being treated for cardiac or COPD

Testing Regression Coefficients• Testing the overall model:

)(

..

))log(2())log(2(..

0 allNot :

0:

22

2,

2

102

10

obs

kobs

obs

iA

k

XPP

XRR

LLXST

H

H

• L0, L1 are values of the maximized likelihood function, computed by statistical software packages. This logic can also be used to compare full and reduced models based on subsets of predictors. Testing for individual terms is done as in model with a single predictor.