introduction to the generalized linear model: logistic ... › courses › 02418 › week9 ›...

Introduction to the Generalized Linear Model:Logistic regression and Poisson regression

Statistical modelling: Theory and practice

Gilles Guillot

[email protected]

November 4, 2013

Gilles Guillot ([email protected]) Intro GLM November 4, 2013 1 / 28

1 Logistic regression

2 Poisson regression

3 References


Logistic regression

US space shuttle Challenger accident data

●

●

●●● ● ●●

● ● ●

●●

●

● ●● ●● ●● ●

●

15 20 25

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Temperature (°C)

Pro

port

ion

of fa

ilure

s

Proportion of O-rings experiencing thermal distress on the US space shuttle Challenger.

Each point corresponds to a �ight. Proportion obtained among 6 O-rings for each �ight.


Logistic regression

US space shuttle Challenger accident data (cont')

●

●

●●● ● ●●

● ● ●

●●

●

● ●● ●● ●● ●

●

15 20 25

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Temperature (°C)

Pro

port

ion

of fa

ilure

s

Key question: what is the probability to experience a failure if thetemperature is close to 0?

For the previous example, we would need a model along the line ofNumber of failures ∼ temperatureorProportion of failures ∼ temperature


Logistic regression

Targeted online advertising

The problem

A person makes a search on Google on say 'cheap �ight Yucatan'

Is it worthwhile to display an ad for say 'hotel del Mar Cancún ?


Logistic regression

Targeted online advertising (cont')

Google knows the preferences of this person from before

Google knows on which ad this person as clicked or not clicked in thepastIn stat. words: Google has vector of binary explanatory variables(x1, ..., xp) that summarizes the prospect's preferences

Google knows the preferences of other persons from before who havebeen shown the ad for 'hotel del Mar Cancún and the same ads as theprospect.Denoting yi (∈ {0, 1}) the pastGoogle knows yi and (xi1, ..., xip) for these persons


Logistic regression

Targeted online advertising (cont')

In statistics word:

We have a dataset yi, xi1, ..., xip with i = 1, ..., nWe have a prospect known through its vector (x1, ..., xp)We want to predict the behavior y (0 or 1) of this person regarding thead 'hotel del Mar Cancún

We need a model along the line of y ∼ x1, ..., xpThe novelty (compared to the linear model) is that y is a count

What follows applies

to any type (quantitative or categorical) of explanatory variablesto any number (p = 1, p > 1) of explanatory variables


Logistic regression

Hints about what is coming below

In a linear regression we write

yi = β0 + β1xi + εi and εi ∼ N (0, σ2)

which we can rephrased as

yi ∼ N (β0 + β1xi, σ2)

Idea of a logistic regression: twinkle the equation above to �t data that cannot be considered Normal.


Logistic regression

The logistic function

De�nition of the logistic function

The logistic function is a function l : R −→ R de�ned as

l(x) =exp(x)

1 + exp(x)=

1

1 + exp(−x)

−4 −2 0 2 4

0.0

0.2

0.4

0.6

0.8

1.0

Logistic function

x

l(x)

This function l is a bijection (one-to-one), with inverseg(x) = l−1(x) = ln(y/(1− y))


Logistic regression

Logistic regression for a 0/1 response variable

We consider here the case of a single explanatory variable x and abinary response y.

The data consist of

y = (y1, ..., yn) observations of a binary (0/1) or two-way categoricalresponse variablex = (x1, ...., xn) observations of a quantitative explanatory variable


Logistic regression

Logistic regression for a 0/1 response variable (cont')

We want a model such that P (Yi = 1) varies with xi

We assume that Yi ∼ b(pi)We assume that pi = l(β0 + β1xi) (l logistic function)

This is an instance of a so-called Generalized Linear Model


Logistic regression

Formal de�nition of the logistic regression for a binary response

pi = l(β0 + β1xi) for a vector of unknown deterministic parametersβ = (β0, β1)

Yi ∼ b(pi) = b(l(β0 + β1xi))


Logistic regression

Note the analogy with the linear regression yi ∼ N (β0 + β1xi, σ2)

Under this model: E[Yi] = pi = 1/(1 + e−(β0+β1x1))

V [Yi] = pi(1− pi). The variance of Yi around its mean is entirelycontrolled by pi, hence by β0 and β1.

The model enjoys the structure: "Response = signal + noise" but wenever write it explicitly nor do we make any assumption or even referto a residual ε. The structure "Response = signal + noise" is implicit.

Straightforward extension for more than one explanatory variables:

pi = l(∑p

j=1 x(j)i )


Logistic regression

Logistic regression for �nite count data

We consider data y = (y1, ..., yn) obtained as counts over several replicatesdenoted N = (N1, ..., Nn). [For example, each Ni is equal to 6 in the0-rings data.]

pi = l(β0 + β1xi) for a vector of unknown deterministic parametersβ = (β0, β1)

Yi ∼ Binom(Ni, pi)

Under this model:E[Yi] = Nipi = Ni/(1 + e−(β0+β1x1))V [Yi] = Nipi(1− pi)


Logistic regression

Maximum likelihood inference for logistic regression

Data: y = (y1, ...., yn)x = (x1, ..., xn) and N = (N1, ..., Nn)Parameters: β = (β0, β1). NB: there is no parameter controlling thedispersion of residualsLikelihood:

L(y1, ...., yn;β0, β1) =

n∏i=1

(Ni

yi

)pyii (1− pi)

Ni−yi

∝n∏i=1

pyii (1− pi)Ni−yi (1)

lnL(y1, ...., yn;β0, β1) =

n∑i=1

yi ln pi + (Ni − pi) ln(1− pi) (2)


Logistic regression

ML inference cont'

Maximizing the log-likelihood

In constrast with the linear model, the maximum log-likelihood of thelogistic regression does not have a closed-from solution.It has to be maximised numerically.


Logistic regression

Model �tting in R

Binary response

With x, y two vectors of equal lengths, y being binary:

glm(formula = (y~x),

family=binomial(logit))

Returns parameters estimates and loads of diagnostic tools (e.g. AIC).


Logistic regression

Example of output of glm function

Call:

glm(formula = (y ~ x), family = binomial(link = logit))

Deviance Residuals:

Min 1Q Median 3Q Max

-2.1886 -1.0344 0.5489 0.9069 2.1294

Coefficients:

Estimate Std. Error z value Pr(>|z|)

(Intercept) 0.54820 0.07364 7.444 9.75e-14 ***

x 1.08959 0.08822 12.351 < 2e-16 ***

---

Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 1342.7 on 999 degrees of freedom

Residual deviance: 1138.1 on 998 degrees of freedom

AIC: 1142.1

Number of Fisher Scoring iterations: 4


Logistic regression

General binomial reponse

With x, N, y three vectors of equal lengths

glm(formula = cbind(y,N-y)~x,

family=binomial(logit))


Poisson regression

Poisson regression what for?

In many statistical studies, one tries to relate a count to someenvironmental or sociological variables.

For example:

Number of cardio-vascular accidents among people over 60 in a USstate ∼ Average income in the state?Number of bicycles in a Danish household ∼ Distance to the city centre

When the response is a count which does not have any natural upperbound, the logistic regression is not appropriate.

Even if there is a natural upper bound, the Binomial disitrbution maygive a poor �t

The Poisson regression is a natural alternative to the logisticregression in these cases.


Poisson regression

Poisson process, Poisson equation, Poisson kernel, Poisson distribution, Poisson

bracket, Poisson algebra, Poisson regression, Poisson summation formula, Poisson's

spot, Poisson's ratio, Poisson zeros....

Simeon Poisson 1781-1840


Poisson regression

The Poisson distribution

De�nition: Poisson distribution

The Poisson distribution is a distribution on N with probability mass

function: P (X = k) =e−λλk

k!for k ∈ N, and some parameter

λ ∈ (0,+∞)

Notation: X ∼ P(λ)


Poisson regression

The Poisson distribution (cont)

If X ∼ P(λ)E[X] = λ

V ar[X] = λ

If X ∼ P(λ) and Y ∼ P(µ), X,Y independent

X + Y ∼ P(λ+ µ)


Poisson regression

Set up for the Poisson regression

y1, ..., yn a discrete, positive variable, e.g. number of bicycles in nhouseholds

x1, ..., xn a quantitative variable, e.g. distance of household to citycentre or household income


Poisson regression

Poisson regression, de�nition

A Poisson regression is a model relating an explanatory variable x to apositive count variable y with the following assumptions:

De�nition: Poisson regression

y1, ..., yn are independant realizations of Poisson random variablesY1, ..., Yn with E[Yi] = µi

lnµi = αxi + β

For short Yi ∼ P(exp(αxi + β))

αxi + β is called the linear predictor

the function that relates the linear predictor to the mean of Yi iscalled the link function


Poisson regression

Poisson distribution �tting in R

glm.res = glm(formula = (nb.cycles ~ dist),

family = poisson(link="log"),

data = dat)


Poisson regression

Output of glm() function

summary(glm.res)

Call:

glm(formula = (nb.cycles ~ dist), family = poisson(link = "log"),

data = dat)

Deviance Residuals:

Min 1Q Median 3Q Max

-1.9971 -0.7196 -0.2033 0.4084 3.3813

Coefficients:

Estimate Std. Error z value Pr(>|z|)

(Intercept) -1.09426 0.24599 -4.448 8.65e-06 ***

dist 0.63211 0.06557 9.640 < 2e-16 ***

---

Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1

(Dispersion parameter for poisson family taken to be 1)

Null deviance: 198.77 on 99 degrees of freedom

Residual deviance: 101.55 on 98 degrees of freedom

AIC: 356.91

Number of Fisher Scoring iterations: 5Gilles Guillot ([email protected]) Intro GLM November 4, 2013 27 / 28

References

Reading

A Modern Approach to Regression with R, Chapter 8 Logistic Regression,Sheather, Springer 2009.


introduction to the generalized linear model: logistic ... › courses › 02418 › week9 ›...

Documents