small sample aic lecture, slides only

8/14/2019 Small sample AIC lecture, slides only

http://slidepdf.com/reader/full/small-sample-aic-lecture-slides-only 1/69

Information-theoretic analysis of -omics dataAn introduction

David R. Bickel

University of Ottawa

17 November 2008

David Bickel (uOttawa) Information theory 17 N ovem ber 2008 1 / 11

http://find/

http://goback/



Today’s class

Di¤erential gene/protein/metabolite expression


http://find/

http://goback/



Today’s class


Which genes express di¤erently between treatment and control?


http://find/

http://goback/



Today’s class


Which genes express di¤erently between treatment and control?Examples of "treatments"


http://find/

http://goback/



Today’s class



Medical: drug or chemotherapy applied to some patients


http://find/

http://goback/



Today’s class



Medical: drug or chemotherapy applied to some patientsBasic: hormone or other chemical added to some cell cultures


http://find/

http://goback/



Today’s class



Medical: drug or chemotherapy applied to some patientsBasic: hormone or other chemical added to some cell culturesOther examples?


http://find/

http://goback/



Today’s class




How much information or evidence is in the measurements


http://find/

http://goback/



Today’s class





for di¤erential expression?


http://find/

http://goback/



Today’s class





for di¤erential expression?for equivalent expression?


http://find/

http://goback/



Pick the di¤erentially expressed genes

What is di¤erential gene/protein/metabolite expression?


http://find/

http://goback/





An average expression ratio of 1 indicates equivalent expression


http://find/

http://goback/





An average expression ratio of 1 indicates equivalent expressionTwo types of di¤erential expression


http://find/

http://goback/






An average expression ratio less than 1 indicates under-expression


http://find/

http://goback/






An average expression ratio less than 1 indicates under-expressionAn average expression ratio greater than 1 indicates over-expression


http://find/

http://goback/







"Average expression" is over the population, not just the observed data


Pi k h di¤ i ll d

http://find/

http://goback/








The histogram of a large expression data set resembles the truedistribution


Pi k h di¤ i ll d

http://find/

http://goback/









Gene expression ratios measured by microarrays


Pi k h di¤ i ll d

http://find/

http://goback/










A sample from the treatment group and a sample from the control

group are hybridized to the same microarray slide


Pi k th di¤ ti ll d

http://find/

http://goback/











group are hybridized to the same microarray slideEach gene’s expression ratio is a measurement of its expression in thetreatment group relative to its expression in the control group


Pi k th di¤ ti ll d

http://find/

http://goback/











group are hybridized to the same microarray slideEach gene’s expression ratio is a measurement of its expression in thetreatment group relative to its expression in the control group

Based on the expression data, which genes are di¤erentially expressed?


http://find/

http://goback/



data set #1 data set #2 data set #4 data set #6

data (n = 3)data (n = 6)model (n = 3)model (n = 6)evidence (n = 3)

evidence (n = 6)For each data set, indicate whether the gene is equivalently expressed (E)or di¤erentially expressed (D) according to the plot of the data, accordingto the model , and according to the evidence for each number of observations (3 or 6). Equivalent expression means the average expression

ratio is 1.


Statistical models

http://find/

http://goback/



Statistical models

p stands for the number of unknown parameters in a model


Statistical models

http://find/

http://goback/



Statistical models


Equivalent expression model


Statistical models

http://find/

http://goback/



Statistical models



Unknown variability of expression


Statistical models

http://find/

http://goback/



Statistical models



Unknown variability of expressionExpression ratio known to be 1


Statistical models

http://find/

http://goback/



Statistical models




One unknown parameter (p = 1)


Statistical models

http://find/

http://goback/



Statistical models





Di¤erential expression model


Statistical models

http://find/

http://goback/



Statistical models






Unknown variability of expression


Statistical models

http://find/

http://goback/



Statistical models






Unknown variability of expressionUnknown expression ratio


Statistical models

http://find/

http://goback/



Statistical models




One unknown parameter (p = 1)Di¤erential expression model


Two unknown parameters (p = 2)


Statistical models

http://find/

http://goback/






One unknown parameter (p = 1)Di¤erential expression model


Two unknown parameters (p = 2)

How do the model plots change your initial assessments?


Balancing complexity and …t

http://find/

http://goback/



g p y

The di¤erential expression model (p = 2) is more complex than theequivalent expression model (p = 1)



http://find/

http://goback/



g p y


More complex models tend to …t data better than simple models,even if the simple models are better



http://find/

http://goback/



g p y



Overly complex models make poor generalizations



http://find/

http://goback/





Overly complex models make poor generalizationsA sample of patients may not represent the population



http://find/

http://goback/





Overly complex models make poor generalizationsA sample of patients may not represent the populationA single experiment may not re‡ect typical biological processes



http://find/

http://goback/






Fit

Complexity

= Evidence



http://find/

http://goback/






Fit

Complexity

= Evidence

How does balancing …t with complexity change your assessments?


Quality of model …t to the data

http://find/

http://goback/



n = sample size



http://find/

http://goback/



n = sample size

number of measured expression ratios



http://find/

http://goback/



n = sample size


MSE = mean of squared errors of the model



http://find/

http://goback/



n = sample size



degree to which the model disagrees with the observed data (log scale)



http://find/

http://goback/



n = sample size




Fit =

1p MSE

n



http://find/

http://goback/



n = sample size




Fit =

1p MSE

n

degree to which the model …ts the observed data (assuming a normal

distribution)


Model complexity

http://find/

http://goback/



n = sample size


Model complexity

http://find/

http://goback/



n = sample size



Model complexity

http://find/

http://goback/



n = sample size


p = model dimension


Model complexity

http://find/

http://goback/



n = sample size


p = model dimension

number of unknown parameters in the model


Model complexity

http://find/

http://goback/



n = sample size


p = model dimension


p = 1 for the equivalent expression model


Model complexity

http://find/

http://goback/



n = sample size


p = model dimension



p = 2 for the di¤erential expression model


Model complexity

http://find/

http://goback/



n = sample size


p = model dimension




p c = p +p (p + 1)

2 (n p + 1)


Model complexity

http://find/

http://goback/



n = sample size


p = model dimension




p c = p +p (p + 1)

2 (n p + 1)

e¤ective number of parameters in the model (corrected for small n)


Model complexity

http://find/

http://goback/



n = sample size


p = model dimension




p c = p +p (p + 1)

2 (n p + 1)


Complexity = 2.718p c


Model complexity

http://find/

http://goback/



n = sample size


p = model dimension




p c = p +p (p + 1)

2 (n p + 1)


Complexity = 2.718p c

Fit

Complexity= Evidence


http://find/

http://goback/



Answers



How do our analyses compare to the truth?

If a statistical method says an equivalently expressed gene isdi¤erentially expressed, is the method useless?


Answers

http://find/

http://goback/




If a statistical method says an equivalently expressed gene isdi¤erentially expressed, is the method useless?If a statistical method says a di¤erentially expressed gene is

equivalently expressed, is the method useless?


Answers

http://find/

http://goback/





equivalently expressed, is the method useless?The advantage of obtaining more data


Answers

http://find/

http://goback/






The best possible assessment given the available data


Answers

http://find/

http://goback/







How con…dent should you be in your assessments?


Answers

http://find/

http://goback/







How con…dent should you be in your assessments?

Should you obtain more data before making an assessment?


The expression data sets

data set #1 data set #2 data set #4 data set #6

http://find/

http://goback/



# # # #

ratio 1 2 1 1.4

expression equivalent di¤erential equivalent di¤erential

n = 10 0.

44/1.

38 0.

14/0.

09 0.

14/0.

17 0.

19/0.

37

n = 25 0.29/0.71 0.03/0.002 4.77/1.00 0.05/0.04

n = 100 36/691 104

2 10716/32 0.03/0.01

Key

n is the number of observed expression ratios.

Each ratio is Evidence di¤erentially expressedEvidence equivalently expressed , the weight of evidence

favoring di¤erential expression over equivalent expression.

* misleading evidence for di¤erential expression

** misleading evidence for equivalent expressionDavid Bickel (uOttawa) Information theory 17 November 2 00 8 10 / 1 1

Further study

http://find/

http://goback/



The method presented is based on the Akaike information criterion(AIC) after correcting it for small numbers of measurements

David Bickel (uOttawa) Information theory 17 N ove mb er 2008 11 / 11

Further study

http://find/

http://goback/




AICc = 2 ln (Evidence)


Further study

http://find/

http://goback/




AICc = 2 ln (Evidence)Software packages with the AIC but without the correction may be

unreliable for small numbers of observations (n < 40)


Further study

http://find/

http://goback/





unreliable for small numbers of observations (n < 40)Kenneth Burnham and David Anderson, Model Selection and

Multi-Model Inference


Further study

http://find/

http://goback/







These slides and …gures will be on the lab website


Further study

http://find/

http://goback/







These slides and …gures will be on the lab website

www.statomics.com


http://find/

http://goback/

small sample aic lecture, slides only

Documents