variable selection in genomics - github...

Variable selection in genomics— methods, challenges, and possibilities

Trial talk in defense of the degree of Ph. D.Einar HolsbøFebruary 8th, 2019

Variable selection

Identifying a suitable subset of variables as relevant for your response and the modeling thereof

Variable selection


(identifying what is irrelevant and can be thrown away)

Variable selection


(identifying what is irrelevant and can be thrown away)

Maybe aka. “Data mining”

1886

2 variables, 100s of observations

Rothamsted experimental station


0

1

2

3

4

5

6

7

8

9

10

1840 1860 1880 1900 1920 1940 1960 1980 2000 2020

Grai

n th

a-1at

85%

dry

mat

ter

Cultivar

Hoosfield. Mean long-term spring barley grain yields 1852-2015

Best yield from FYM+N plots (max 144kgN)

FYM

Best yield from NPKMg plots (max 144kgN)

PKMg+48kgN

FYM 1852-1871

Unfertilized

Chevalier Archer's

Carter's

Archer's

Hallett's

Archer'sPlumage Archer

Julia

Georgie

Triumph

Alexis

Cooper

Optic

Tipple

Modern Cultivars

since 1970

Better Weed Control

Limingsince 1955

Fungicidessince 1978

© Rothamsted Research 2017 licensed under a Creative Commons Attribution 4.0 International License


0

1

2

3

4

5

6

7

8

9

10

1840 1860 1880 1900 1920 1940 1960 1980 2000 2020

Grai

n th

a-1at

85%

dry

mat

ter

Cultivar



FYM


PKMg+48kgN

FYM 1852-1871

Unfertilized

Chevalier Archer's

Carter's

Archer's

Hallett's


Julia

Georgie

Triumph

Alexis

Cooper

Optic

Tipple

Modern Cultivars

since 1970

Better Weed Control

Limingsince 1955



Response


0

1

2

3

4

5

6

7

8

9

10

1840 1860 1880 1900 1920 1940 1960 1980 2000 2020

Grai

n th

a-1at

85%

dry

mat

ter

Cultivar



FYM


PKMg+48kgN

FYM 1852-1871

Unfertilized

Chevalier Archer's

Carter's

Archer's

Hallett's


Julia

Georgie

Triumph

Alexis

Cooper

Optic

Tipple

Modern Cultivars

since 1970

Better Weed Control

Limingsince 1955



Variable


0

1

2

3

4

5

6

7

8

9

10

1840 1860 1880 1900 1920 1940 1960 1980 2000 2020

Grai

n th

a-1at

85%

dry

mat

ter

Cultivar



FYM


PKMg+48kgN

FYM 1852-1871

Unfertilized

Chevalier Archer's

Carter's

Archer's

Hallett's


Julia

Georgie

Triumph

Alexis

Cooper

Optic

Tipple

Modern Cultivars

since 1970

Better Weed Control

Limingsince 1955



6 variables, “enough” observations


• Experimental design • Small sample inference • Comparison of multiple contrasts • Hypothesis testing • &c.

Why do you have irrelevant variables?– Ronald Fisher, probably

~100 years later…

~100 years later…

Genome [n.] – the complete set of genes or genetic material present in a cell or organism.

~100 years later…

Genome [n.] – the complete set of genes or genetic material present in a cell or organism.

Genomics [n., pl.] – (treated as singular) the study of the structure, function, evolution, and mapping of genomes.

~100 years later…

-ome [suffix] – “all of them/it”-omics [suffix] – the study of all the different things

(my interpretation)

Central dogma of molecular biologyCrick, F. (1970). Central dogma of molecular biology. Nature, 227(5258):561.


AT GC CG TA TA CG

……

DNA


AT GC CG TA TA CG

U C G A A G

……

……

DNA mRNA

transcription


AT GC CG TA TA CG

U C G A A G

protein

……

……

DNA mRNA

transcription translation


U C G A A G

……

mRNA


U C G A A G

……

mRNA} Transcriptomics [n.]

subfield to genomics to do with gene transcripts and the function of the genome.

So why variable selection?


• There are about 20.000 protein-coding genes • We are able to measure them all simultaneously • Which ones are associated with disease?


• There are about 20.000 protein-coding genes • We are able to measure them all simultaneously • Which ones are associated with a given process?


• There are about 20.000 protein-coding genes • We are able to measure them all simultaneously • Which ones are associated with disease?


• There are about 20.000 protein-coding genes • We are able to measure them all simultaneously • Which ones are associated with disease?😭


The computer as both problem and solution

experiment_4 experiment_5 experiment_6 experiment_7 experiment_8 experiment_9






−10 −5 0 5 −10 −5 0 5 −10 −5 0 5 −10 −5 0 5 −10 −5 0 5 −10 −5 0 5

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

class one two

Genomics light: 36 measurements, 20 obs







−10 −5 0 5 −10 −5 0 5 −10 −5 0 5 −10 −5 0 5 −10 −5 0 5 −10 −5 0 5

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

class one two







−10 −5 0 5 −10 −5 0 5 −10 −5 0 5 −10 −5 0 5 −10 −5 0 5 −10 −5 0 5

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

class one two

Danger of modeling noise, or “overfitting”

Want to find true signal, discard noise

Message

(i) Old times: careful choice of variables; These times: measure everything (ii) In genomics there are many variables to choose from.

(iii) So little known that careful choice of variables is virtually impossible (iv) Be careful

❧ Leaving genomics behind, mostly

Linear model

y = �0 + �1x1 + . . .+ �dxd<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>

Linear model


response =

Pweights⇥ variables

<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>

Linear model


response =



outcome of interest measurements

Linear model


response =



find the �s/weights<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>

outcome of interest measurements

A typical taxonomy

• Filters

• Wrappers

• Embedded methods

Filters

rank variables select “best” ones put in model➠ ➠

Filters

rank variables select “best” ones put in model➠ ➠t.test(x_1, y) …

Filters

rank variables select “best” ones put in model➠ ➠t.test(x_1, y) … p < .1, top 10, etc.

Filters

rank variables select “best” ones put in model➠ ➠t.test(x_1, y) … p < .1, top 10, etc. maybe linear

Some transcriptome “filters”

• Significance analysis of microarrays (SAM) (also SAMSeq)

• Linear models for microarray/RNASeq data (LIMMA)

• K top-scoring pairs (K-tsp)

A typical taxonomy

• Filters

• Wrappers


Wrappers

candidate subset

evaluate fit

put in model➠

➠➠

Wrappers

candidate subset

evaluate fit

put in model➠

➠➠

stagewise,simulated annealing, …

Wrappers

candidate subset

evaluate fit

put in model➠

➠➠

stagewise,simulated annealing, … maybe linear

Wrappers

candidate subset

evaluate fit

put in model➠

➠➠

stagewise,simulated annealing, … maybe linear

metrics: R2, empirical risk, …

A typical taxonomy

• Filters

• Wrappers


Embedded

Combined model estimation and variable selection

Embedded

Combined model estimation and variable selectionoptimize model fit – model complexity

N << d


N << d


x 𝛃 y=

N << d


x 𝛃 y=

∞ solutions

N << d


Standard rule-of-thumb calculations suggest 10–20 observations per parameter:

200 000 may be too few!

N << d


Idea: constrain the solution 𝛃 to lie within a certain region

N << d


Idea: constrain the solution 𝛃 to lie within a certain region

Eg find 𝛃 s.t.X

�2i t

<latexit sha1_base64="MKxOuK7+4jD6KuDz6gDROXoIIys=">AAAB/XicbVA9SwNBEN2LXzF+nYqVzWIQrMIlCGoXtLGMYEwgF8PeZpIs2ds7d+eEcAT8KzYWKrb+Dzv/jZvkCk18MPB4b4aZeUEshUHP+3ZyS8srq2v59cLG5tb2jru7d2eiRHOo80hGuhkwA1IoqKNACc1YAwsDCY1geDXxG4+gjYjULY5iaIesr0RPcIZW6rgHvklC6geArCPuK9SX8ECx4xa9kjcFXSTljBRJhlrH/fK7EU9CUMglM6ZV9mJsp0yj4BLGBT8xEDM+ZH1oWapYCKadTs8f02OrdGkv0rYU0qn6eyJloTGjMLCdIcOBmfcm4n9eK8HeeTsVKk4QFJ8t6iWSYkQnWdCu0MBRjixhXAt7K+UDphlHm1jBhlCef3mR1Culi5J3c1qsXmZp5MkhOSInpEzOSJVckxqpE05S8kxeyZvz5Lw4787HrDXnZDP75A+czx+K5pTD</latexit><latexit sha1_base64="MKxOuK7+4jD6KuDz6gDROXoIIys=">AAAB/XicbVA9SwNBEN2LXzF+nYqVzWIQrMIlCGoXtLGMYEwgF8PeZpIs2ds7d+eEcAT8KzYWKrb+Dzv/jZvkCk18MPB4b4aZeUEshUHP+3ZyS8srq2v59cLG5tb2jru7d2eiRHOo80hGuhkwA1IoqKNACc1YAwsDCY1geDXxG4+gjYjULY5iaIesr0RPcIZW6rgHvklC6geArCPuK9SX8ECx4xa9kjcFXSTljBRJhlrH/fK7EU9CUMglM6ZV9mJsp0yj4BLGBT8xEDM+ZH1oWapYCKadTs8f02OrdGkv0rYU0qn6eyJloTGjMLCdIcOBmfcm4n9eK8HeeTsVKk4QFJ8t6iWSYkQnWdCu0MBRjixhXAt7K+UDphlHm1jBhlCef3mR1Culi5J3c1qsXmZp5MkhOSInpEzOSJVckxqpE05S8kxeyZvz5Lw4787HrDXnZDP75A+czx+K5pTD</latexit><latexit sha1_base64="MKxOuK7+4jD6KuDz6gDROXoIIys=">AAAB/XicbVA9SwNBEN2LXzF+nYqVzWIQrMIlCGoXtLGMYEwgF8PeZpIs2ds7d+eEcAT8KzYWKrb+Dzv/jZvkCk18MPB4b4aZeUEshUHP+3ZyS8srq2v59cLG5tb2jru7d2eiRHOo80hGuhkwA1IoqKNACc1YAwsDCY1geDXxG4+gjYjULY5iaIesr0RPcIZW6rgHvklC6geArCPuK9SX8ECx4xa9kjcFXSTljBRJhlrH/fK7EU9CUMglM6ZV9mJsp0yj4BLGBT8xEDM+ZH1oWapYCKadTs8f02OrdGkv0rYU0qn6eyJloTGjMLCdIcOBmfcm4n9eK8HeeTsVKk4QFJ8t6iWSYkQnWdCu0MBRjixhXAt7K+UDphlHm1jBhlCef3mR1Culi5J3c1qsXmZp5MkhOSInpEzOSJVckxqpE05S8kxeyZvz5Lw4787HrDXnZDP75A+czx+K5pTD</latexit>

X�2i t


Figures from Hastie, Tibshirani, and Friedman: The Elements of Statistical Learning

“ridge”


X�2i t



“ridge”


Tromsdalstinden

X�2i t



“ridge”


Tromsdalstinden

*

X|�i| t

<latexit sha1_base64="8NqjQ2vieh0SxUmPxqGqRkxR35Y=">AAAB/XicbVDLSgNBEJyNrxhfUfHkZTAInsKuCOot6MVjBGMC2RBmJ73JkJnddaZXCJuAv+LFg4pX/8Obf+PkcdDEgoaiqpvuriCRwqDrfju5peWV1bX8emFjc2t7p7i7d2/iVHOo8VjGuhEwA1JEUEOBEhqJBqYCCfWgfz3264+gjYijOxwk0FKsG4lQcIZWahcPfJMqOvQDQNYWQ+pLeKDYLpbcsjsBXSTejJTIDNV28cvvxDxVECGXzJim5ybYyphGwSWMCn5qIGG8z7rQtDRiCkwrm5w/osdW6dAw1rYipBP190TGlDEDFdhOxbBn5r2x+J/XTDG8aGUiSlKEiE8XhamkGNNxFrQjNHCUA0sY18LeSnmPacbRJlawIXjzLy+S2mn5suzenpUqV7M08uSQHJET4pFzUiE3pEpqhJOMPJNX8uY8OS/Ou/Mxbc05s5l98gfO5w8rx5Ur</latexit><latexit sha1_base64="8NqjQ2vieh0SxUmPxqGqRkxR35Y=">AAAB/XicbVDLSgNBEJyNrxhfUfHkZTAInsKuCOot6MVjBGMC2RBmJ73JkJnddaZXCJuAv+LFg4pX/8Obf+PkcdDEgoaiqpvuriCRwqDrfju5peWV1bX8emFjc2t7p7i7d2/iVHOo8VjGuhEwA1JEUEOBEhqJBqYCCfWgfz3264+gjYijOxwk0FKsG4lQcIZWahcPfJMqOvQDQNYWQ+pLeKDYLpbcsjsBXSTejJTIDNV28cvvxDxVECGXzJim5ybYyphGwSWMCn5qIGG8z7rQtDRiCkwrm5w/osdW6dAw1rYipBP190TGlDEDFdhOxbBn5r2x+J/XTDG8aGUiSlKEiE8XhamkGNNxFrQjNHCUA0sY18LeSnmPacbRJlawIXjzLy+S2mn5suzenpUqV7M08uSQHJET4pFzUiE3pEpqhJOMPJNX8uY8OS/Ou/Mxbc05s5l98gfO5w8rx5Ur</latexit><latexit sha1_base64="8NqjQ2vieh0SxUmPxqGqRkxR35Y=">AAAB/XicbVDLSgNBEJyNrxhfUfHkZTAInsKuCOot6MVjBGMC2RBmJ73JkJnddaZXCJuAv+LFg4pX/8Obf+PkcdDEgoaiqpvuriCRwqDrfju5peWV1bX8emFjc2t7p7i7d2/iVHOo8VjGuhEwA1JEUEOBEhqJBqYCCfWgfz3264+gjYijOxwk0FKsG4lQcIZWahcPfJMqOvQDQNYWQ+pLeKDYLpbcsjsBXSTejJTIDNV28cvvxDxVECGXzJim5ybYyphGwSWMCn5qIGG8z7rQtDRiCkwrm5w/osdW6dAw1rYipBP190TGlDEDFdhOxbBn5r2x+J/XTDG8aGUiSlKEiE8XhamkGNNxFrQjNHCUA0sY18LeSnmPacbRJlawIXjzLy+S2mn5suzenpUqV7M08uSQHJET4pFzUiE3pEpqhJOMPJNX8uY8OS/Ou/Mxbc05s5l98gfO5w8rx5Ur</latexit>

X�2i t



“lasso”“ridge”


X|�i| t


X�2i t



“lasso”“ridge”


t usually a data-dependent decision

X|�i| t


“optimize model fit – model complexity”

X|�i| t


measure of model complexity

“optimize model fit – model complexity”

X|�i| t


Figure from Christophe Giraud “Introduction to High-Dimensional Statistics”

X|�i| t


Figure from Christophe Giraud “Introduction to High-Dimensional Statistics”

End-result: a model with many coefficients = 0

Filters and wrappers considered harmful


Coefficients biased away from 0 ➠ “overfitting”


Collinearity introduces arbitrariness ➠ instability


Standard errors too small ➠ overconfidence


Use of arbitrary inclusion criteria


Even if we only care about predictions, the overfitting should worry us

Embedded methods

Embedded methods

Also unstable under collinearity

Embedded methods

Only real contenders of penalized likelihood variety(eg. LASSO)

Embedded methods

Difficult to sensibly use categorical variables

Embedded methods

Difficult to embed prior information(pathway info &c.)

All variable-selected models difficult to interpret

Post–selection inference


Classical inference treats hypothesis as fixed; now it is often random


Resampling, data splitting possible,can be hard to get right


Bayesian methodology mostyl sidesteps the inferential problems.

More work to model, compute-heavy. “Subjective.”

Reducing number of variables blinded to Y

• Remove low-variance variables

• Remove mostly-missing variables

• Statistical tricks to combine collinear variables &c. (see refs)

• DOMAIN KNOWLEDGE™

Reducing number of variables blinded to Y

• Remove low-variance variables

• Remove mostly-missing variables

• Statistical tricks to combine collinear variables &c. (see refs)

• Domain knowledge

To summarize

• Variable selection is a modern “problem” • Genomics is an archetypal application area• Penalized likelihood methods probably most reliable• Inference is tricky• Domain knowledge is both a challenge and a possibility

Nobel laureate Lars Hansen (emphasis mine).

Data seldom, if ever, speaks for itself. To use data effectively requires valid and revealing conceptual frameworks for

understanding and interpreting patterns in data.

https://qz.com/1417105/lars-hansen-a-nobel-winning-economist-has-tips-for-dealing-with-uncertainty/

THANK YOU

Einar HolsbøFebruary 8th, 2019

Bibliography

• Harrell: “Regression modeling strategies”

• Hastie &al.: “Elements of statistical learning”

• Hira & Gillies: “A review of feature selection and feature extraction methods applied on microarray data”

• The methods SAM, LIMMA, and k-TSP

variable selection in genomics - github...

Documents