Consistent probabilistic outputs for protein function prediction

William Stafford NobleDepartment of Genome Sciences

Department of Computer Science and EngineeringUniversity of Washington

Outline

• Motivation and background

• Methods– Shared base method– Reconciliation methods

• Results

The problem

Given:• protein sequence,• knockout phenotype,• gene expression

profile,• protein-protein

interactions, and • phylogenetic profile

Predict• a probability for every

term in the Gene Ontology

Heterogeneous dataMissing dataMultiple labels per geneStructured output

Consistent predictions

Cytoplasmic membrane-bound

vesicle(GO:0016023)

Cytoplasmic vesicle

(GO:0031410)

is a

The probability that protein X is a cytoplasmic membrane-bound vesicle must be less than or equal to the probability that protein X is a cytoplasmic vesicle.

Data sets

Kernels

SVM → Naïve BayesData 1

Data 2

Data 3

Data 4

Data 5

Data 6

Data 7

Data 8

Data 33

SVM/AL 1

SVM/AL 2

SVM/AL 3

SVM/AL 4

SVM/AL 5

SVM/AL 6

SVM/AL 7

SVM/AL 8

SVM/AL 33

Product, plus Bayes’ rule

Probability 1

Probability 2

Probability 3

Probability 4

Probability 6

Probability 8

Probability 33

Probability

Gaussian

Asymmetric Laplace

SVM → logistic regressionData 1

Data 2

Data 3

Data 4

Data 5

Data 6

Data 7

Data 8

Data 33

SVM 1

SVM 2

SVM 3

SVM 4

SVM 5

SVM 6

SVM 7

SVM 8

SVM 33

Logisticregressor 1

Logisticregressor 2

Logisticregressor 3

Logisticregressor 11

Predict 1

Predict 2

Predict 3

Predict 4

Predict 6

Predict 8

Predict 33

Probability

Reconciliation Methods

• 3 heuristic methods

• 3 Bayesian networks

• 1 cascaded logistic regression

• 3 projection methods

Heuristic methods

• Max: Report the maximum probability of self and all descendants.

• And: Report the product of probabilities of all ancestors and self.

• Or: Compute the probability that at least one descendant of the GO term is “on,” assuming independence.

jDj

i ppi

ˆmax

• All three methods use probabilities estimated by logistic regression.

iAj

ji pp ˆ

iDj

ji pp ˆ11

Bayesian network

• Belief propagation on a graphical model with the topology of the GO.

• Given Yi, the distribution of each SVM output Xi is modeled as an independent asymmetric Laplace distribution.

• Solved using a variational inference algorithm.• “Flipped” variant: reverse the directionality of edges in the graph.

Cascaded logistic regression

• Fit a logistic regression to the SVM output only for those proteins that belong to all parent terms.

• Models the conditional distribution of the term, given all parents.

• The final probability is the product of these conditionals:

iAj

ji pp

Ejipp

pp

ij

Iiii

Iipi

, , s.t.

ˆ 2

,min

Isotonic regression

• Consider the squared Euclidean distance between two sets of probabilities.

• Find the closest set of probabilities to the logistic regression values that satisfy all the inequality constraints.

Ejipp

pp

ij

Iiii

Iipi

, , s.t.

ˆ 2

,min

Ejipp

ppD

ij

Iiii

Iipi

, , s.t.

ˆmin ,

Isotonic regression

• Consider the squared Euclidean distance between two sets of probabilities.

• Find the closest set of probabilities to the logistic regression values that satisfy all the inequality constraints.

Küllback-Leibler projection

• Küllback-Leibler projection on the set of distributions which factorize according to the ontology graph.

• Two variants, depending on the directions of the edges.

Likelihood ratiosobtained from

logistic regression

Hybrid method

• Replace the Bayesian log posterior for Yi by the marginal log posterior obtained from the logistic regression.

• Uses discriminative posteriors from logistic regression, but still uses a structural prior.

BPAL KLP

BPLR

Axes of evaluation

• Ontology– biological process– cellular compartment– molecular function

• Term size – 3-10 proteins– 11-30 proteins– 31-100 proteins– 100-200 proteins

• Evaluation mode– Joint evaluation– Per protein– Per term

• Recall– 1%– 10%– 50%– 80%

Legend

Belief propagation, asymmetric Laplace

Belief propagation, asymmetric Laplace, flipped

Belief propagation, logistic regression

Cascaded logistic regression

Isotonic regression

Logistic regressionKüllback-Leibler projection

Küllback-Leibler projection, flipped

Naïve Bayes, asymmetric Laplace

Pre

cisi

on

TP

/(T

P+

FP

)

Recall TP / (TP+FN)

Joint evaluation

Biological process ontology

Large terms (101-200)

Biological process ontology

Molecular function ontology

Cellular compartment

ontology

Conclusions: Joint evaluation

• Reconciliation does not always help.

• Isotonic regression performs well overall, especially for recall > 20%.

• For lower recall values, both Küllback-Leibler projection methods work well.

Average precision per protein

Biological process

All term sizes

Biological process

Statistical significanceBiological process

Large terms

Biological process

Large terms

Biological process

101-

200

31-1

0011

-30

3-10

953 proteins

435 proteins

239 proteins

100 proteins

Molecular function

101-

200

31-1

0011

-30

3-10

476 proteins

142 proteins

111 proteins

35 proteins

Cellular component

101-

200

31-1

0011

-30

3-10

196 proteins

135 proteins

171 proteins

278 proteins

Conclusions: per protein

• Several methods perform well– Unreconciled logistic regression– Unreconciled naïve Bayes– Isotonic regression– Belief propagation with asymmetric Laplace

• For small terms– For molecular function and biological process, we do

not observe many significant differences.– For cellular components, belief propagation with

logistic regression works well.

Average precision per term

Biological process

All term sizes

Biological process

101-

200

31-1

0011

-30

3-10

953 terms

435 terms

239 terms

100 terms

Molecular function

101-

200

31-1

0011

-30

3-10

476 terms

142 terms

111 terms

35 terms

Cellular component

101-

200

31-1

0011

-30

3-10

152 terms

97 terms

48 terms

30 terms

Conclusions

• Reconciliation does not always help.

• Isotonic regression (IR) performs well overall.

• For small biological process and molecular function terms, it is less clear that IR is one of the best methods.

Acknowledgments

Guillaume Obozinski

Charles Grant

Gert Lanckriet

Michael Jordan

The mousefunc organizers• Tim Hughes• Lourdes Pena-Castillo• Fritz Roth• Gabriel Berriz• Frank Gibbons

Per term for small termsBiological process

Molecular function

Cellular component