supervised convolutional gsn for protein secondary ...jzthree/datasets/icml2014/slides.pdf•...
TRANSCRIPT
![Page 1: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/1.jpg)
Supervised Convolutional GSN
for Protein Secondary Structure
Prediction
Jian Zhou
Olga Troyanskaya
Princeton University
![Page 2: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/2.jpg)
What’s In this talk..
• Problem: Predict protein secondary structure
• Iterative prediction with multi-layer hierarchical
representation
– Supervised GSN
– Convolutional architecture for GSN
– A trick for improving convergence and performance
• Performance evaluations
![Page 3: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/3.jpg)
Previous Approaches: neural network
from 1988 (Qian & Sejnowski); bidirectioal
recurrent neural network (Baldi et al.,
1999); conditional neural fields (Peng et al.,
2009); many more…
Protein secondary structure
prediction
MDLSALRVEEVQNVINAMQKILECP
ICLELIKEPVSTKCDHIFCKFCMLKL
LNQKKGPSQCPLCKNDITKRSLQE
STRFSQLVEELLKIICAFQLDTGLEY
ANSYNFAKKGK
Protein sequence
CCGGGSSHHHHHHHHHHHHHHTS
CSSSCCCCSSCCBCTTSCCCCSH
HHHHHHHSSSSSCCCTTTSCCCC
TTTCBCCCSSSHHHHHHHHHHHH
HHHHTCCCCCC
Secondary structure
Image credit:
Wikimedia common
Predict
20 types of amino acids
8 classes
3D structure
![Page 4: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/4.jpg)
Protein Sequence -> Secondary Structure
Protein sequence 20 types of amino acids
8 classes Secondary structure
label sequence
Predict
Evolutionary
neighborhood
3D structure
![Page 5: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/5.jpg)
Motivation
• Challenge: Prediction with both local and long-range dependencies
• Plan:
Multi-layer hierarchical representation
Both ‘upward’ and ‘downward’
connections
Supervised GSN formulation
![Page 6: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/6.jpg)
Model
𝐻𝑡+1 ~ 𝑃𝜃1 𝐻 𝐻𝑡, 𝑋𝑡𝑋𝑡+1 ~ 𝑃𝜃2 𝑋 𝐻𝑡+1)
𝐻1 𝐻2
𝑋0 𝑋1 𝑋2
𝐻3
• Generative Stochastic Network
Learning the transition operators of a Markov chain whose
stationary distribution estimates the data distribution 𝑃 (𝑋).
Learning 𝑃 𝑋 𝐻) can be much easier than 𝑃 (𝑋) by design.
Trainable using back-propagation
𝐻0
Bengio, Y., Thibodeau-Laufer, É., Alain, G., and Yosinski, J.
Deep Generative Stochastic Networks Trainable by Backprop
![Page 7: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/7.jpg)
Model
𝐻𝑡+1 ~ 𝑃𝜃1 𝐻 𝐻𝑡, 𝑋𝑡𝑋𝑡+1 ~ 𝑃𝜃2 𝑋 𝐻𝑡+1)
𝐻𝑡+1 ~ 𝑃𝜃1 𝐻 𝐻𝑡 , 𝑌𝑡, 𝑋0𝑌𝑡+1 ~ 𝑃𝜃2 𝑌 𝐻𝑡+1)
𝐻1 𝐻2
𝑋0 𝑋1 𝑋2
𝐻3
𝐻1 𝐻2
𝑌0 𝑌1
𝑋0
𝑌2
𝐻3
P(X)
P(Y|X)
GSN
Supervised
GSN
𝐻0
𝐻0
Learning 𝑃 𝑌 𝐻) can be much easier than 𝑃 𝑌 𝑋 , utilizing
previous state of the chain
![Page 8: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/8.jpg)
Model
𝐻𝑡+1 ~ 𝑃𝜃1 𝐻 𝐻𝑡 , 𝑌𝑡, 𝑋0𝑌𝑡+1 ~ 𝑃𝜃2 𝑌 𝐻𝑡+1)
𝐻1 𝐻2
𝑌0 𝑌1
𝑋0
𝑌2
𝐻3
P(Y|X)
Supervised
GSN
Maximize log-likelihoods
True 𝑃(𝑌|𝑋0)
𝑃𝜃(𝑌|𝐻1)
𝑃𝜃(𝑌|𝐻2)
𝑌0 𝑌1
![Page 9: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/9.jpg)
Architecture for protein secondary structure prediction
Multi-scale representation – multi-layer convolutional architecture
Local information sensitive – output unit at bottom layer
𝑌
𝐻0
𝑋
W1’W1 W1’ W1 W1 W1’
W2 W2’ W2 W2W2’
𝑌1
𝐻1
𝑌2 𝑌3
𝐻1 𝐻2
𝑌0
𝑋0
𝐻3
Model
Conv
Pool
Conv
𝑌
𝐻0
𝑋
𝐻1
tanh
tanh
Mean
pooling
…
![Page 10: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/10.jpg)
Training
Initialize at a specified test initialization
value for a subset of training batches:
Experiments on initialization of chain during training
𝑌
𝐻0
𝑋
W1’W1 W1’ W1 W1 W1’
W2 W2’ W2 W2W2’
𝑌1
𝐻1
𝑌2 𝑌3
𝐻1 𝐻2
𝑌0
𝑋0
𝐻3
Accuracy
# of iterations
0%20%
50%
80%
100%
Accuracy
# of iterations
- Optimal performance at 50% test initialization
𝑌0 𝑡𝑟𝑢𝑒
𝑌0 𝑡𝑒𝑠𝑡
![Page 11: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/11.jpg)
Performance
CullPDB-30
test set
Overall Accuracy
(8-class)
1 layer 0.714± 0.006
2 layers 0.720± 0.006
3 layers 0.721 ± 0.006
CB513 dataset Overall Accuracy
(8-class)
RaptorSS8/CNF 0.649 ± 0.003
Our method 0.664 ± 0.005
Cull PDB dataset (6133 proteins with <30% identity between any protein pairs);
available at www.princeton.edu/~jzthree/datasets
single protein prediction example Performance through averaging iterative predictions:
𝑌1
𝑌2
𝑌4
𝑌8
𝑌16
𝑌32
𝐿𝑎𝑏𝑒𝑙
![Page 12: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/12.jpg)
Summary
• We developed supervised convolutional GSN model for
protein secondary structure prediction.
• Supervised GSN
– Stochastic iterative prediction through Markov chain
– Initialization trick improve both performance and convergence
rate empirically
• Convolutional architecture for Supervised GSN
– Combine high level representation and local prediction
– Improved over previous best performance
![Page 13: Supervised Convolutional GSN for Protein Secondary ...jzthree/datasets/ICML2014/slides.pdf• Supervised GSN –Stochastic iterative prediction through Markov chain –Initialization](https://reader036.vdocument.in/reader036/viewer/2022071413/610afb2ac8c9100fdf488a61/html5/thumbnails/13.jpg)
• Filters: Layer1, 𝑋, 𝑌 ↔ 𝐻0
𝑊𝑋→𝐻0
(Amino
acids)
Position
Channel
𝑊𝑌→𝐻0
(Secondary
structure)
𝑊𝐻0→𝑌(Secondary
structure)