![Page 1: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/1.jpg)
FM Model Assessment: Old scores and New Combinations
ShuoYong ShiNick Grishin Lab
![Page 2: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/2.jpg)
Scores used in CASP5 Assessment*: Ten scores encompass structure and sequence characteristics
Generate Scores
GDT
Distance
DALI Qdali
CE Qce
Mammoth Qmm
LGA Qlga
SOV
Q ratios: Correctly aligned residues/ target length
*Kinch LN, et al. Proteins. 2003; 53 Suppl 6:395-409
![Page 3: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/3.jpg)
Structure Score Distributions: Target 531
GDT
Distance
Dali
Mammoth
![Page 4: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/4.jpg)
Structure Score Distributions: Target 531
GDT
Distance
Dali
Mammoth
399_5 113_1360_4
399_5113_2119_1215_1
![Page 5: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/5.jpg)
Sequence Score Distributions: Target 531
Qlga
SOV
Qdali
Qmm
399_5 1_3
55_1 208_2
![Page 6: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/6.jpg)
Bad Score Distributions: Target 531
CE Qce
![Page 7: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/7.jpg)
Replace Score Distributions: Target 531
CE Qce
Tmalign Qtm113_4 490_2
![Page 8: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/8.jpg)
Combine Ten Scores: TenS
Strategy to combine scores:
1) Transform raw scores to Z-scores
2) Sum of Z-scores using equal weight (TenS)
• Throw out zeros, calculate Z-score
• Throw out raw score with Z-score<-2, recalculate mean and std
• Recalculate Z-score on entire population, and assign Z-score <-2 to -2.
![Page 9: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/9.jpg)
Combine Scores (TenS): Target 531
TenS
GDT
GDT winner: TS399_5
TenS winner: TS208_2
![Page 10: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/10.jpg)
Score Used in Components Raw FormTenS new 10 (GDT) Z scoreQCS new 6 (CS) % MaxGDT CASP* % Max CS CASP8 % Max
Strategy to combine scores:
1) Transform raw scores to Z-scores
2) Sum of Z-scores for each target (ComS)
Combine Main Scores: TenS, QCS, GDT, and CS
![Page 11: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/11.jpg)
Motivation: Top performing groups use similar strategies that rank server models with various scoring functions and refine top picks
Who did better than servers?
Score Strategy: Ratio of best group model scores to top server model for each of the main scores
Additional Score: Comparison to top server Model
![Page 12: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/12.jpg)
Server Ratio Score Combinations:
1)Ratio scores below 1 are ignored
2)Average Scores(4) for each target
3)Sum score averages
Additional Score: Comparison to top server Model
The Sum of average ratios (which are rarely much larger than 1) indicates the number of times each group outperformed servers
![Page 13: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/13.jpg)
Additional Score: Comparison to top server Model (Target 531 Example)
TopServers
![Page 14: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/14.jpg)
Additional Score: Comparison to top server Model (Combined Scores)
Servers
Outperformed servers 14 times
![Page 15: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/15.jpg)
Report Scores: Many, Many, Many Sortable Tables and a Web Site
http://prodata.swmed.edu/CASP9/evaluation/CASP9Home.htm
![Page 16: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/16.jpg)
Report Scores: Many, Many, Many Sortable Tables and a Web Site
![Page 17: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/17.jpg)
Report Scores: Many, Many, Many Sortable Tables and a Web Site
![Page 18: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/18.jpg)
Report Scores: Many, Many, Many Sortable Tables and a Web Site
![Page 19: FM Model Assessment: Old scores and New Combinations](https://reader030.vdocument.in/reader030/viewer/2022032709/56813031550346895d95c8e8/html5/thumbnails/19.jpg)
Report Scores: Many, Many, Many Sortable Tables and a Web Site
What’s the final ranking?