reproducibility of model-based results: standards, infrastructure, and recognition

http://sems.uni-rostock.de

Dagmar WaltemathSeptember 2015, Rostock-Warnemünde | dcite

Reproducibility of model-based results:Standards, infrastructure and recognition


What is a model?

Fig.: Modeling Cellular Reprogramming Using Network-based Models. Courtesy Antonio del Sol Mesa, LCSB Luxembourg

Fig.: Modeling the cell cycle using ODE systems. Goldbeter (1991), http://www.ncbi.nlm.nih.gov/pubmed/1833774

Fig.: Modeling large-scale networks. Lee et al (2013), http://www.nature.com/articles/srep02197.

2In systems biology, a computational model represents biological facts in

the computer. Often, the representation is simulated to help understand

the system's dynamic behavior.

http://www.ncbi.nlm.nih.gov/pubmed/1833774

http://www.nature.com/articles/srep02197


Re[usea|produci]bility challenge

3

Slide courtesy Mike Hucka @ 2012 Computational Cell Biology Summer School


Re[usea|produci]bility challenge

4


“With greater interaction between tools, anda common format for publications and databases, userswould be better able to spend more time on actual research

rather than on struggling with data format issues.”


Re[usea|produci]bility challenge (2003)

5


“With greater interaction between tools, anda common format for publications and databases, userswould be better able to spend more time on actual research

rather than on struggling with data format issues.” (SBML L1)


→ Standardised model representation

6

Ron Henkel et al. Database 2015;2015:bau130



7

Fig.: Nature Blogs: Of Schemes and Dreams (2014)

Nine Worrying Stats on the Effect of Poor Scientific Data Management

Vijayalakshmi Chelliah et al. Nucl. Acids Res. 2015;43:D542-D548

Finding relevant models.

http://blogs.nature.com/ofschemesandmemes/2014/02/03/nine-worrying-stats-on-the-effect-of-poor-scientific-data-management


→ Strategies for model similarity, ranking, clustering, filtering

Fig.: Henkel et al 2010 http://www.biomedcentral.com/1471-2105/11/423/

Fig.: Schulz et al 2011 DOI: 10.1038/msb.2011.41

x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x

CellCycle Models

x x x x x x

x x x x x

x x

x

x x x x

x x x

x

x x x x

x x x

x

x

x x x x

x

x x x x

x x x

x x

x x

x x x

x x

x x x x

x x x

x

x x x x x

x x x x x x

x x x x x x

x x x x x x x

x x x x x

x x x x x

x x x x

x x x x x x x

x x x x x x

x x x x x x x x x

x x x x x x

x x

Fig.:Alm et al (2014) doi:10.1186/s13326-015-0014-4

http://www.biomedcentral.com/1471-2105/11/423/



Reproducing published models.


→ Standardised simulation descriptions

Fig.:Waltemath et al (2012) doi:10.1186/1752-0509-5-198



Model-related data in the systems biology workflow

Linking the relevant files.


→ Retrieval and archiving of simulation studies and asssociated files

Model-related data in the systems biology workflow

Linking model-related data

Give me all the files I need to run this simulation study.

Which are the most frequently used GO annotations in my model set?

Which models contain reactions with 'ATP' as reactant and 'ADP' as product?

Find good candidates for features describing my set of

models.


State of affairs in 2015

● Standards:

– support for all steps of the modeling cycle

– support of various modeling techniques

– Still: some modeling concept not yet covered (→ Report of whole Cell modeling workshop, Waltemath et al 2015 (under review))

● Infrastructures:

– Software tools export/import standards

– Open model repositories and management systems

– Education

● Recognition


COMBINE Standards

● COmputational Modeling in BIology Network

● Goals:– Avoid overlap of standardisation efforts– Coordinate standard developments– Coordinate meetings – Coordinate development of procedures & tools– common infrastructure for specification development, semantic

annotation, and dissemination

● All specifications now citable and accessible in one place: Schreiber et al. (2015) http://journal.imbio.de/articles/pdf/jib-258.pdf

http://journal.imbio.de/articles/pdf/jib-258.pdf


COMBINE Standards

Fig. : COMBINE standards today. Slide courtesy M. Hucka. http://www.slideshare.net/thehuck/a-summary-of-various-combine-standardization-activities

http://www.slideshare.net/thehuck/a-summary-of-various-combine-standardization-activities


COMBINE Standards

● Data formats– Community-developed representation formats for models and

related data– Format: XML, OWL, RDF/XML

● Minimum Information/Reporting guidelines:– Minimum amount of data and information required reproduce

and interpret an experiment– Format: human-readable specification documents

● Basis for the specification of data models and metadata● Bio-ontologies


SBML

Fig.: SBML Level 3 Packages. Slide courtesy M. Hucka (ICSB 2014).


SBML

Fig.: SBML Level 3 Packages. Slide courtesy M. Hucka (ICSB 2014).

Lucky modelers: You should not need to worry about the details of these (XML) formats, the tools should handle import and export! (Tool developers should though.)


Minimum Information Guidelines

● Reporting guidelines and checklists● Narrative description of the information necessary to

reproduce a model-based result● MIRIAM: Minimum Information about the Annotation of a

Model● MIASE: Minimum Information about a Simulation Experiment● MIAPE,MIAME… for experimental setups


MIRIAM – information to provide about a model

● Models must– be encoded in a public machine readable format– be clearly linked to a single publication– reflect the structure of the biological processes described in the

reference paper (list of reactions, …)– be instantiable in a simulation (possess initial conditions, …)– be able to reproduce the results given in the reference paper– contain creator’s contact details– unambiguously identify each model constituent through annotation


MIRIAM – information to provide about a model

● Models must– be encoded in a public machine readable format– be clearly linked to a single publication– reflect the structure of the biological processes described in the

reference paper (list of reactions, …)– be instantiable in a simulation (possess initial conditions, …)– be able to reproduce the results given in the reference paper– contain creator’s contact details– unambiguously identify each model constituent through annotation

You should worry about the details of the guidelines, as they help you to check whether you provide all necessary information.


Bio-ontologies for model annotation

● Major ontologies

● Linking framework: RDF/XML

● Annotation scheme: used to semantically enrich model files with detailed descriptions of the underlying biological entities, mathematical concepts or algorithms used during analysis

● De facto standard: SBML annotation scheme



enzyme

enzyme

product

substrate

enzymatic rate law

catalytic rate constant

urn:miriam:SBO:0000011urn:miriam:SBO:0000014

urn:miriam:SBO:0000014





Tyrosine

Phenylalanine-4-hydroxylase

Phenylalanine-4-hydroxylase

Tetrahydrobiopterin

urn:miriam:uniprot:P00439

urn:miriam:uniprot:Q03393




Levels of standardisation

Fig.: COMBINE standards that are relevant to this workshop; adapted from (Chelliah et al., 2009, DILS)



● Standards:


– support of various modeling technique




– Open model repositories and management systems

– Education

● Recognition


Software tool support

● Standard converters (SBML ↔ SBGN; SBML ↔ CellML...)● Standard support in software● Interoperability tools

– Cytoscape for network analysis and visualization (SBML, SBGN, BioPax)

– The Virtual Cell for modeling (SBML, BioPAx)– VANTED for network analysis, visualization and manipulation

(SBML, SBGN)Check COMBINE Website

for details

http://co.mbine.org/


Software tool support in SBML

Fig.: Software supporting SBML. Slide courtesy M. Hucka (ICSB 2014).

Also check the SBML Software Matrix

http://sbml.org/SBML_Software_Guide/SBML_Software_Matrix


Open model repositories

● Structured, type-specific archives● Offer download of curated, annotated, published models

and associated files (visual representations, simulation descriptions, publication…)

CCDB


Model management systems

Fig.: The SEEK. Wolstencroft et al (2015). doi:10.1186/s12918-015-0174-y

Model management tasks:● Storage & Integration of data● Search & Retrieval ● Version Control● Provenance


Getting involved

● COMBINE user meeting→ next: COMBINE 2015, OCT 11-16, Salt Lake City

● COMBINE developers meeting → next: HARMONY 2016, June 7-11, Auckland

● FAIR-DOM activities: webinars, blogs, foundries● COMBINE activities: workshops, presentations, tutorials● Help through specification documents, show cases, mailing

lists, ...

http://co.mbine.org/ http://fair-dom.org/


http://fair-dom.org/



● Standards:


– support of various modeling technique



– Open model repositories


– Model management systems

– Education

● Recognition


Recognition

33

1) Higher visibility of research

2) Long-term availability

3) Link to other resources

4) Quality-checks

Fig.: Piwowar and Vision (2013) Data reuse and the open data citation advantage. PeerJ

https://peerj.com/articles/175/


Model curation and publication in BioModels Database

Fig.: Li et al (2010)


Functional curation of models through virtual experiments

Fig.: Functional curation of models in the Web Lab. Cooper et al (2015) https://peerj.com/preprints/1338/ ; Cooper et al (2014) doi:10.1016/j.pbiomolbio.2014.10.001

Try out theCardiac physiology

Web Lab

https://peerj.com/preprints/1338/

https://travis.cs.ox.ac.uk/FunctionalCuration/


Enabling model version control

Fig.: courtesy Martin Scharm, BudHat

http://sems.uni-rostock.de/budhat


Enabling on-the-fly reproduction of the model-based results

Fig.: Software supporting SBML and SED-ML.Waltemath et al (2011). doi:10.1186/1752-0509-5-198


So far for the theory… and in practice?

● Check for existing standards and specifications thereof: http://co.mbine.org

● Get involved in standard development → through the relevant mailing lists

● Problems with getting your model into the right format?

– Is it a problem with finding the approriate format or tool? → Ask on the relevant mailing list... people are friendly and happy to help.

– Is it a tool problem? → Complain with tool developers... who will hopefully change it.

– Is is a problem with the lack of a standards? → Feed back into the standard's community… people are friendly and happy to improve the standard.

● Follow best practices when aiming at publishing a result.



Best practices for publishing reproducible modeling results

1) Encode the model in a standard format, e.g. SBML.

2) Annotate the SBML model, following MIRIAM.

3) Publish the simulation experiment descriptions in standard format, e.g. SED-ML. If unsure what to include, consult the MIASE guidelines.

4) Try to reproduce the results *yourself*.

5) Ask a colleague to reproduce the results.

6) If successful: Archive all steps that led to your results.

7) Disseminate model code and simulation description through an open repository. Adapted from: Waltemath et al (2013), doi:10.1007/978-94-007-6803-1_10

reproducibility of model-based results: standards, infrastructure, and recognition

Science