data collection methods to improve reproducibility

Data collection methods to improve reproducibility

Louis Culot, CEO, BioData

Published 1977Main thesis:

Running = Good !

Fix dies at age 52 (running)In 1984.

Two camps: Running = Good ! Running = maybe good… moderation too much of a good thing?

Conclusions The findings suggest a U-shaped association between all-cause mortality and dose of jogging as calibrated by pace, quantity, and frequency of jogging. Light and moderate joggers have lower mortality than sedentary nonjoggers, whereas strenuous joggers have a mortality rate not statistically different from that of the sedentary group.

UHow much running

Bad Bad

Great !

Schnohr, Peter, et al. "Dose of jogging and long-term mortality: the Copenhagen City Heart Study." Journal of the American College of Cardiology 65.5 (2015): 411-419.

Why Jogging May Be Better For Your Health Than Running

So the connection was this: Joggers of mild and moderate intensity had a lower risk of death than the strenuous joggers. In fact, the lowest mortality risk was that of the mild intensity joggers. The fast-paced joggers had about the same rate of mortality as sedentary people. This suggests that there may be an upper limit to in vigorous exercise, after which the benefits fall off.

A recent study reported that joggers who exercise strenuously have the same life expectancy as people who do barely any exercise at all. But the author has now admitted he hasn't actually proved this.

But, although the overall number of people studied was large, the number of strenuous joggers was not. Only 36 people fitted the strenuous jogging category - two of them had died.

State of Reproducibility

-Amgen study: Only 6 of the 53 studies were reproduced (about 11%).1

-Bayer study: Only 14 out of the 67 projects (about 21%).2

-From 28% to 18% for clinical trials 2006-2007 – 2008-2010.3

”…poor training, an emphasis on provocative conclusions in papers, a dearth of experimental details, and an overemphasis on publications in high-impact journals” – Francis Collins, NIH Director

1. Begley, C. Glenn, and Ellis, Lee M., "Drug development: Raise standards for preclinical cancer research." Nature 483.7391 (2012): 531-533

2. Prinz, Florian, Thomas Schlange, and Khusru Asadullah. "Believe it or not: how much can we rely on published data on potential drug targets?." Nature reviews Drug discovery 10.9 (2011): 712-712.

3. Nature Reviews Drug Discovery 10, 328-329 2011

Examples of Recommendations:

“We recommend the following steps to change the culture of oncology research and improve the relevance of translational studies” (Begley and Ellis):

There must be more opportunities to present negative data. It should be the expectation that negative preclinical data will be presented at conferences and in publications. Preclinical investigators should be required to report all findings, regardless of the outcome. To facilitate this, funding agencies, reviewers and journal editors must agree that negative data can be just as informative as positive data.

Journal editors must play an active part in initiating a cultural change. There must be mechanisms to report negative data that are accessible through PubMed or other search engines. There should be links to journal articles in which investigators have reported alternative findings to those in an initial (sometimes considered landmark) publication. One suggestion is to include 'tags' that report whether the key findings of a seminal paper were confirmed.

Other: Reporting unethical behavior, dialog between clinicians and patients and scientists; universities prioritizing teaching in tenure decisions (research is too “high stakes”). Better access to technologies – e.g., new cell lines, capabilities for genetic characterization of new tumor cell lines and xenografts.

“5 Practical Points to Consider”Poland, C. A., Miller, M. R., Duffin, R., & Cassee, F. (2014). The elephant in the room: reproducibility in toxicology. Particle and fibre toxicology, 11(1), 42.

1. Be specific.2. Characterize, Characterize, Characterize 3. Use statistics to question your results rather than simply confirming a theory4. Write your hypothesis in advance of running the experiment.5. Take a ‘belts and suspenders’ approach to confirming findings.


1. Be specific.

For example, what is your sample? If it’s graphene, then is it actually graphene i.e. a monolayer, or is it few layer graphene, or graphite platelets? If they are ambient particles, where and how were they collected (date, time, weather conditions etc.)? State this, it is important - later studies may not replicate your results simply due to inadvertently testing the non-similar materials i.e. comparing apples and oranges.

=?


1. Be specific.2. Characterize, Characterize, Characterize

Only with independent characterisation do you truly know what you are testing and allow others to reproduce your findings. …. characterise in relation to your hypothesis or research question - do not characterise because there is a default list of parameters that you need to check.

=? +

1. Sorge, Robert E., et al. "Olfactory exposure to males, including men, causes stress and related analgesia in rodents." Nature methods 11.6 (2014): 629-632.

JAX CD34 JAX CD34


Ask yourself, is barely statistically significant really biologically relevant? Can you support/validate an entire hypothesis or more controversially, a provocative statement using P < 0.05?

1. Be specific.2. Characterize, Characterize, Characterize3. Use statistics to question your results rather than simply confirming a theory

Revisit the power of negative results

The Economist ‘Trouble at the Lab’ 2013 Oct 19; available at http://go.nature.com/dstij3

The Economist ‘Trouble at the Lab’ 2013 Oct 19; available at http://go.nature.com/dstij3

“The root of all superstition is that men observe when a thing hits, but not when it misses.”

…Francis Bacon


1. Be specific.2. Characterize, Characterize, Characterize 3. Use statistics to question your results rather than simply confirming a theory4. Write your hypothesis in advance of running the experiment.

It should be clear from the experimental design how you have tried to disprove this. If your theory is robust it should survive your best efforts to challenge it.


1. Be specific.2. Characterize, Characterize, Characterize 3. Use statistics to question your results rather than simply confirming a theory4. Write your hypothesis in advance of running the experiment.5. Take a ‘belts and suspenders’ approach to confirming findings.

Replicate time points, dosages, protocols, etc.

IC50 = 3.25 μl/mgvs

vs

Researcher 1

Researcher 2

Researcher 3

How labs use Labguru

- Biospecimen management / sample tracking.- Project and experiment planing.- Sharing of data (internally and externally).- Managing collaborations.- Communicating with funders.- In-house repository for raw data and analysis.

What does it enable ?Science is struggling with reproducibility

Scientific traceability01

Ability to build your own and others’ work

Communication as part of the experiment

02

03

It starts with how scientists manage their research.

What affects reproducibility?It starts with scientists

1. Protocol Management: Lost protocols, inconsistent protocol updates, lab members using different protocols, and a lack of protocol – experimental and materials linkage.

2. Reagent and Specimen Details: Reagent – experiment coupling. Reagent conditions. Location of reagents. Inconsistent procedures for reagent synthesis. Genealogy.

3. Access to negative results: Often more reliable, and containing valuable information to develop new hypotheses.

4. Experimental Design: Disconnected data, lack of scientific context, and an incomplete picture make it hard for other scientists to reproduce work.

“a problem well put is half solved.”… John Dewey

1. Protocol Management

• Lost protocols• Inconsistent protocol

updates• Lab members using

different protocols• Lack of protocol -

experimental linkage

The problem:

What affects reproducibility?

• Keep track of all protocols used• Designate & protocols between

lab members• Keep a record of what versions

were used for which experiments

• Annotate protocols as they are executed.

The solution:

• Protocol Versioning• Built in protocol versioning and tracking system • Link to experiments for full scientific context

The Labguru Approach:

2. Reagent and Specimen Details

• Reagent – experiment coupling

• Reagent conditions• Specimen meta-data and

provenance.• Location, origin, date, and

handling.

The problem:


• Record & track all successful reagents and specimens along with their metadata

• Link specific specimens & reagents back to individual experiments

• Identify problems from experimental records

The solution:

• Specimen and Reagent Tracking• Purchased materials are linked to experiments which use them• Smart modules for plasmids, cell lines bacteria, tube storage, and more• Specimens are tracked with meta-data and storage history.


3. Access to Negative Results

• Negative results often ‘lost’ over time in laboratories. Never published.

The problem:


• Ready access to all project and experiments regardless of outcome.

• Ability to back-test and build new hypotheses.

• Increase ability to rescue failed experiments.

The solution:

• All data available within a lab (under PI control).• Data can be selectively shared and pushed to public repositories.• Simple ability to save “all data” in one place (including linking to

big data, such as genomics and images).


4. Experimental Design

• Disconnected data• Lack of scientific context• Incomplete picture• Loss of data when

someone leaves the lab

The problem:


• Begin with hypothesis testing.• Create detailed mini-reports that

encompass all relevant data for an experiment

• Connect related experiments and assets/materials.

• Selectively reproduce “difficult” parts of experiments.

The solution:

• Platform for Searchability and Collaboration• Access data based on researcher, materials used, project, keywords• Integrated commenting and activity feeds increase accessibility• Simple “project management” approach to managing experiments.


VersioningTrackingSharinghistory

ProvenanceGenealogyVendors

Experiments & Protocols

CalibrationExperiments

VendorResults (e.g., drift)

Change historyDuplicatesResults

Contributors

The Digital Laboratory Ecosystem

LabguruLab Logistics/Notebook



Physical Materials/Specimens

Equipment

Big data servers

GenomicsImaging



Physical Materials/Specimens

Equipment

Big data servers

General Repositories

Domain Repositories

GenomicsImaging

Manuscripts

Please contact us with any questions

Visit labguru.com for a 30 day free trial of Labguru

THANK YOU FOR COMING!

data collection methods to improve reproducibility

Science

nature reviews drug

drug development

potential drug targets

amgen study

bayer study

published data

recent study

state of reproducibility