cockerill rs350-day3-what-can-be-done
Post on 18-Aug-2015
14 Views
Preview:
TRANSCRIPT
Matthew Cockerill, Co-Founder, Riffyn Incwww.riffyn.com
What can be done to improve reproducibility?
The future of scientific scholarly communication, Royal Society, 5th May 2015
Reproducibility issues begin upstream of publication
• Biomedical research still has more in common with a “blacksmith shop” than a reliable modern process engineered for reliability
• Essential methodological knowledge is implicit and undocumented
• To make things work “spend time in the relevant lab”
• Technology transfer too often fails• If we want published results to be more
reliable, we need to to change the approach to methodology
Reproducibility issues begin upstream of publication
• Biomedical research still has more in common with a “blacksmith shop” than a reliable modern process engineered for reliability
• Essential methodological knowledge is implicit and undocumented
• To make things work “spend time in the relevant lab”
• Technology transfer too often fails• If we want published results to be
more reliable, we need to to change the approach to methodology
In industry, the costs of haziness are clear
Certainly it is a problem that we don’t write down our
processes - people change what they know, not what
necessarily matters. - Head of Process
Development of a biopharma company
When we first transferred our manufacturing process to full-
scale operations, our yields dropped 90%.
- CEO of a diagnostics company
We lost four months tracking-down a problem at the demo
scale. Root cause was a parameter change that a new
employee thought did not matter.
- VP Process Development of an algae company
Below: Process design sent to manufacturing
Industrial R&D seeks to improve processes
Improve
Design
Analyse
Measure
Industrial R&D seeks to improve processes
Improve
Design
Analyse
Measure
Noise and irreproducibility gets in the way
Insights from manufacturing
Quality-oriented approachDesign• Unambiguous CAD files• Manage complexity via
modularity
Standardize & automate • Reduce variability in execution
Continuous improvement• Use analytics to identify and
resolve remaining causes of noise
Change of mindset • Make metrics transparent to
entire organization is focused on quality
Insights from manufacturing
How can this benefit R&D?• It means you can build reliably on your own results, and those of others• But needs to be more flexible – in R&D processes change all the time!
Quality-oriented approachDesign• Unambiguous CAD files• Manage complexity via
modularity
Standardize & automate • Reduce variability in execution
Continuous improvement• Use analytics to identify and
resolve remaining causes of noise
Change of mindset • Make metrics transparent so
entire organization is focused on quality
Examples of application in real world R&D
Reducing noise allows more rapid progress
Reproduced from Gardner TS, Trends in Biotechnology, March 2013, Vol. 31, No. 3
Data from product development at leading biotech firm
Quality improvement cuts error by 6X. R&D productivity doubles overnight.Fe
rmen
tatio
n pr
oces
s yi
eld
Date
trend lines
30% relative error 5% relative error
Analytics to track down root cause of noiseH
ow it
wor
ks
Define process inputs, outputs and critical variables1
Measure inputs and outputs2
Stra
in H
T sc
reen
ing
scor
e
Tray #
Analyse and improve3
How do we generalize to basic research?
New generation of lab software tools
Make experiments more reproducible via:• Cloud-based software• CAD approach to experimental design • Integrated data-stream capture and
analysis• Sharing and versioning of methodology
via new publication outlets with github-like approaches
Parallel from cloud computing
• Traditional ad hoc approach to configuring and managing IT systems unreliable and unscalable
• Configuration management systems now use automated ‘recipes’ which reproducibly deliver a fully-configured virtual server which behaves the same every time
Reusable workflows are well established in computational science
We need to move this to the lab bench
Taverna Kepler
Galaxy
New approaches are starting to bring similar automation to the lab bench
So what does the future look like?
Standards for structured, interoperable data
Software and tools for lab automation/analytics
Hubs supporting sharing of data and methodology
An open ecosystem for reproducible science
How will we make it happen?
Incentives to drive adoption of new approach
In industry• Measurable improvements to R&D productivity
create powerful financial incentives
In academia • Tools will gain adoption if they make
researchers life easier and more productive• Endorsement and encouragement from funders• Requirement from participating journals for
sharing of experimental process descriptions• Metrics that reward the sharing of reusable
protocols
Cultural shift won’t be easy, but it is needed
Change in the lab requires corresponding change in publishing
top related