Program UpdateDecember 13, 2012

Andrew J. Buckler, MSPrincipal Investigator,

QI-Bench

WITH FUNDING SUPPORT

PROVIDED BY NATIONAL

INSTITUTE OF STANDARDS AND

TECHNOLOGY

Agenda• Enterprise Architecture:

– Requirements overview – Background for non-software engineering professionals– Enterprise architecture modeling

• Analysis library:– Current status and active extensions in progress– Drill-down on segmentation analysis activities

• Update on workflow engine for the Compute Services :– First demonstration of Kepler, using the segmentation

analysis as example

22

Requirements Overview

Background for non-software engineering professionals

• MVC – Model, View, Controller

• Design Patterns

• Frameworks

44

Background for non-software engineering professionals: Model – View – Controller• Model: Represents the state of what we are doing and how we

think about it• View: How we perceive and seem to manipulate the model• Controller: mediator between the Model and View

55Model

View

Controller

Background for non-software engineering professionals: Design Patterns

66

Background for non-software engineering professionals: Frameworks

77

88

Most familiar: Data Services

99

Same as what we’ve been doing with the Reference Data Set Manager in Midas and the ISA files, but extended with a data virtualization layer

for federated and heterogeneous storage

Also familiar: Compute Services

1010

More on this later in the agenda

Less familiar to some, but foundational to the full vision: The Blackboard

1111

Specify already has an early version of this,but substantial modification is planned

Interfacing to existing ecosystem: Workstations

1212

In short: the primary way of working for clinical users is extended rather than changed.

Internal components within QI-Bench to make it work: Controller and Model Layers

1313

Largely hidden from users, but supporting the various use cases.

Internal components within QI-Bench to make it work: QI-Bench REST

1414

To support QI-Bench GUI as well as external systems, notably workstations.

Last but not least: QI-Bench Web GUI

1515

These are the views that we’ve been evolving since early in the program.

(GO TO LATEST TOP LEVEL GUI CONCEPT DEMO)

161616

Compute Services: Objects for the Analyze Library• Capabilities to analyze literature, to extract

• Reported technical performance• Covariates commonly measured in clinical trials

• Capability to analyze data to• Characterize image dataset quality• Characterize datasets of statistical outliers.

• Capability to analyze technical performance of datasets to, e.g.• Characterize effects due to scanner settings, sequence, geography, reader, – scanner model, site, and patient status.• Quantify sources of error and variability• Characterize intra- and inter-reader variability in the reading process.• Evaluate image segmentation algorithms.

• Capability to analyze clinical performance, e.g.• response analysis in clinical trials.• analyze relative effectiveness of response criteria and/or read paradigms.• overcome metric‘s limitations and add complementarity• establish biomarker characteristics and/or value as a surrogate endpoint.

1717

In Place

In Progress

In Queue

Analyze Library: Coding View• Core Analysis Modules:

• AnalyzeBiasAndLinearity• PerformBlandAltmanAndCCC• ModelLinearMixedEffects• ComputeAggregateUncertainty

• Meta-analysis Extraction Modules: • CalculateReadingsFromMeanStdev (written in MATLAB to generate synthetic Data)• CalculateReadingsFromStatistics (written in R to generate synthetic data. • Inputs are number of readings, mean, standard deviation, inter- and intra-reader correlation

coefficients).• CalculateReadingsAnalytically

• Utility Functions: • PlotBlandAltman• GapBarplot• Blscatterplotfn

1818

Metric Purpose Source Language StatusSTAPLE To compute a probabilistic estimate

of the true segmentation and a measure of the performance level by each segmentation

FDA MATLAB testing

STAPLE Same as above ITK C++ implementedsoftSTAPLE

Extension of STAPLE to estimate performance from probabilistic segmentations

TBD TBD TBD

DICE Metric evaluation of spatial overlap ITK C++ implementedVote Probability map ITK C++ implemented

P-Map Probability map C. Meyer Perl TBD

Jaccard, Rand, DICE, etc.

Pixel-based comparisons Versus (Peter Bajcsy)

JAVA TBD

More?

1919

Drill-down on segmentation analysis activities

Update on Workflow Engine for the Compute Services• allows users to create their own workflows and facilitates sharing and re-

using of workflows.• has a good interface for capture of the provenance of data.• ability to work across different platforms (Linux, OSX, and Windows).• easy access to a geographically distributed set of data repositories,

computing resources, and workflow libraries. • robust graphical interface.• can operate on data stored in a variety of formats, locally and over the

internet (APIs, Web RESTful interfaces, SOAP, etc…).• directly interfaces to R, MATLAB, ImageJ, (or other viewers).• ability to create new components or wrap existing components from other

programs (e.g., C programs) for use within the workflow. • provides extensive documentation.• grid-based approaches to distributed computation.

2020

Supported b

y Taverna

Could also b

e done in

Taverna, b

ut alre

ady

supported in

Kepler

Taverna and Kepler…Two powerful suites for workflow management. However, Kepler improves Taverna by:

• grid-based approaches to distributed computation. • directly interfaces to MATLAB, ImageJ, (or other viewers).• ability to wrap existing components from other programs (e.g., C programs)

for use within the workflow. • provides extensive documentation.

2121

…go to demo

2222

Value proposition of QI-Bench• Efficiently collect and exploit evidence establishing

standards for optimized quantitative imaging:– Users want confidence in the read-outs– Pharma wants to use them as endpoints– Device/SW companies want to market products that produce them

without huge costs– Public wants to trust the decisions that they contribute to

• By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data

• Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders

2323

Summary:QI-Bench Contributions• We make it practical to increase the magnitude of data for increased

statistical significance. • We provide practical means to grapple with massive data sets.• We address the problem of efficient use of resources to assess limits of

generalizability. • We make formal specification accessible to diverse groups of experts that are

not skilled or interested in knowledge engineering. • We map both medical as well as technical domain expertise into

representations well suited to emerging capabilities of the semantic web. • We enable a mechanism to assess compliance with standards or

requirements within specific contexts for use.• We take a “toolbox” approach to statistical analysis. • We provide the capability in a manner which is accessible to varying levels of

collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access.

2424

QI-BenchStructure / Acknowledgements• Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette, Kjell Johnson, Jovanna

Danagoulian)

• Co-Investigators– Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer)– Stanford (David Paik)

• Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu)

• Collaborators / Colleagues / Idea Contributors– Georgetown (Baris Suzek)– FDA (Nick Petrick, Marios Gavrielides) – UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha)– Northwestern (Pat Mongkolwat)– UCLA (Grace Kim)– VUmc (Otto Hoekstra)

• Industry– Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner)– Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, …

• Coordinating Programs– RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)– Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien)

2525