aug2015 zivana tezak analytical validation

Analytical Evaluation Standards for NGS Zivana Tezak, PhD OIR/ CDRH/FDA August 28, 2015 NIST, Gaithersburg, MD Genome in a Bottle Consortium-- August 2015 Workshop

Upload: genomeinabottle

Post on 17-Jan-2017



Health & Medicine

0 download


Page 1: Aug2015 zivana tezak analytical validation

Analytical Evaluation Standards for NGS

Zivana Tezak, PhDOIR/ CDRH/FDA

August 28, 2015NIST, Gaithersburg, MD

Genome in a Bottle Consortium--August 2015 Workshop

Page 2: Aug2015 zivana tezak analytical validation


Disclaimer• NGS is raising new policy / regulatory issues; thoughts

presented here are preliminary and do not represent proposed or finalized FDA policy – these are just preliminary ideas or approaches we are considering

Background (“Vision”)• February 2015 - “Optimizing FDA’s Regulatory Oversight

of Next Generation Sequencing Diagnostic Tests”

Page 3: Aug2015 zivana tezak analytical validation

Precision Medicine Initiative - FDA

Vision: Implement new regulatory policies to promote research and accelerate the translation of precision medicine technologies into treatments that benefit patients.•Near Term: Implement standards and shared resources that will enable the development of knowledge for research and patient decision making•Longer Term: Implement standards-based regulation of NGS diagnostic tests that will ensure that the tests patients receive provide accurate, reproducible, and meaningful results


Page 4: Aug2015 zivana tezak analytical validation

FDA – Modernizing Regulation for NGS tests

Adopt a flexible, dynamic regulatory system(knowledgebase /

databases)Develop and implement standards

to assure quality

Develop cloud-based open-source tools

to help test developers meet


Analytical evaluation


Clinical significance

Page 5: Aug2015 zivana tezak analytical validation

Build growing suite of standards

– Performance standard documentse.g., ACMG, CLSI, NYS DOH, CAP

– Material standards e.g., NIST/GIAB


Page 6: Aug2015 zivana tezak analytical validation

General Overview / Framework

• Analytical: Performance Standards as a Regulatory Tool for NGS Testing – Analytical performance standard elements – Uses / areas in need of separate analytical standards – Overall architecture / workflow of clinical NGS test (from sample to

EHR)– Example of a possible performance standard (single use)– Existing guidelines and standards for NGS tests– Stakeholder communities building some of the standards – Cloud based, open source bioinformatics tools repository to facilitate

standard development


Page 7: Aug2015 zivana tezak analytical validation

Performance standard elements• Intended use of the test (e.g., carrier screening, diagnosis of

inherited disease, cancer risk prediction)• Transparency • Test description (methodology, instrumentation, reagents,

software, etc)• Metrics (quality scores, read depth, sensitivity/specificity, etc)

– what to measure• Protocols and studies to perform (specimen type, sample

numbers, comparator methods, etc) – approaches on how to measure analytical performance

• Acceptance criteria, thresholds for measurements• Performance data / result presentation


Page 8: Aug2015 zivana tezak analytical validation

Major areas in need of separate standardsInitial assessments; will develop and change. Where to start?

– Targeted clinical assays • Germline targeted assays • Somatic targeted assays (cancer panels)

– Exome-based clinical assays (heritable, rare undiagnosed disease)– WGS-based clinical assays

• For germline uses• For somatic uses

– Method or analytic quality-based standards• for methodology, that would include different classes of variants, genomic regions,

allele ratios, different technology• for platform (use of standard materials, panels) • for test building blocks (e.g., preanalytical, library, software, quality system

approach, etc)– Run quality standards – including computational solutions (WGS only?)– Terminology (language agreement and definitions)– Agreement on appropriate comparators and “gold standards”– Material standards, reference samples, disease-specific sample sets, and

controls– Data formats and interchangeability / interoperability 8

By intended use

Page 9: Aug2015 zivana tezak analytical validation

Overall architecture of NGS test• High-level NGS test steps, from patient sample through variant

call and interpretation: – Preanalytical (from tissue -> DNA); includes specimen quality

assessment, DNA preparation, library preparation– Sequencing (from DNA -> raw sequence data) – Sequence bioinformatics (from raw sequence data -> VCF); includes

pipeline from aligning, filtering, read mapping, annotation, contamination, through call determination

– Interpretation (including knowledgebases) – Reporting – EHR and archival

• Will differ for different platforms• Overall standards architecture?


Page 10: Aug2015 zivana tezak analytical validation

NGS Workflow Standards and Tools

Examples of Intended Uses

Hereditary Diseases (targeted panels)

Hereditary Diseases (exomes)

Cancer panels Methodology



Data Processing (informatics)

Knowledgebase (databases)



NGS Workflow

Page 11: Aug2015 zivana tezak analytical validation

Performance standard example

• Single use• Possible analytical approach • Heritable monogenic disorders

– Diagnosis– Targeted panels– Exomes?

• Additional examples - – Special controls for NGS platform and autosomal recessive carrier



Need flexibility for evolving technology and knowledge!

Page 12: Aug2015 zivana tezak analytical validation

Existing standards and guidelines

• Currently existing or ongoing standard development, how they correspond to FDA needs:– Material standards – e.g., NIST, sample panels– Methodological standards – e.g., NYSDOH, CAP, MolDx, GA4GH, Nex-

StoCT, ACMG, HGVS, CPIC, CLSI • What can be used, for what purpose• Identify gaps • Questions:

– Who should fill identified gaps – What is acceptable timeline?

• Need built-in flexibility12

Page 13: Aug2015 zivana tezak analytical validation

Stakeholder communities

• Stakeholders involved for each area• Ways to work together • Possible conveners

– SEQC Consortium– NIST / GIAB Consortium– CERSI– CDC– CLSI– GA4GH– others?


Page 14: Aug2015 zivana tezak analytical validation

Bioinformatics - General Landscape

In the NGS-based assay space -

•Some labs offer the test by sequencing the samples but outsourcing bioinformatics analysis to other companies

•Some labs require support from 3rd party companies to host their bioinformatics pipeline, but the labs perform the analysis

•Some labs offer the test by sequencing the samples AND have deep bioinformatics and analytic capability





Page 15: Aug2015 zivana tezak analytical validation


CommunityNGS-Based Test Developers (large and small), NIST, FDA Scientists, Standards Bodies, Academic Centers, Patient-Facing Providers, Consortiums

Analysis and Pipeline(s)

Security and Privacy HIPAA/HITECH, CAP, ISO27001Uniquely identified and immutable dataVersion-controlled applications

Reference Data


Quality Threshold




Courtesy of Taha Kass-Hout, FDA Chief Health Informatics Officer

Advancing the accuracy and reproducibility of NGS•Crowd-sourced, cloud-based platform•Will provide tools and open access resources•Will allow the community to test, pilot, and validate approaches to NGS

Page 16: Aug2015 zivana tezak analytical validation


Development and Use of Analytical Standards

• Current practices• Technical knowledge


NIST Others


Regulatory proposalCompanies

NGS labs

Stakeholder input Discussion paper

Public workshop

Comment analysis

Standards bodiesProfessional


Page 17: Aug2015 zivana tezak analytical validation

Success of Precision Medicine Requires:

• Safe and accurate diagnostic tests that reliably identify individual variation

• Learning health systems that enable researchers and clinicians to learn from and inform the patient experience

• Development of targeted therapies that are more efficacious or have less deleterious side effects for specific individuals

• Updated research and regulatory policies that catalyze the development of new treatments while protecting patients