please stand by… the webinar clinical validation of … · clinical validation of cancer...

CLINICAL VALIDATION OF CANCER BIOMARKER SIGNATURES USING ARRAY TECHNOLOGY

to login to Twitter and send tweets

Facebook login

if you need help

shows speaker bios

download slides and more info

LinkedIn login

shows slide window

Change the size of any window by dragging the lower left corner. Use controls in top right corner to close or maximize each window.

What each widget does:

opens the Ask a Question box

Twitter login (#ScienceWebinar)

search Wikipedia

shows the audio media player

PLEASE STAND BY… the webinar

will begin shortly…

Webinar Series Science

Sponsored by:

Participating Experts:

Paul “Mickey” Williams, Ph.D. SAIC-Frederick Frederick, MD

Timothy J. Triche, M.D.-Ph.D. University of Southern California Los Angeles, CA

Brought to you by the Science/AAAS Custom Publishing Office

Webinar Series Science

12 September, 2012

CLINICAL VALIDATION OF CANCER BIOMARKER SIGNATURES USING ARRAY TECHNOLOGY

Clinical Validation of Cancer Biomarker

Signatures using Array Technology

P. Mickey Williams, Ph.D. Director of the Molecular Characterization Laboratory

Clinical Research Directorate Frederick National Laboratory for Cancer Research

Frederick National Laboratory for Cancer Research

Definitions • Analytical Validity:

– Assay analytical performance and accuracy

• Sensitivity

• Specificity

• Reproducibility

• Precision

• Clinical Validity: – Accuracy of the assay to identify a clinical condition or outcome

• Positive and Negative Predictive Value

• Clinical Utility – Does the assay provide useful clinical information

– Does the assay improve clinical outcome of a patient

Frederick National Laboratory for Cancer Research

Often Published Array Studies Do Not Find Their Way Into Clinical Practice

• Biological heterogeneity confounds verification – Cellular, tumor, patient

• Assay analytical variability – Within assay, between assays

• Specimen variability – Tumor content, necrosis, inflammation

• Platform biases: not all assays are the same • Effect size: not only about p-value but also the

magnitude of difference between groups

A lot of “noise” that blurs marker and outcome correlation and validation

Frederick National Laboratory for Cancer Research

IOM Report: Evolution of Translational Omics: Lessons

Learned and the Path Forward

• http://www.nap.edu/catalog.php?record_id=13297

• Best practices for Translating Omic Tests into Clinical Use

Frederick National Laboratory for Cancer Research

Questions to Consider • What is the assay’s intended use?

– Critical to clearly state clinical intended use before beginning validation studies

– Be certain specimens used in assay validation are similar to the intended clinical specimens (should be similar to signature discovery specimens)

– How will assay results be reported?

• As a number? As a recurrence score? Positive/Negative?

• How will report be used for clinical action?

– How will the assay be used in the clinic?

• Integral assay (results used for patient management)

• Integrated assay (patients get assay as part of study but no direct clinical action results)

Frederick National Laboratory for Cancer Research

Hurdles for Validation of Gene Signatures

• Specimens of appropriate type are often hard to obtain, BUT critical to success

• Specimens from a single clinical study may add bias to results, ideally a second study provides the best specimens for assay validation – Assess possible bias introduced when only a portion of

specimens in a study are available (site bias, size bias, etc.)

• Multi-gene signatures lead to over-fitting of discovery data – Employ statistical tools to minimize

• Technical and reagent lot to lot variability

• Performance standards can be difficult to define

Frederick National Laboratory for Cancer Research

Some Suggestions For Success Prior to Validation

• Clearly state the clinical intended use

• Identify acceptable specimens and assess biases

• Consider the assay as a system, beginning with removal of the specimen from the patient and ending with an assay report to a clinician, the system must be well defined in SOPs

• Assemble a multi-disciplinary team: pathologist, laboratorian (CLIA accredited), technical staff, biologist/bioinformatician, oncologist

• Develop a well mapped plan for validation

• Lock down SOPs

• Lock down instrument performance specifications and instrument validation procedures

• If possible, find a second site

Frederick National Laboratory for Cancer Research

Some Suggestions to Consider: 2

• Attempt to find biological explanation for the genes and signature and their correlation with clinical outcome

• If assay is intended for use in clinical study or for implementation as a test for a CLIA laboratory: – Seriously consider an informal discussion with FDA sooner rather

than later

– If assay results will be used to manage a patient or stratify in a study OR if specimens used will be obtained for the sole purpose of the assay, I recommend speak with FDA

• If intending to charge for the assay, consult a business/marketing person familiar with the Dx industry

• Determine if other tests address similar outcomes – If yes, you will want to think carefully why your test will be better

and design clinical study to demonstrate that superiority

Frederick National Laboratory for Cancer Research

Validation Testing Locked Assay System

• Analytical validation testing assay report: – Sensitivity

– Specificity

– Repeatability

– Testing assay performance standards

– Is assay fit for purpose i.e. intended use and specimens

• Clinical validation (best done with new untested specimens) – Does the assay report accurately correlate with intended clinical

outcome

• May consider comparing across laboratories for comparability of results/report, this permits transparency and strong support for assay validity

Frederick National Laboratory for Cancer Research

Predictive Value • Positive Predictive Value =

The proportion for people with a positive test result who actually have the condition or outcome

• Negative Predictive Value = The proportion of people with a negative test result who do not

have the condition or outcome

Frederick National Laboratory for Cancer Research

Case Study LLMPP’s Lymphoma Assay

• Research discovery reported in 2000 Nature Alizadeh et al. – Two distinct molecular sub-classes of Diffuse Large B-Cell

Lymphoma identified by non-commercial microarray

– These sub-classes named GCB and ABC had significantly different prognosis and/or response to CHOP therapy

Frederick National Laboratory for Cancer Research

Unsupervised Clustering of DLBCL & Purified “Normal” B Cells

GCB

ABC

Effect Size of biological difference was sufficient to permit unsupervised identification of 2 classes

Frederick National Laboratory for Cancer Research

GCB/ABC Provides Independent Prognostic Value to IPI

ABC/GCB IPI Lo IPI & ABC/GCB

Frederick National Laboratory for Cancer Research

What Happened Between 2000 and 2012? • Subsequent to many experiments examining the genes of

ABC and GCB sub-classes, there is now an understanding of the underlying biology of these sub-classes – ABCs appear to have constitutive B-cell receptor signaling

– Many ABC tumors also exhibit NFkB signaling, ABC’s appear to be similar to activated B-cells found in the periphery

– GCBs appear similar to germinal center B cells found in lymph nodes

• Knowing these biological differences and results from multiple synthetic lethal experiments, relevant and drug-able targets have been identified for the poor prognostic ABC sub-class

Frederick National Laboratory for Cancer Research

What Happened Between 2000 and 2012? • Multiple platform switches have occurred

– Affymetrix custom LymphoChip

– Affymetrix U133 plus 2

– IHC

• A expression algorithm established

• LLMPP beginning a 2 PHASE study – Phase 1 Feasibility: A locked protocol and algorithm based on

Fresh Frozen Data will be employed for testing of 100 blinded FFPET specimens (matching fresh frozen section already tested)

– Algorithm tweaks will be made and assay will validated (analytically)

– Phase 2: Test set, including specimens collected outside of LLMPP will be tested with locked and analytical validated assay

Frederick National Laboratory for Cancer Research

Parting Comments • Think about the intended clinical use early and often

• Lock the entire assay system and capture protocols with SOPs

• Build a strong inter-disciplinary team

• Know the weak points of the assay system and build in quality checks in order to prevent an assay from reporting a false results

• If intended for clinical use engage a CLIA accredited laboratory and speak early with FDA

Frederick National Laboratory for Cancer Research

Clinical Assay Development Program • NCI has initiated the CADP to provide services needed to

support the clinical development of research-grade assays

• 8 CLIA accredited laboratories are part of Clinical Laboratory Network and available to assist with providing services

• Applications accepted 3 times/year

• Services include but are not limited to: – Clinical assay development/optimization

– Generation of locked documented protocols

– Assay performance and fit for purpose testing

– Statistical support

– Specimen collection

• www.cadp.nci.gov

Frederick National Laboratory for Cancer Research

Disclaimer • Views expressed are my own and not necessarily those of the

Frederick National Laboratory for Cancer Research-SAIC, National Cancer Institute, NIH, or DHHS

Frederick National Laboratory for Cancer Research

ACKNOWLEDGEMENTS

• Affiliation: Dr. P. Mickey Williams, Director Molecular Characterization and Clinical Assay Development Laboratory (MoCha), Clinical Research Directorate/MoCha, Frederick National Laboratory for Clinical Research, Frederick, Maryland, 21702.

• Funding: This project has been funded in whole or in part with federal funds from the National Cancer Institute, National Institutes of Health, under Contract No. HHSN261200800001E. The content of this publication does not necessarily reflect the views or policies of the Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.

Sponsored by:

Participating Experts:

Paul “Mickey” Williams, Ph.D. SAIC-Frederick Frederick, MD

Timothy J. Triche, M.D.-Ph.D. University of Southern California Los Angeles, CA

Brought to you by the Science/AAAS Custom Publishing Office

Webinar Series Science

12 September, 2012

CLINICAL VALIDATION OF CANCER BIOMARKER SIGNATURES USING ARRAY TECHNOLOGY

Microarrays for the creation of diagnostic and prognostic

biomarker profiles

Timothy J. Triche, MD, PhD Children’s Hospital Los Angeles USC Keck School of Medicine

RNA Seq vs. Microarrays Parameter Microarrays RNA Seq

Starting material amount Nanograms, More, typically >100 ng

Starting material type Fresh, FFPE (>20 yrs) Fresh preferable, FFPE ?

Signal Semiquantitative Digital

Specificity Cross hybridization Ambiguous mapping

Genome Coverage Limited to known content Unlimited (potentially)

Isoform detection Limited Unambiguous

Sequence variants No Yes

Interpretation Routine Challenging

Cost <$1,000 COGs <$1,000, interpretation ?

Clinical utility FDA acceptable Unproven

Discovery potential Limited to content Unlimited

Research Potential Limited, utilitarian Greatest potential

Coding Genes Account for Just 1-2% of the Genome:

-protein encoding

?

Non-coding Genes Account for Most Transcription from the Genome:

miRNA sprcRNA

sncRNA

PASR ncRNA

Antisense

Introns

lncRNA

-protein encoding

Transcriptionally active

Splice Variants

Large-Scale Transcriptional Activity in Chromosomes 21 and 22 Philipp Kapranov, Simon E. Cawley, Jorg Drenkow, Stefan Bekiranov, Robert L. Strausberg, Stephen P. A. Fodor, and Thomas R. Gingeras. Science 3 May 2002

The majority of total nuclear-encoded non-ribosomal RNA in a human cell is 'dark matter' un-annotated RNA

Tissue RNA Distribution Tumor RNA Distribution

Kapranov, P, et al: BMC Biology, December, 2010

Non-coding RNA networks underlying cognitive disorders across the lifespan

Summary: ncRNAs are numerous, diverse, and essential for brain development and cognition

Irfan A.Qureshi and MarkF.Mehler, Trends in Molecular Medicine, 684

Human Exon Array: Derived from ENCODE RNA expression data

• 5 million features on array • 1.4 million RNA transcripts • 0.2 million mRNA exons • 0.2 million intronic/anti-sense transcripts • ~ 1 million non-coding RNA transcripts!

Lessons Learned #1:

Non-coding RNA is highly relevant to biologic function and produces better

biomarkers

Coding vs Non-coding RNA Expression in Ewing’s Sarcoma

•

•

AK057037

ENPP3

Non-coding RNA

Gene

= non-coding RNA

= annotated genes

Lessons Learned #2:

Microarrays can deliver data that is very comparable to newer methods like RNA

seq, at a lower cost and faster.

Relative RNA Transcript Abundance: 3 platform comparison

RNA Seq (long read) HuEx (microarray) RNA Seq (short read)

Lessons Learned #3:

Exon-level microarrays can detect clinically relevant splice variants

A - A 3 A 7 ERMS EFT OS Wilms NB

Tumor-Specific Gene Splice Variants: ASS1 in Translocated ARMS

Absent in ERMS on U133 data Splice Variant Seen on HuEx Array data

ASS1: Alternative Splicing Database

Lessons Learned #4:

Exon-level microarrays that span the genome can detect novel RNA

transcripts

Results

N.B.: Underlined transcripts chosen for 24 feature ncRNA meta-feature

32 exon ncRNA meta-feature composition

Candidate PSR Runs Identified (corresponding to prognosis)

RNASeq of 600 kb Transcript

But Aceview reveals a 230 kb transcript

RefSeq

Aceview

No evidence of TSS or discontinuous transcription over entire 600 kb region

Summary

• Gene expression arrays that include coding and non-coding RNA transcripts can generate diagnostic biomarker profiles that outperform those composed of coding genes alone.

• Most RNA transcripts found by RNA Seq are detected by whole-transcriptome microarrays at comparable relative expression levels.

• When cost and ease of analysis are considered, gene expression microarrays remain an attractive means of assessing global RNA expression in a clinical setting.

Sponsored by:

Participating Experts:

Paul “Mickey” Williams, Ph.D. SAIC-Frederick Frederick, MD

Timothy J. Triche, M.D.-Ph.D. University of Southern California Los Angeles, CA

Brought to you by the Science/AAAS Custom Publishing Office

Webinar Series Science

12 September, 2012

CLINICAL VALIDATION OF CANCER BIOMARKER SIGNATURES USING ARRAY TECHNOLOGY

To submit your questions, type them into the text box

and click .

Look out for more webinars in the series at: webinar.sciencemag.org

For related information on this webinar topic, go to: www.affymetrix.com/genomics

To provide feedback on this webinar, please e-mail your comments to webinar@aaas.org

Sponsored by:

Brought to you by the Science/AAAS Custom Publishing Office Brought to you by the Science/AAAS Custom Publishing Office

Webinar Series Science

12 September, 2012

CLINICAL VALIDATION OF CANCER BIOMARKER SIGNATURES USING ARRAY TECHNOLOGY