steps on the road to predictive oncology richard simon, d.sc. chief, biometric research branch...
TRANSCRIPT
Steps on the Road to Predictive Oncology
Richard Simon, D.Sc.Chief, Biometric Research Branch
National Cancer Institutehttp://brb.nci.nih.gov
Biometric Research Branch Website
brb.nci.nih.gov
• Powerpoint presentations
• Reprints
• BRB-ArrayTools software
• Sample Size Planning – Clinical Trials using predictive biomarkers
• Many cancer treatments benefit only a minority of patients to whom they are administered– Some early stage patients don’t require systemic rx– Some who do, don’t benefit from a specific regimen
• Being able to predict which patients are likely to benefit would – save patients from unnecessary toxicity, and enhance
their chance of receiving a drug that helps them– Help control medical costs – Improve the success rate of clinical drug development
• “Hypertension is not one single entity, neither is schizophrenia. It is likely that we will find 10 if we are lucky, or 50, if we are not very lucky, different disorders masquerading under the umbrella of hypertension. I don’t see how once we have that knowledge, we are not going to use it to genotype individuals and try to tailor therapies, because if they are that different, then they’re likely fundamentally … different problems…”– George Poste
Biomarkers
• Prognostic– Measured before treatment to indicate long-term outcome for
patients untreated or receiving standard treatment• Single arm study of patients receiving a particular rx can identify
patients with good prognosis on that rx– Those patients may not benefit from that rx but they don’t need
additional rx
• Predictive– Measured before treatment to identify who will benefit from a
particular treatment• Single arm study with response endpoint• RCT with survival or dfs endpoint
Prognostic and Predictive Biomarkers in Oncology
• Single gene or protein measurement– ER expression– HER2 amplification– KRAS mutation– Usually related to putative molecular target
• Scalar index or classifier that summarizes contributions of multiple genes– Empirically determined based on selecting
genes with expression correlated to outcome
Prognostic Factors in Oncology
• Most prognostic factors are not used because they are not therapeutically relevant
• Most prognostic factor studies do not have a clear medical objective– They use a convenience sample of patients
for whom tissue is available. – Generally the patients are too heterogeneous
to support therapeutically relevant conclusions
Prognostic Biomarkers Can be Therapeutically Relevant
• <10% of node negative ER+ breast cancer patients require or benefit from the cytotoxic chemotherapy that they receive
• OncotypeDx– 21 gene assay
Predictive Biomarkers
• In the past often studied as un-focused post-hoc subset analyses of RCTs.– Numerous subsets examined– Same data used to define subsets for analysis
and for comparing treatments within subsets– No control of type I error
• Statisticians have taught physicians not to trust subset analysis unless the overall treatment effect is significant– This was good advice for post-hoc data
dredging subset analysis– For many molecularly targeted cancer
treatments being developed, the subset analysis will be an essential component of the primary analysis and analysis of the subsets will not be contingent on demonstrating that the overall effect is significant
Prospective Co-Development of Drugs and Companion Diagnostics
1. Develop a completely specified genomic classifier of the patients likely to benefit from a new drug
2. Establish analytical validity of the classifier• Reproducibility & robustness
3. Use the completely specified classifier to design and analyze a new clinical trial to evaluate effectiveness of the new treatment and it’s relationship to the classifier with a pre-defined analysis plan that preserves the overall type-I error of the study.
Guiding Principle
• The data used to develop the classifier should be distinct from the data used to test hypotheses about treatment effect in subsets determined by the classifier– Developmental studies can be exploratory– Studies on which treatment effectiveness
claims are to be based should be definitive studies that test a treatment hypothesis in a patient population completely pre-specified by the classifier
New Drug Developmental Strategy I
• Restrict entry to the phase III trial based on the binary predictive classifier, i.e. targeted design
Using phase II data, develop predictor of response to new drugDevelop Predictor of Response to New Drug
Patient Predicted Responsive
New Drug Control
Patient Predicted Non-Responsive
Off Study
Applicability of Design I
• Primarily for settings where the classifier is based on a single gene whose protein product is the target of the drug and the biology is well understood– eg Herceptin
• With substantial biological basis for the classifier, it may be unacceptable ethically to expose classifier negative patients to the new drug
Evaluating the Efficiency of Strategy (I)
• Simon R and Maitnourim A. Evaluating the efficiency of targeted designs for randomized clinical trials. Clinical Cancer Research 10:6759-63, 2004; Correction and supplement 12:3229, 2006
• Maitnourim A and Simon R. On the efficiency of targeted clinical trials. Statistics in Medicine 24:329-339, 2005
• Relative efficiency of targeted design depends on – proportion of patients test positive– effectiveness of new drug (compared to control) for
test negative patients
• When less than half of patients are test positive and the drug has little or no benefit for test negative patients, the targeted design requires dramatically fewer randomized patients
• The targeted design may require fewer or more screened patients than the standard design
TrastuzumabHerceptin
• Metastatic breast cancer• 234 randomized patients per arm• 90% power for 13.5% improvement in 1-year
survival over 67% baseline at 2-sided .05 level• If benefit were limited to the 25% test + patients,
overall improvement in survival would have been 3.375%– 4025 patients/arm would have been required
Web Based Software for Comparing Sample Size
Requirements
• http://brb.nci.nih.gov
Developmental Strategy (II)
Develop Predictor of Response to New Rx
Predicted Non-responsive to New Rx
Predicted ResponsiveTo New Rx
ControlNew RX Control
New RX
Developmental Strategy (II)
• Do not use the test to restrict eligibility, but to structure a prospective analysis plan
• Having a prospective analysis plan is essential• “Stratifying” (balancing) the randomization is useful to
ensure that all randomized patients have tissue available but is not a substitute for a prospective analysis plan
• The purpose of the study is to evaluate the new treatment overall and for the pre-defined subsets; not to modify or refine the classifier
• The purpose is not to demonstrate that repeating the classifier development process on independent data results in the same classifier
Analysis Plan B
(Limited confidence in test)
• Compare the new drug to the control overall for all patients ignoring the classifier.– If poverall 0.03 claim effectiveness for the eligible
population as a whole
• Otherwise perform a single subset analysis evaluating the new drug in the classifier + patients– If psubset 0.02 claim effectiveness for the classifier +
patients.
Analysis Plan C
• Test for difference (interaction) between treatment effect in test positive patients and treatment effect in test negative patients
• If interaction is significant at level int then compare treatments separately for test positive patients and test negative patients
• Otherwise, compare treatments overall
DNA Microarray Technology
• Powerful tool for understanding mechanisms and enabling predictive medicine
• Challenges the ability of biomedical scientists to analyze data
• Challenges statisticians with new problems for which existing analysis paradigms are often inapplicable
• Excessive hype and skepticism
• Good microarray studies have clear objectives, but not generally gene specific mechanistic hypotheses
• Design and analysis methods should be tailored to study objectives
Class Prediction
• Predict which tumors will respond to a particular treatment
• Predict survival or relapse-free survival risk group
Class Prediction ≠ Class ComparisonPrediction is not Inference
• The criteria for gene selection for class prediction and for class comparison are different– For class comparison false discovery rate is important– For class prediction, predictive accuracy is important
• Most statistical methods were not developed for p>>n prediction problems
Validating a Predictive Classifier
• Goodness of fit is no evidence of prediction accuracy for independent data
• Demonstrating statistical significance of prognostic factors is not the same as demonstrating predictive accuracy
• Demonstrating stability of selected genes is not demonstrating predictive accuracy of a model for independent data
Types of Validation for Prognostic and Predictive Biomarkers
• Analytical validation– When there is a gold standard
• Sensitivity, specificity
– No gold standard• Reproducibility and robustness
• Clinical validation– Does the biomarker predict what it’s supposed to
predict for independent data
• Clinical utility– Does use of the biomarker result in patient benefit– Depends on available treatments and practice
standards
Internal Clinical Validation of a Predictive Classifier
• Split sample validation– Training-set
• Used to select features, select model type, fit all parameters including cut-off thresholds and tuning parameters
– Test set• Count errors for single completely pre-specified model
• Cross-validation– Omit one sample– Build completely specified classifier from scratch in the training
set of n-1 samples– Classify the omitted sample– Repeat n times– Total number of classification errors
• Cross validation is only valid if the test set is not used in any way in the development of the model. Using the complete set of samples to select genes violates this assumption and invalidates cross-validation
• The cross-validated estimate of misclassification error is an estimate of the prediction error for model fit using specified algorithm to full dataset
Prediction on Simulated Null Data
Generation of Gene Expression Profiles
• 14 specimens (Pi is the expression profile for specimen i)
• Log-ratio measurements on 6000 genes
• Pi ~ MVN(0, I6000)
• Can we distinguish between the first 7 specimens (Class 1) and the last 7 (Class 2)?
Prediction Method
• Linear classifier based on compound covariate built from the log-ratios of the 10 most differentially expressed genes.
Number of misclassifications
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Pro
po
rtio
n o
f sim
ula
ted
da
ta s
ets
0.00
0.05
0.10
0.90
0.95
1.00
Cross-validation: none (resubstitution method)Cross-validation: after gene selectionCross-validation: prior to gene selection
Evaluating a Classifier
• “Prediction is difficult, especially the future.”– Neils Bohr
Comparison of Internal Validation MethodsMolinaro, Pfiffer & Simon
• For small sample sizes, LOOCV is much less biased than split-sample validation
• For small sample sizes, LOOCV is preferable to 10-fold, 5-fold cross-validation or repeated k-fold versions
• For moderate sample sizes, 10-fold is preferable to LOOCV
• Some claims for bootstrap resampling for estimating prediction error are not valid for p>>n problems
Sample Size Planning References
• K Dobbin, R Simon. Sample size determination in microarray experiments for class comparison and prognostic classification. Biostatistics 6:27, 2005
• K Dobbin, R Simon. Sample size planning for developing classifiers using high dimensional DNA microarray data. Biostatistics 8:101, 2007
• K Dobbin, Y Zhao, R Simon. How large a training set is needed to develop a classifier for microarray data? Clinical Cancer Res 14:108, 2008
Sample Size Planning for Classifier Development
• The expected value (over training sets) of the probability of correct classification PCC(n) should be within of the maximum achievable PCC()
Probability Model• Two classes• Log expression MVN in each
class – Mean vector and - for the
two classes– Common covariance matrix
– If classes are equi-prevalent
1( ) 'PCC
1.0 1.2 1.4 1.6 1.8 2.0
40
60
80
100
2 delta/sigma
Sam
ple
siz
e
gamma=0.05gamma=0.10
Sample size as a function of effect size (log-base 2 fold-change between classes divided by
standard deviation). Two different tolerances shown, . Each class is equally represented in the population. 22000 genes on an array.
BRB-ArrayTools
• Architect – R Simon• Developer – Emmes Corporation
• Contains wide range of analysis tools that I like• Designed for use by biomedical scientists• Imports data from all gene expression and copy-number platforms
– Automated import of data from NCBI Gene Expression Omnibus
• Highly computationally efficient• Extensive annotations for identified genes• Integrated analysis of expression data & copy number data• Utilizes some Bioconductor packages
– SAM in Fortran– Almost-RMA
Predictive Classifiers in BRB-ArrayTools
• Classifiers– Diagonal linear discriminant– Compound covariate – Bayesian compound covariate– Support vector machine with inner
product kernel– K-nearest neighbor– Nearest centroid– Shrunken centroid (PAM)– Random forest– Tree of binary classifiers for k-
classes• Survival risk-group
– Supervised pc’s– With clinical covariates– Cross-validated K-M curves
• Predict quantitative trait– LARS, LASSO
• Feature selection options– Univariate t/F statistic– Hierarchical random variance
model– Fold effect– Univariate classification power– Recursive feature elimination– Top-scoring pairs
• Validation methods– Split-sample– LOOCV– Repeated k-fold CV– .632+ bootstrap
• Permutational statistical significance
Cross-validated Kaplan-Meier curves for risk groups using 50th percentile cut-off
GENEMODEL
COVARIATESMODEL
COMBINEDMODEL
DIS
TA
NT
EV
EN
T F
RE
E S
UR
VIV
AL
DIS
TA
NT
EV
EN
T F
RE
E S
UR
VIV
AL
DIS
TA
NT
EV
EN
T F
RE
E S
UR
VIV
AL
BRB-ArrayToolsJuly 2008
• 8934 Registered users • 68 Countries• 616 Published citations
• Registered users– 4655 in US
• 898 at NIH– 387 at NCI
• 2994 US EDU• 1161 US Gov (non NIH)
– 4279 Non US
Countries With Most BRB ArrayTools Registered Users
• Germany 292• France 289• Canada 287• UK 278• Italy 250• China 241• Netherlands 240• Taiwan 222• Korea 192• Japan 187• Spain 168
• Australia 155• India 139• Belgium 103• New Zeland 63• Brazil 54• Singapore 53• Denmark 52• Sweden 50• Israel 45
Conclusions
• New technology provides important opportunities to identify which patients require systemic therapy and which are most likely to benefit from a specified treatment– Preforming the appropriate clinical trials and having
tissue available is rate limiting• Targeting treatment can provide
– Patient benefit– Economic benefit for society– Improved chance of success for new drug
development• Not necessarily simpler or less expensive development
Conclusions
• Achieving the potential of new technology requires paradigm changes in methods of “correlative science.”
• Accelerating progress in discovering and developing effective therapeutics requires increased emphasis on trans-disciplinary training of laboratory, clinical and statistical/computational scientists
Acknowledgements• BRB Senior Staff
– Kevin Dobbin– Boris Freidlin– Ed Korn– Lisa McShane– Joanna Shih– George Wright– Yingdong Zhao,
• Post-docs– Alain Dupuy– Wenyu Jiang– Aboubakar Maitournam– Annette Molinaro– Michael Radmacher
• BRB-ArrayTools Development Team