carrie bennette on behalf of andrew vickers
DESCRIPTION
How do we know whether a marker or model is any good? A discussion of some simple decision analytic methods. Carrie Bennette on behalf of Andrew Vickers Pharmaceutical Outcomes Research and Policy Program (PORPP) University of Washington. Overview of talk. - PowerPoint PPT PresentationTRANSCRIPT
How do we know whether a marker or model is any good?
A discussion of some simple decision analytic methods
Carrie Bennetteon behalf of Andrew Vickers
Pharmaceutical Outcomes Research and Policy Program (PORPP)
University of Washington
Overview of talk
• Marker research in cancer: state of the science
• Traditional statistical methods for evaluating predictions
• Decision analytic approaches
Overview of talk
• Marker research in cancer: state of the science
• Traditional statistical methods for evaluating predictions
• Decision analytic approaches
A combination of common and minor variations in five regions of DNA can help predict a man’s risk of getting prostate cancer, researchers reported Wednesday. A company formed by researchers at Wake Forest University School of Medicine is expected to make the test available in a few months …. It should cost less than $300. This is, some medical experts say, a first taste of what is expected to be a revolution in medical prognostication
SNP panel
• Predictive accuracy of SNP panel (as calculated by AV): 0.57
• Predictive accuracy of single PSA in middle age: 0.75
• Doesn’t add to standard predictors (Nam et al.)
Systematic review of molecular markers in cancer
• 129 papers published in 2005 and 2006 eligible for analysis
• More markers than papers
• 97% included inference statistics
• 36% included marker in a multivariable model
• 11% measured predictive accuracy
• 0 used decision analytic techniques
Overview of talk
• Marker research in cancer: state of the science
• Traditional statistical methods for evaluating predictions
• Decision analytic approaches
Example: Binary test for cancer on biopsy
• Patients with high PSA are referred to biopsy
• But most patients with high PSA don’t have prostate cancer
• Could a second marker help?
• Study of biopsy cohort: 26% had cancer
– Assess presence of two markers
Traditional biostatistical metrics
Sensitivity Specificity PPV NPV LR+ LR- AUC (Youden)
Brier(mean squared error)
Test A 91% 40% 35% 92% 1.52 0.23 0.65 0.47
Test B 51% 78% 45% 82% 2.32 0.63 0.64 0.29
Which test is best?
• Sensitivity / specificity insufficient to determine which test should be used:
– “Depends on whether sensitivity or specificity is more important”
Conclusion about traditional metrics
• Traditional biostatistical techniques for evaluating models, markers and tests do not incorporate clinical consequences
• Accordingly, they cannot inform clinical practice
Overview of talk
• Marker research in cancer: state of the science
• Traditional statistical methods for evaluating predictions
• Decision analytic approaches
Threshold probability
• Predicted probability of disease is p=
• Define a threshold probability of disease as pt
• Patient accepts treatment if p= ≥ pt
• pt describes how patients values relative harm of false positive and false negative
Decision theory
“I would biopsy a man if his risk of prostate cancer was 20% or more, that is, I would conduct no more than 5 biopsies to find one cancer. I consider the harms associated with delaying the diagnosis of prostate cancer to be four times worse than the harms, risks and inconvenience of biopsy.”
Treat: Sens. Spec. Prev. Net benefit
Test A 91% 40% 26%91% × 26% -
(1 – 40%) × (1 – 26%) × (0.2 ÷ 0.8) = 0.1256
Test B 51% 78% 26%51% × 26% -
(1 – 78%) × (1 – 26%) × (0.2 ÷ 0.8) = 0.0919
Everyone 100% 0% 26%100% × 26% -
(1 – 0%) × (1 – 26%) × (0.2 ÷ 0.8) = 0.075
No-one 0% 100% 26%0% × 26% -
(1 – 100%) × (1 – 26%) × (0.2 ÷ 0.8) = 0
Worked example at pt of 20%
Net benefit has simple clinical interpretation
• Net benefit of 0.126 at pt of 20%
• Using the model is the equivalent of a strategy that led to 126 patients per 1000 with cancer being biopsied with no unnecessary biopsies
Net benefit has simple clinical interpretation
• Difference between model and treat all at pt of 20%.
– 0.051
• Divide by weighting 0.051/ 0.25 = 0.204
– 204 fewer false positives per 1000 patients for equal number of true positives
– E.g. 204 fewer patients undergoing biopsy without missing any cancers
Decision curve analysis
4. Vary pt over an appropriate range
Vickers & Elkin Med Decis Making 2006;26:565–574
1. Select a pt 2. Positive test defined as 3. Calculate “Clinical Net Benefit” as:
tppˆ
Decision analysis
All markers
PSA
Free, Total PSA
Biopsy all
Biopsy none
Vickers JCO 2009
Gallina vs. Partin
AUC 0.81 AUC 0.78
P=0.02
Decision curve analysis
Conclusion
• Huge number of markers proposed
• Evidence base is very weak for most
• Traditional biostatistical methods do not assess clinical value of a marker
• Simple decision analytic methods can distinguish potentially useful from useless models and markers