A Risk Prediction Model for Recurrent Events in Chronic

Coronary Heart Disease: The Heart and Soul Study

Ivy Ku, Eric Vittinghoff, Kirsten Bibbins-Domingo, Michael Shlipak, Mary Whooley

January 14, 2011

Background and Significance

• 1 in 3 Americans live with cardiovascular disease

• With advances in therapies, patients live longer with CHD

• Prognosis varies widely

• Risk stratification integral to patient management

Risk Prediction is Useful

Risk Prediction in Primary Prevention

• 10-year risk of incident coronary heart disease (CHD)

• Guides cholesterol and BP treatment in primary prevention

Risk Prediction in ACS

Predictors of worse outcomes in stable CHD

• Biomarkers: CRP, BNP, hs-troponin

Risk Prediction in Stable CHD

• Clinically useful, up to date, simple, integrated risk scores lacking

• HERS, LIPID, Framingham severe limitations

• Furthermore, long-term risk in CHD has not been well-characterized and quantified

• CHF not included in CHD risk prediction

Project Aims

• To develop a clinical prediction model and point score for 5-year risk of recurrent CV events in stable CHD

• To quantify and categorize the range of long-term risk in stable CHD

Methods

• The Heart and Soul Study– Cohort study of 1024 subjects with stable

CHD enrolled 2000-02– Effect of psychosocial factors on prognosis

in stable CHD– Thorough phenotyping of baseline

condition, biomarkers, echo, stress– Mean 6 years follow-up, > 400 CV events

Population

• SF bay area

• VA, UCSF, CHN clinics

• Inclusion: hx MI, revascularization, angiographic CAD, abnormal stress test

• Exclusion: MI within 6 mo, unable to walk 1 block, moving away within 3 years

Methods

• 2 Cox models– Dichotomized predictors– Continuous predictors

• Composite outcome: time to MI, CVA, CHF hospitalization, or CV death

• Use baseline survival function, relative hazards to calculate 5-year risk

Coding of Predictors

• Selected functional form of continuous predictors using AIC – categorical (quantiles, clinical cutpoints)– linear– 3, 4, 5 knot restricted cubic splines

• Steyerberg recommends doing this a priori if possible, to avoid over-fitting

• Cross-validation can also be used

Model selection

• Need to maximize the signal without over-fitting• Three main strategies:

1. Outcome-free data reduction: use the literature, expert opinion, practical considerations to eliminate candidate predictors without looking at the outcome

2. Parsimony: select highly significant predictors

3. Cross-validation (CV): mimics external validation

Our implementation

• Outcome free data reduction: eliminated 18 of 36 candidate predictors on the basis of expert judgment, practical considerations

• Parsimony: cut 4 more using backward selection• Cross-validation: 10-fold CV of C-index for ~1,000

candidate models• Final decision between top candidates again

considered clinical convenience and face validity

How cross-validation works• Divide sample into 5-10 subsets

• For each subset:– set aside, fit model to remaining subsets– calculate predictions for set-aside subset

• Estimate prediction error using quasi- external predictions for all observations

• Repeat ~20 times and average results– repetition needed to reduce noise

C-index

• A measure of model discrimination • Extension of C-statistic, area under ROC

curve to survival models• Estimates probability that in a randomly

selected pair of observations, the earlier failure has the higher predicted risk

• Naïve C-index is optimistic; cross-validation reduces the optimism

Selecting Point Score Model

• Cross-validation involves five steps for each candidate point score model:1. fit model using binary predictors only

2. round coefficients to obtain point scores

3. refit model using calculated point scores as sole (continuous) predictor

4. save predictions from the refitted model

5. use predictions to calculate CV C-index

Shrinkage using calibration slope• Cross-validation to get calibration slope:

– calculate xb for omitted subsets– re-fit model using xb as the sole predictor– coefficient for xb <1.0 signals over-fitting

• Use slope to improve calibration– shrink coefficients by calibration slope (i.e.,

the coefficient for xb in the refitted model)– pulls in extreme high and low predictions– does not affect discrimination

Model Performance

• Discrimination: C-index

• Net reclassification improvement (NRI)– continuous vs point score models– continuous model vs Framingham

• Calibration: goodness-of-fit test, visual inspection, calibration slope

External Model Validation

• Cross-validation is strictly internal– reduces over-fitting– but does not protect against predictor

effects that differ across populations

• Plan external validation in separate cohort– recommended by Altman and Royston,

often demanded by reviewers

Results

Included vs ExcludedIncluded (n=912) Excluded (n=108) P value

Outcome, % 27% 32% 0.23

Follow-up time, yrs 5.8 (5.6-5.9) 5.6 (5.0-6.2) 0.47

Age, yrs 67 ± 11 68 ± 11 0.36

History of CHF 17% 22% 0.27

smoker 20% 18% 0.69

LVEF, % 62 ± 10 61 ± 9 0.60

UACR, mg/g 8.7 (5.1-17.9) 7.9 (2.2-11.6) 0.18

BNP, pg/mL 173 (74-452) 222 (89-532) 0.20

Included vs Excluded

Included (n=912) Excluded (n=108) P value between HR

HR (CI) P HR (CI) P

Age 1.04 (1.03-1.06) < 0.001 1.01 (0.98-1.05) 0.43 0.10

Hx CHF 2.27 (1.72-3.0) < 0.001 2.3 (1.12-4.72) 0.02 0.90

smoker 1.17 (0.86-1.59) 0.32 1.66 (0.75-3.67) 0.21 0.39

LVEF 2.74 (2.03-3.71) < 0.001 4.96 (2.15-11.47)

< 0.001 0.18

BNP 4.9 (3.81-6.31) < 0.001 8.4 (3.45-20.45) < 0.001 0.22

Functional Form

• Determined by AIC

• Age linear

• LVEF dichotomized at 50%

• UACR, BNP, BMI, CRP: 3-knot restricted cubic splines

Backward Selection

• Eliminated 4 weakest predictors (p>0.5)– HDL, LDL, hx MI, HTN

• Top 4 predictors were always the same by all exploratory methods– Age, EF, BNP, UACR

• Remaining 10 candidates– Gender, BMI, smoker, diabetes, CRP, CKD,

troponin, hx CHF, med nonadherence, physical inactivity

Screening models using CV

• Base model age, LVEF, BNP, UACR• Screened all 5 to 11-predictor models

using 20 repetitions of 10-fold cross-validated C-index

• Targeting 5 to 7 predictor range, for practicality

• Done for both point score and continuous models

Top Models

Final Model

• Age, LVEF, BNP, UACR, smoker

• Point score– Naïve C-index 0.742– CV C-index 0.736

• Continuous model– Naïve C-index 0.768– CV C-index 0.763

Final Model with Dichotomized Predictors

Point score

• Age ≥ 65 1

• Smoker 1

• LVEF < 50% 2

• BNP > 500 3

• UACR ≥ 30 3

Continuous Model

Calibration Continuous Model

Pseudo-Hosmer-Lemeshow goodness-of-fit test: p = 0.94Cross-validated calibration slope = 0.94

Calibration with shrinkage

NRI with FHS model93 cases moved up47 cases moved down46 net cases46 / 243 =18.9%, p < 0.001

329 non-cases moved down82 non-cases moved up247 net non-cases247 / 661 = 37.4%p < 0.001

Net reclassification = 56.3%, p < 0.001

Cases

FHS Adding HS variables

0-10% 10-20% 20-50% ≥ 50% Total

0-10% 3 1 1 0 5

10-20% 6 3 13 6 28

20-50% 3 25 61 72 161

≥ 50% 0 0 13 36 49

Total 12 32 88 114 243

Non-Cases

FHS Adding HS variables

0-10% 10-20% 20-50% ≥ 50% Total

0-10% 25 5 2 0 32

10-20% 109 49 25 5 188

20-50% 33 166 164 45 408

≥ 50% 0 4 17 12 33

Total 167 224 208 62 661

NRI comparing point to cont.54 cases moved up29 cases moved down25 net cases25 / 246 =10.2%, p = 0.006

153 non-cases moved down94 non-cases moved up59 net non-cases59 / 670 = 8.8%p = 0.002

Net reclassification = 19%, p < 0.001

Cases

Point score

Continuous model

0-10% 10-20% 20-50% ≥ 50% Total

0-10% 8 2 1 0 11

10-20% 12 35 32 1 80

20-50% 0 8 52 18 78

≥ 50% 0 0 9 68 77

Total 20 45 94 87 246

Non-Cases

Point score

Continuous model

0-10% 10-20% 20-50% ≥ 50% Total

0-10% 146 24 1 0 171

10-20% 126 157 56 0 339

20-50% 3 18 100 13 134

≥ 50% 0 0 6 20 26

Total 275 199 163 33 670

Summary of results

• Our model had good discrimination (CV C-statistic 0.76), and had 56% net reclassification vs framingham secondary events model

• Many traditional risk factors (HTN, lipids, obesity) were not significant predictors

Limitations

• Population (VA men, CHN, urban)

• No external validation yet

Conclusion

• Developed a risk model with 5 predictors

• Can stratify 5-year recurrent CV event risk in stable CHD

External Validation

• PEACE cohort– Clinical trial of trandolapril vs placebo in

low-risk stable CAD– 3600 subjects with biomarkers– Patients were less sick, excluded EF<40% – 1996-2000

References

• Steyerberg E. Clinical Prediction Models: A practical approach to development, validation and updating. Springer, NY 2009.

• Lloyd-Jones D. Cardiovascular risk prediction: Basic concepts, current status, and future directions. Circ 2010; 121: 1768-77.

• Morrow D. Cardiovascular risk prediction in patients with stable and unstable coronary heart disease. Circ 2010; 121: 2681-91.

• D’Agostino R. Primary and subsequent coronary risk appraisal: new results from the Framingham study. AHJ 2000; 139: 272-81.

• Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med, 2000;19:453-473.