the effect of include crp

The Effect of Including C-Reactive Protein in Cardiovascular RiskPrediction Models for Women

Background: While high-sensitivity C-reactive protein (hsCRP) is anindependent predictor of cardiovascular risk, global risk prediction

use.

Objective: To develop and compare global cardiovascular risk pre-diction models with and without hsCRP.

Design: Observational cohort study.

Setting: U.S. female health professionals.

Participants: Initially healthy nondiabetic women age 45 years andolder participating in the Women’s Health Study and followed anaverage of 10 years.

Measurements: Incident cardiovascular events (myocardial infarc-tion, stroke, coronary revascularization, and cardiovascular death).

Results: High-sensitivity CRP made a relative contribution to globalrisk at least as large as that provided by total, high-density lipopro-tein (HDL), and low-density lipoprotein (LDL) cholesterol individu-ally, but less than that provided by age, smoking, and blood pres-sure. All global measures of fit improved when hsCRP wasincluded, with likelihood-based measures demonstrating strong

preference for models that include hsCRP. With use of 10-year riskcategories of 0% to less than 5%, 5% to less than 10%, 10% toless than 20%, and 20% or greater, risk prediction was moreaccurate in models that included hsCRP, particularly for risk be-tween 5% and 20%. Among women initially classified with risks of5% to less than 10% and 10% to less than 20% according to theAdult Treatment Panel III covariables, 21% and 19%, respectively,were reclassified into more accurate risk categories. Although addi-tion of hsCRP had minimal effect on the c-statistic (a measure ofmodel discrimination) once age, smoking, and blood pressure wereaccounted for, the effect was nonetheless greater than that of total,LDL, or HDL cholesterol, suggesting that the c-statistic may beinsensitive in evaluating risk prediction models.

Limitations: Data were available only for women.

Conclusions: A global risk prediction model that includes hsCRPimproves cardiovascular risk classification in women, particularlyamong those with a 10-year risk of 5% to 20%. In models thatinclude age, blood pressure, and smoking status, hsCRP improvesprediction at least as much as do lipid measures.

Ann Intern Med. 2006;145:21-29. www.annals.orgFor author affiliations, see end of text.

The Framingham risk model (1) is used extensively fordetecting risk for coronary heart disease and has been

adapted by the Adult Treatment Panel III (ATP III) of theNational Cholesterol Education Program (2). The tradi-tional risk factors included are strong predictors of cardio-vascular risk, and the model has been validated in severalpopulations (3). However, despite the model’s success, upto 20% of all coronary events occur in the absence of anymajor risk factor (4, 5). In addition, most individuals whodo not develop coronary heart disease have at least 1 clin-ically elevated Framingham risk factor (6).

Given these modest levels of sensitivity and specificity,research over the past decade has focused on novel blood-based atherosclerotic risk factors that, like cholesterol, canbe inexpensively obtained and interpreted in the primarycare setting. One of the most promising of these is high-sensitivity C-reactive protein (hsCRP), a biomarker of in-flammation that has consistently been shown to predictincident myocardial infarction, stroke, and cardiovasculardeath among apparently healthy men and women afteradjustment for all components of the Framingham riskscore (7–16). Blood levels of hsCRP also correlate withhypofibrinolysis and abnormal glucose metabolism, andthus reflect pathophysiologic processes that are related tovascular occlusion but are not easily measured with tradi-tional risk factors (17–20). On that basis, in 2003 theCenters for Disease Control and Prevention and the Amer-

ican Heart Association published the first set of guidelinesto endorse the use of hsCRP as a potential adjunct totraditional risk factor screening (21).

Despite these data, no simple clinical algorithm thatincludes Framingham covariables and hsCRP has been de-veloped, and thus it has not been possible to determinewhether individuals might be more accurately classified ifhsCRP were added to global risk prediction models formajor cardiovascular disease (CVD), including myocardialinfarction, coronary revascularization, stroke, and cardio-vascular death (22).

See also:

PrintEditors’ Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22Editorial comment. . . . . . . . . . . . . . . . . . . . . . . . . . . 70Related article. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Summary for Patients. . . . . . . . . . . . . . . . . . . . . . . I-19

Web-OnlyAppendixAppendix TableConversion of figures and tables into slides

Annals of Internal Medicine Article

© 2006 American College of Physicians 21

models incorporating hsCRP have not been developed for clinical

Nancy R. Cook, ScD; Julie E. Buring, ScD; and Paul M. Ridker, MD

METHODS

We compared the clinical utility of global cardiovas-cular risk prediction models based on Framingham covari-ables with and without hsCRP among participants in theWomen’s Health Study (WHS) (23–25), a large-scale, na-tionwide cohort of U.S. women age 45 years and olderwho were free of CVD and cancer at study entry. Womenwere followed annually for the development of CVD, withan average follow-up of 10 years. All reported CVD out-comes, including myocardial infarction, ischemic stroke,coronary revascularization procedures, and deaths fromcardiovascular causes, were adjudicated by an end pointscommittee after medical record review. All study partici-pants provided written informed consent, and the studyprotocol was approved by the institutional review board ofBrigham and Women’s Hospital in Boston, Massachusetts.

Baseline blood samples were assayed for C-reactiveprotein with a validated, high-sensitivity assay (DenkaSeiken, Tokyo, Japan) and for total, high-density lipopro-tein (HDL), and low-density lipoprotein (LDL) cholesterolwith direct-measurement assays (Roche Diagnostics, Basel,Switzerland). Women who were diabetic at baseline wereexcluded from predictive modeling because ATP III label-ing considers diabetes to be a risk equivalent for coronaryheart disease (2). In parallel with guidelines established forlipid evaluation (26), models were initially fitted in a der-ivation cohort limited to women not taking hormone re-placement therapy at baseline (n ! 15 048 with data on allvariables) and were then applied to all nondiabetic women(n ! 26 927) for clinical risk prediction.

Development of Risk Prediction ModelsModels were fitted by using Cox proportional hazards

models (27), restricting predictors to components of theFramingham risk score (including either total or LDL cho-lesterol as well as HDL cholesterol), and adding hsCRP.To determine the functional form used for each predictor,we examined spline plots and fitted power functions todetermine the best fit for each variable. When fit was sim-ilar, we chose the simplest form, usually a linear term orlog transformation, especially when supported by previousdata from existing ATP III algorithms.

To model blood pressure more fully, we included anonlinear term for systolic blood pressure. As in the Fra-mingham score, we included antihypertensive medicationuse and considered its interaction with blood pressure. Atbaseline in the WHS, use of cholesterol-lowering medica-tions was rare, was not composed of statins, and was notstatistically significant in these models. As also in the Fra-mingham score, current but not past smoking was in-cluded. We considered interactions of all predictors withsmoking and age, particularly on the basis of inclusion inthe ATP III risk score. To enhance model simplicity, thesewere not included when they were only of marginal statis-tical significance. Of interest, when body mass index wasadded to the final model, it was not a statistically signifi-cant predictor of CVD in these data, an observation con-sistent with its absence from the Framingham risk score.

In addition to comparing the performance of the finalmodel derived from the WHS data with hsCRP to theWHS model without hsCRP, we also compared the per-formance of the formal ATP III model (2), with and with-out hsCRP (Appendix, available at www.annals.org). TheATP III model includes terms for the natural logarithms ofage, of total and HDL cholesterol, and of systolic bloodpressure and was developed for the prediction of “hard”coronary heart disease events, including myocardial infarc-tion and death from coronary heart disease. Total CVD,including ischemic stroke as well as revascularization pro-cedures, was used as the end point in these analyses. To beconservative and allow the best possible fit for all tradi-tional covariables, we recalculated !-coefficients for theATP III model in the WHS data before evaluating anyadditive effects of hsCRP.

Measures of Model FitThe primary means of comparing predictive models

based on Framingham covariables with and without hsCRPwas the Bayes information criterion, a likelihood-basedmeasure that adds a penalty for model complexity (28, 29).Lower values indicate better fit. Because of its common usein the medical literature, we also computed the c-index(28), or concordance probability, which is a generalizationof the c-statistic, or the area under the receiver-operatingcharacteristic curve (30), that allows for censored data. Forthese analyses, the c-index was computed and adjusted foroptimism due to overfitting with bootstrap sampling (31)

Context

The value of adding high-sensitivity C-reactive protein(hsCRP) to a global risk assessment model is unknown.

Contribution

The authors used the Women’s Health Study, a nation-wide cohort of 15 048 initially healthy women, to developa cardiovascular disease (CVD) risk prediction model usinghsCRP and Framingham risk model predictors. WhilehsCRP improved overall model fit, the clinical utility ofhsCRP in terms of reclassification was most substantial forthose with a 5% or greater 10-year risk based on tradi-tional risk factors.

Cautions

The study does not address the clinical value of loweringhsCRP level.

Implications

In this largely low-risk population, adding hsCRP to theFramingham model reclassified patients into groups thatbetter reflected their actual CVD risk. This effect was mostclinically relevant for those at intermediate risk.

—The Editors

Article Risk Prediction Models with and without hsCRP

22 4 July 2006 Annals of Internal Medicine Volume 145 • Number 1 www.annals.org

using the Design library in S-PLUS software (InsightfulCorp., Seattle, Washington) (28).

For comparison, we also computed several other mea-sures of global model fit (provided in the Appendix, avail-able at www.annals.org), including other likelihood-basedmeasures such as model weights for the Bayes informationcriterion, which provide an estimate of the posterior prob-ability of each model given the set of candidate modelsconsidered (29, 32); the Akaike information criterion andits corresponding model weights (32); and Nagelkerke’sgeneralized model R2 (33, 34). We computed the D-statis-tic of Royston and Sauerbrei (35), based on the separationof survival curves by predictor variables, and again adjustedfor optimism. Differences in statistics between nested mod-els were tested with a 1-sided test using bootstrap sampling(31). We also calculated the Brier score (28), which di-rectly compares the observed outcomes with the fittedprobabilities.

To assess model calibration, or how closely the pre-dicted probabilities reflect actual risk, observed risk wascalculated on the basis of 8 years of follow-up (available forall participants) and was extrapolated to 10 years for dis-play purposes. We computed the Hosmer–Lemeshow cal-ibration statistic (36) comparing observed and predictedrisk using 10 categories based on 2–percentage point incre-ments in predicted risk, ranging from less than 2% to 18%or greater. We also computed this statistic using decilecategories of predicted probabilities.

To address clinical utility, we directly compared pre-dicted risk estimates that are based on models using Fra-mingham covariables with and without hsCRP with actualrisk that was observed during study follow-up among all26 927 women for whom data were available. We usedweighted " statistics (37) to compare the predicted proba-bilities with and without hsCRP. To approximate clinical

criteria commonly used in current treatment guidelines (2,38), we grouped the predicted probabilities into 10-yearrisk categories of 0% to less than 5%, 5% to less than 10%,10% to less than 20%, and 20% or greater.

Finally, to address the generalizability of the finalWHS risk prediction model with hsCRP, we calibrated thepredicted probabilities to observed risk in the FraminghamHeart Study, using a limited-access data set available fromthe National Heart, Lung, and Blood Institute at www.nhlbi.nih.gov/resources/deca/default.htm. Mean values

Table 1. Best-Fitting Global Cardiovascular Risk Prediction Model among the Model Derivation Cohort of 15 048 Women fromthe Women’s Health Study*

Variable WHS Model

!-Coefficient SE Hazard Ratio P Value

Age 0.074 0.006 1.08 "0.001SBP # 125† 0.032 0.006 1.03 "0.001(SBP # 125)2† #0.0003 0.0002 1.00 0.058Antihypertensive use 0.264 0.123 1.30 0.032Current smoking 0.965 0.117 2.62 "0.001Ln(HDL)† #1.244 0.205 0.29 "0.001Ln(Total cholesterol)† 1.569 0.269 4.80 "0.001Ln(hsCRP)† 0.196 0.050 1.22 "0.001

Measures of fit‡ With hsCRP Without hsCRPBIC 6960.26 6969.60C-index 0.815 0.813Calibration P value (risk percentage) 0.23 0.039

* BIC ! Bayes information criterion; HDL ! high-density lipoprotein; hsCRP ! high-sensitivity C-reactive protein; Ln ! natural logarithm; SBP ! systolic blood pressure;WHS ! Women’s Health Study.† SBP is measured in mm Hg, HDL and total cholesterol are measured in mg/dL, and hsCRP is measured in mg/L.‡ Lower values of BIC and higher values of the c-index and calibration P value reflect better fit.

Figure 1. Relative risk (RR) of future cardiovascular eventsaccording to baseline high-sensitivity C-reactive protein(hsCRP) levels in the model derivation cohort (n " 15 048),adjusted for Framingham covariables.

Risk estimates are provided on a natural log scale and were derived froma Cox regression model using a flexible spline curve. Dotted lines repre-sent 95% CIs.

ArticleRisk Prediction Models with and without hsCRP

www.annals.org 4 July 2006 Annals of Internal Medicine Volume 145 • Number 1 23

among women age 47 years and older at the 10th Framing-ham examination were evaluated. Mean HDL cholesterollevel at the 10th examination was estimated on the basis ofage and HDL cholesterol level at the 15th examination.The population mean hsCRP value was estimated from a

model derived from the WHS, including the other riskfactors plus body mass index. The average 10-year risk formajor CVD in the Framingham data, including myocar-dial infarction, stroke, and cardiovascular death, was esti-mated by using a product-limit estimator. The projected

Figure 2. Cardiovascular point scoring system for women based on Framingham covariables and high-sensitivity C-reactive protein(hsCRP).

This scoring system is intended as an illustration only. CVD ! cardiovascular disease; HDL ! high-density lipoprotein; RR ! relative risk; SBP !systolic blood pressure. To convert cholesterol values to mmol/L, multiply by 0.02586.



10-year risk from the WHS models was calibrated to the10-year rate of cardiovascular outcomes among women in theFramingham data (Appendix, available at www.annals.org).

Role of the Funding SourcesThis work was supported by grants from the Donald

W. Reynolds Foundation, the Leducq Foundation, theDoris Duke Charitable Foundation, and the National In-stitutes of Health. The funding agencies had no role in thedesign, conduct, or reporting of the study or in the deci-sion to submit the manuscript for publication.

RESULTS

Among the 15 048 women used for model develop-ment, the mean age was 54 years (SD, 8); 1841 women(12%) were current smokers. A total of 2227 women(15%) had blood pressure of 140/90 mm Hg or higher,and 1802 women (12%) were taking antihypertensivemedication at baseline. Median lipid values were as follows:total cholesterol level, 5.3 mmol/L (interquartile range, 4.7to 6.1 mmol/L) [206 mg/dL (interquartile range, 181 to234 mg/dL)]; LDL cholesterol level, 3.2 mmol/L (inter-quartile range, 2.6 to 3.8 mmol/L) [124 mg/dL (interquar-tile range, 102 to 147 mg/dL)]; and HDL cholesterol level,1.3 mmol/L (interquartile range, 1.1 to 1.5 mmol/L) [49mg/dL (interquartile range, 42 to 59 mg/dL)]. The medianhsCRP level was 1.5 mg/L (interquartile range, 0.6 to 3.4mg/L). Over the mean 10-year follow-up, 390 women de-veloped CVD, including 116 myocardial infarctions, 100ischemic strokes, 217 coronary revascularization proce-dures, and 65 deaths due to cardiovascular cause. Somewomen experienced more than 1 of these events.

The best-fitting global cardiovascular risk predictionmodel in these data included all Framingham covariablesand hsCRP (Table 1). Figure 1 shows that after adjust-ment for all Framingham variables, there was a log-linearrelationship between hsCRP and risk for CVD. While riskestimates based on the model coefficients given in Table 1provide the best estimates, for clinical utility we also calcu-lated a point scoring system (39) for cardiovascular riskanalogous to that used by the National Cholesterol Educa-tion Program (Figure 2). The absolute risk estimates inFigure 2 are calibrated to the overall incidence in the Fra-mingham Heart Study sample to enhance generalizability.

Relative Contributions to Global Risk of Age, BloodPressure, Smoking, Lipids, and hsCRP

Table 2 presents a comparison of the relative contri-butions to global risk made by each individual Framing-ham covariate and by hsCRP. Age is the strongest predic-tor of risk in these data, leading to a high likelihood ratioand a c-index of 0.73. After adjustment for age, likelihoodratio statistics demonstrated the strongest improvement infit for systolic blood pressure, followed in descending orderby hsCRP; current smoking; and HDL, total, and LDLcholesterol. In models that included age, systolic blood

pressure, and current smoking (the 3 strongest predictorsin aggregate), likelihood ratio statistics similarly demon-strated the strongest fit for hsCRP, followed in order byHDL, total, and LDL cholesterol.

Table 2 also demonstrates the relative insensitivity ofthe c-index as a method of determining model fit. Forexample, while the likelihood ratio statistic was able torank the relative contributions of systolic blood pressure,hsCRP, smoking, and HDL cholesterol in the age-adjustedanalyses, the c-index had little ability to distinguish amongany of these (all values between 0.75 and 0.77). Similarly,in the analyses adjusted for age, systolic blood pressure, andsmoking, the likelihood ratio statistic indicated that eitherhsCRP or HDL cholesterol improved fit more than didtotal or LDL cholesterol, yet the c-index was again unableto distinguish between these measures (all values 0.80).Discrimination and Calibration in Models with andwithout hsCRP

For both the ATP III model and the final WHSmodel, the likelihood ratio test for the inclusion of hsCRPwas highly significant (P " 0.001). Of note, the Bayesinformation criterion indicated a strong preference for theinclusion of hsCRP (Table 1) after adjustment for addinga variable. This suggests that the model including hsCRPprovided better fit, even after adjustment for the increase innumber of predictors. Similarly, models that includedhsCRP demonstrated better calibration (higher P value forcalibration), while models without hsCRP had larger devi-ations between the observed and predicted probabilities inthe higher-risk categories (Figure 3). By contrast, the c-

Table 2. Relative Contribution of Individual FraminghamCovariables and High-Sensitivity C-Reactive Protein toGlobal Cardiovascular Risk*

Variable Variable LRChi-Square

Overall Model Fit

Model LRChi-Square

C-Index

Age only 267.05 0.731$ SBP 100.60 367.66 0.768$ Ln(hsCRP) 86.72 353.77 0.763$ Current smoking 75.04 342.10 0.757$ Ln(HDL cholesterol) 70.19 337.24 0.765$ Ln(total cholesterol) 36.72 303.78 0.747$ LDL cholesterol 31.13 298.18 0.746

Age, SBP, and smoking 444.59 0.791$ Ln(hsCRP) 44.05 488.64 0.800$ Ln(HDL cholesterol) 41.89 486.48 0.801$ Ln(total cholesterol) 26.28 470.87 0.796$ LDL cholesterol 22.94 467.53 0.796

* Estimated from Cox proportional hazards models. Variable chi-square is the1-degree-of-freedom likelihood ratio chi-square statistic for inclusion of each vari-able separately, with larger values indicating greater improvement in fit. Modelchi-square is the multi–degree-of-freedom test for the fit of the entire model. The$ sign indicates the addition of each variable separately to the model with age onlyor with age, SBP, and smoking only. HDL ! high-density lipoprotein; hsCRP !high-sensitivity C-reactive protein; LDL ! low-density lipoprotein; Ln ! naturallogarithm; LR ! likelihood ratio; SBP ! systolic blood pressure.



index again showed minimal ability to detect differences inmodel fit. As shown in the Appendix Table (available atwww.annals.org), all global measures of fit showed im-provement when hsCRP was added to prediction modelsbased on Framingham covariables alone.Clinical Risk Classification and Accuracy

To better compare model performance within clinicalcategories, we classified all nondiabetic women (n !26 927) into 4 risk groups defined by the ATP III catego-ries of 10-year risk for CVD of 0% to less than 5%, 5% toless than 10%, 10% to less than 20%, and 20% or greater.We then compared the WHS models with and withouthsCRP by cross-classifying expected risks and comparingthese to the observed proportions of events in each group.While there was general agreement between these classifi-cations (weighted " ! 0.86), the predicted risk categorieschanged substantially with the addition of hsCRP forwomen with at least a 5% 10-year risk according to only

the Framingham risk variables (Table 3). Specifically, morethan 20% of all participants with intermediate risk werereclassified with the addition of hsCRP; among those orig-inally classified as having 5% to less than 10% risk, 12%moved down a category in risk and 10% moved up.Among those originally classified as having 10% to lessthan 20% risk, 19% were reclassified: 14% to a lower and5% to a higher category. Among those at high risk (#20%risk), 14% were reclassified into a lower-risk category. Bycontrast, among those with less than 5% risk according toFramingham covariables, only 2% were reclassified. Thus,overall in this low-risk cohort, with 88% of women in thelowest risk group, 4% were reclassified. However, thisoverall percentage depends heavily on the underlying riskand would be substantially greater in an older or higher-risk population. For comparison, according to ATP III riskcategories based on National Cholesterol Education Pro-gram !-coefficients rather than those derived from the

Figure 3. Calibration curves for risk prediction models without (top) and with (bottom) high-sensitivity C-reactive protein (hsCRP)in the model.

The model that includes hsCRP shows closer agreement between observed and model-based predicted risk. WHS ! Women’s Health Study.



WHS, among women originally classified as having lessthan 5%, 5% to less than 10%, 10% to less than 20%, and20% or greater 10-year risk, 4%, 38%, 42%, and 20%,respectively, were reclassified in the WHS model that in-cluded hsCRP.

Of note, for women reclassified, the models that in-cluded hsCRP also estimated the actual risk more accu-rately. For example, among those classified in the 5% toless than 10% category according to the model withouthsCRP, the 12% reclassified to the lower-risk category ac-tually experienced only a 2% risk, and the 10% reclassifiedto a higher-risk category actually experienced a higher 15%risk (Table 3). The addition of hsCRP also led to bettercalibration for women initially classified into the other 3groups.

DISCUSSION

The Framingham risk score provides a useful measureof risk stratification for coronary heart disease and has beenvaluable in clinical practice. Whether it can be improvedby including other simple and inexpensive measures hasnot previously been determined. We fitted predictive mod-els for major CVD, including myocardial infarction, coro-nary revascularization, stroke, and cardiovascular death, us-ing traditional Framingham predictors, with and withoutthe addition of hsCRP. As shown, all global measures ofpredictive accuracy were improved in prediction modelsthat included hsCRP. In particular, likelihood-based mea-sures showed a strong preference for the models with

hsCRP, and the Bayes information criterion weights, cor-responding to the posterior model probabilities, stronglysupported the addition of hsCRP. As also shown in thesedata, the relative contribution to global risk made byhsCRP was at least as large as that made by total, HDL, orLDL cholesterol. The added predictive value of includinghsCRP was most evident among those at 5% or greater10-year risk.

While all the likelihood-based measures of fit im-proved when hsCRP was added, the c-index, or generalizedc-statistic, changed little with the addition of any blood-based risk factor (including total, HDL, and LDL choles-terol) once we accounted for age, smoking, and blood pres-sure. In fact, the increase in the c-index associated withhsCRP was greater than that for any of the lipids. Thisobservation underscores the limitations of the c-index, orarea under the receiver-operating characteristic curve, as amethod for determining model fit, despite its continuedpopular use in the medical literature. The c-index is par-ticularly suited to retrospective case–control studies, inwhich the actual outcome probabilities cannot be esti-mated (40). Since it is based exclusively on ranks, however,it measures only how well the predicted values can rank-order the responses. It may not be as sensitive as the like-lihood function in choosing between models, especiallywhen the models are strong (28). This may be particularlytrue in settings where many individuals fall into low-riskgroups, as is true of cardiovascular risk detection in thegeneral population as well as in the WHS. In the current

Table 3. Observed and Expected Risks among all 26 927 Nondiabetic Women in the Women’s Health Study Using the FinalGlobal Risk Prediction Model with and without High-Sensitivity C-Reactive Protein*

10-Year Risk in WHS Model without hsCRP 10-Year Risk in WHS Model with hsCRP Total Reclassified

0% to <5% 5% to <10% 10% to <20% >20%

0% to <5%Total participants, n 23 174 488 0 0Participants classified in each risk stratum

by the WHS model with hsCRP, %97.9 2.1 0.0 0.0 2.1

10-y risk, % 1.6 5.8 – –

5% to <10%Total participants, n 267 1773 213 0Participants classified in each risk stratum


10-y risk, % 2.4 7.8 15.2 –

10% to <20%Total participants, n 0 110 653 40Participants classified in each risk stratum


10-y risk, % – 6.8 11.5 19.9

>20%Total participants, n 0 0 30 179Participants classified in each risk stratum


10-y risk, % – – 18.8 27.1

* All estimated and observed risk estimates have been extrapolated to 10-y risk. hsCRP ! high-sensitivity C-reactive protein; WHS ! Women’s Health Study.



data, the c-index could not distinguish between blood pres-sure, smoking, or any of the measured blood predictors,even though the likelihood ratio statistics clearly orderedvariable contributions, with systolic blood pressure beingthe strongest predictor after age. In fact, hsCRP was astronger individual predictor than any of the lipid mea-sures in these data, including HDL cholesterol, its closestcompetitor. Thus, reliance solely on the c-statistic formodel development could erroneously lead to the exclu-sion of lipids as well as hsCRP from risk prediction models.

Accuracy, or the predictive ability of a model, has 2major components, discrimination and calibration (28).The c-statistic is a measure of discrimination, or the abilityto separate 2 groups, such as case-patients and controls.Particularly in a prospective study, calibration, or how wellthe predicted probabilities reflect actual risk, is anotheraspect of accuracy not captured by the c-statistic. A modelcould discriminate well but lack even internal calibration ifthe fitted scores do not reflect the true probability of anevent. The predicted probability given the risk factors, orthe post-test probability, can be more useful clinically inassessing future cardiovascular risk than sensitivity or spec-ificity, on which the c-statistic is based. Put another way, asMoons and Harrell have stated (41), sensitivity and speci-ficity have no direct diagnostic meaning; for the patient,the issue is not the risk for having a positive test result, butthe risk for developing the disease. In our data, the modelthat included hsCRP was better calibrated and was a betterpredictor of the probability of disease.

Clinically, the inclusion of hsCRP in global predictionmodels more accurately predicted true cardiovascular riskin these data. While it did not strongly affect estimated riskamong women originally at very low risk, hsCRP had asubstantial effect among women at 5% or higher 10-yearrisk. Among those classified as having 5% to less than 20%risk, the addition of hsCRP reclassified about 20% ofwomen into more accurate risk strata. This would suggestthat an effective clinical strategy to improve risk predictionmight be to evaluate hsCRP among women with at least5% predicted risk based on traditional risk factors.

One of the primary goals of any risk prediction algo-rithm for CVD is to identify individuals at increased riskwho will benefit from aggressive lifestyle changes, includ-ing dietary moderation, exercise, and smoking cessation, allof which reduce hsCRP levels in addition to decreasingcardiovascular risk. Thus, knowledge of hsCRP level andmore precise global risk estimation may help motivate pa-tients to improve adherence to therapeutic lifestylechanges, as currently advocated in the ATP III guidelines(2). Ten-year risks are also commonly estimated to maketherapeutic decisions with regard to lipid-lowering therapy,particularly with statin agents. Previous research demon-strates that statin agents decrease hsCRP levels in a mannerlargely independent of LDL cholesterol, and the efficacy ofstatin therapy is linked in part to underlying hsCRP levels(42–46). As shown in our data, approximately 20% of

individuals at “intermediate risk” had their risk estimatessubstantively increased or decreased with improved accu-racy when hsCRP was added to the Framingham covari-ables. Thus, the use of an hsCRP-modified Framinghamrisk score also has the potential to help more patients andphysicians to better direct the use of preventive statin ther-apy to appropriate risk groups.

Limitations of our analysis merit consideration. First,our analysis is limited to women, and thus care must betaken before these data are generalized to men. However,data from several other large cohorts of men have foundhsCRP to predict risk independently of the traditional Fra-mingham covariables (7, 9–12, 14, 16, 47). Second, ourdata are based on a single determination of hsCRP insteadof 2 measures, as is currently recommended (21). Thislimitation would tend to increase variability in our mea-sures of hsCRP and thus lead, if anything, to an underes-timation of true effects. Third, in contrast to LDL choles-terol, there remains no evidence to date that reducinghsCRP level itself will reduce cardiovascular risk. As such,although these data show that in women the addition ofhsCRP can improve global risk prediction models, theyshould not be construed as implying a direct benefit ofhsCRP reduction, an issue now being evaluated in severalclinical trials. Finally, we evaluated only hsCRP in thisstudy and recognize the possibility that other biomarkers ofinflammation, hemostasis, and thrombosis may becomeavailable in the future. In this regard, we believe that themethods developed here to address incremental value forrisk screening will be of use as other novel biomarkers emerge.

From Brigham and Women’s Hospital, Harvard Medical School, andHarvard School of Public Health, Boston, Massachusetts.

Grant Support: By grants from the Donald W. Reynolds Foundation(Las Vegas, Nevada), the Leducq Foundation (Paris, France), and theDoris Duke Charitable Foundation (New York). The overall Women’sHealth Study cohort is supported by grants HL-43851 and CA-47988from the National Heart, Lung, and Blood Institute and the NationalCancer Institute (both in Bethesda, Maryland).

Potential Financial Conflicts of Interest: Honoraria: P.M. Ridker(Dade Behring); Grants received: P.M. Ridker (Reynolds Foundation,Leducq Foundation, Doris Duke Foundation, National Heart, Lung,and Blood Institute, National Cancer Institute, American Heart Association,Dade Behring, AstraZeneca, Novartis, Sanofi-Aventis). Dr. Ridker is listed asa co-inventor on patents held by the Brigham and Women’s Hospital thatrelate to the use of inflammatory biomarkers in cardiovascular disease.

Requests for Single Reprints: Nancy R. Cook, ScD, Division of Pre-ventive Medicine, Brigham and Women’s Hospital, 900 CommonwealthAvenue East, Boston, MA 02215; e-mail, [email protected].

Current author addresses and author contributions are available at www.annals.org.

References1. Wilson PW, D’Agostino RB, Levy D, Belanger AM, Silbershatz H, KannelWB. Prediction of coronary heart disease using risk factor categories. Circulation.1998;97:1837-47. [PMID: 9603539]



2. Executive Summary of The Third Report of The National Cholesterol Edu-cation Program (NCEP) Expert Panel on Detection, Evaluation, And Treatmentof High Blood Cholesterol In Adults (Adult Treatment Panel III). JAMA. 2001;285:2486-97. [PMID: 11368702]3. D’Agostino RB Sr, Grundy S, Sullivan LM, Wilson P. Validation of theFramingham coronary heart disease prediction scores: results of a multiple ethnicgroups investigation. JAMA. 2001;286:180-7. [PMID: 11448281]4. Ridker PM, Brown NJ, Vaughan DE, Harrison DG, Mehta JL. Establishedand emerging plasma biomarkers in the prediction of first atherothromboticevents. Circulation. 2004;109:IV6-19. [PMID: 15226246]5. Khot UN, Khot MB, Bajzer CT, Sapp SK, Ohman EM, Brener SJ, et al.Prevalence of conventional risk factors in patients with coronary heart disease.JAMA. 2003;290:898-904. [PMID: 12928466]6. Greenland P, Knoll MD, Stamler J, Neaton JD, Dyer AR, Garside DB, et al.Major risk factors as antecedents of fatal and nonfatal coronary heart diseaseevents. JAMA. 2003;290:891-7. [PMID: 12928465]7. Ridker PM, Cushman M, Stampfer MJ, Tracy RP, Hennekens CH. Inflam-mation, aspirin, and the risk of cardiovascular disease in apparently healthy men.N Engl J Med. 1997;336:973-9. [PMID: 9077376]8. Ridker PM, Rifai N, Rose L, Buring JE, Cook NR. Comparison of C-reactiveprotein and low-density lipoprotein cholesterol levels in the prediction of firstcardiovascular events. N Engl J Med. 2002;347:1557-65. [PMID: 12432042]9. Koenig W, Lowel H, Baumert J, Meisinger C. C-reactive protein modulatesrisk prediction based on the Framingham Score: implications for future risk as-sessment: results from a large cohort study in southern Germany. Circulation.2004;109:1349-53. [PMID: 15023871]10. Ballantyne CM, Hoogeveen RC, Bang H, Coresh J, Folsom AR, Heiss G,et al. Lipoprotein-associated phospholipase A2, high-sensitivity C-reactive pro-tein, and risk for incident coronary heart disease in middle-aged men and womenin the Atherosclerosis Risk in Communities (ARIC) study. Circulation. 2004;109:837-42. [PMID: 14757686]11. Danesh J, Wheeler JG, Hirschfield GM, Eda S, Eiriksdottir G, Rumley A,et al. C-reactive protein and other circulating markers of inflammation in theprediction of coronary heart disease. N Engl J Med. 2004;350:1387-97. [PMID:15070788]12. Pai JK, Pischon T, Ma J, Manson JE, Hankinson SE, Joshipura K, et al.Inflammatory markers and the risk of coronary heart disease in men and women.N Engl J Med. 2004;351:2599-610. [PMID: 15602020]13. Ridker PM, Wilson PW, Grundy SM. Should C-reactive protein be addedto metabolic syndrome and to assessment of global cardiovascular risk? Circula-tion. 2004;109:2818-25. [PMID: 15197153]14. Cushman M, Arnold AM, Psaty BM, Manolio TA, Kuller LH, Burke GL,et al. C-reactive protein and the 10-year incidence of coronary heart disease inolder men and women: the cardiovascular health study. Circulation. 2005;112:25-31. [PMID: 15983251]15. Ridker PM, Rifai N, Cook NR, Bradwin G, Buring JE. Non-HDL choles-terol, apolipoproteins A-I and B100, standard lipid measures, lipid ratios, andCRP as risk factors for cardiovascular disease in women. JAMA. 2005;294:326-33. [PMID: 16030277]16. Laaksonen DE, Niskanen L, Nisskanen K, Punnonen K, Tuomainen TP,Salonen JT. C-reactive protein in the prediction of cardiovascular and overallmortality in middle-aged men: a population-based cohort study. Eur Heart J.2005;26:1783-9. [PMID: 15821003]17. Festa A, D’Agostino R Jr, Howard G, Mykkanen L, Tracy RP, HaffnerSM. Chronic subclinical inflammation as part of the insulin resistance syndrome:the Insulin Resistance Atherosclerosis Study (IRAS). Circulation. 2000;102:42-7.[PMID: 10880413]18. Festa A, D’Agostino R Jr, Tracy RP, Haffner SM. Elevated levels of acute-phase proteins and plasminogen activator inhibitor-1 predict the development oftype 2 diabetes: the insulin resistance atherosclerosis study. Diabetes. 2002;51:1131-7. [PMID: 11916936]19. Pradhan AD, Manson JE, Rifai N, Buring JE, Ridker PM. C-reactiveprotein, interleukin 6, and risk of developing type 2 diabetes mellitus. JAMA.2001;286:327-34. [PMID: 11466099]20. Ridker PM, Buring JE, Cook NR, Rifai N. C-reactive protein, the metabolicsyndrome, and risk of incident cardiovascular events: an 8-year follow-up of 14719 initially healthy American women. Circulation. 2003;107:391-7. [PMID:12551861]21. Pearson TA, Mensah GA, Alexander RW, Anderson JL, Cannon RO 3rd,Criqui M, et al. Markers of inflammation and cardiovascular disease: application

to clinical and public health practice: a statement for healthcare professionals fromthe Centers for Disease Control and Prevention and the American Heart Associ-ation. Circulation. 2003;107:499-511. [PMID: 12551878]22. Grundy SM. The changing face of cardiovascular risk [Editorial]. J Am CollCardiol. 2005;46:173-5. [PMID: 15992653]23. Buring JE, Hennekens CH. The Women’s Health Study: summary of thestudy design. Journal of Myocardial Ischemia. 1992;4:27-9.24. Rexrode KM, Lee IM, Cook NR, Hennekens CH, Buring JE. Baselinecharacteristics of participants in the Women’s Health Study. J Womens HealthGend Based Med. 2000;9:19-27. [PMID: 10718501]25. Ridker PM, Cook NR, Lee IM, Gordon D, Gaziano JM, Manson JE, et al.A randomized trial of low-dose aspirin in the primary prevention of cardiovascu-lar disease in women. N Engl J Med. 2005;352:1293-304. [PMID: 15753114]26. Hainline A, Karon J, Lippel K, eds. Manual of Laboratory Operations, LipidResearch Clinics Program, and Lipid and Lipoprotein Analysis. 2nd ed. Bethesda,MD: Department of Health and Human Services; 1982.27. Cox DR. Regression models and life tables (with discussion). Journal of theRoyal Statistical Society B. 1972;34:187-220.28. Harrell FE Jr. Regression Modeling Strategies. New York: Springer; 2001.29. Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning:Data Mining, Inference and Prediction. New York: Springer-Verlag; 2001.30. Hanley JA, McNeil BJ. The meaning and use of the area under a receiveroperating characteristic (ROC) curve. Radiology. 1982;143:29-36. [PMID:7063747]31. Efron B, Tibshirani R. An Introduction to the Bootstrap. New York: Chap-man & Hall; 1993.32. Burnham KP, Anderson DR. Model Selection and Inference: A PracticalInformation-Theoretic Approach. New York: Springer-Verlag; 1998.33. Nagelkerke NJ. A note on a general definition of the coefficient of determi-nation. Biomka. 1991;78:691-2.34. Ash A, Shwartz M. R2: a useful measure of model performance when pre-dicting a dichotomous outcome. Stat Med. 1999;18:375-84. [PMID: 10070680]35. Royston P, Sauerbrei W. A new measure of prognostic separation in survivaldata. Stat Med. 2004;23:723-48. [PMID: 14981672]36. Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison ofgoodness-of-fit tests for the logistic regression model. Stat Med. 1997;16:965-80.[PMID: 9160492]37. Fleiss JL. Statistical Methods for Rates and Proportions. 2nd ed. New York:Wiley; 1981.38. Greenland P, Smith SC Jr, Grundy SM. Improving coronary heart diseaserisk assessment in asymptomatic people: role of traditional risk factors and non-invasive cardiovascular tests. Circulation. 2001;104:1863-7. [PMID: 11591627]39. Sullivan LM, Massaro JM, D’Agostino RB Sr. Presentation of multivariatedata for clinical use: The Framingham Study risk score functions. Stat Med.2004;23:1631-60. [PMID: 15122742]40. Gail MH, Pfeiffer RM. On criteria for evaluating models of absolute risk.Biostatistics. 2005;6:227-39. [PMID: 15772102]41. Moons KG, Harrell FE. Sensitivity and specificity should be de-emphasizedin diagnostic accuracy studies. Acad Radiol. 2003;10:670-2. [PMID: 12809422]42. Ridker PM, Rifai N, Pfeffer MA, Sacks FM, Moye LA, Goldman S, et al.Inflammation, pravastatin, and the risk of coronary events after myocardial in-farction in patients with average cholesterol levels. Cholesterol and RecurrentEvents (CARE) Investigators. Circulation. 1998;98:839-44. [PMID: 9738637]43. Ridker PM, Rifai N, Clearfield M, Downs JR, Weis SE, Miles JS, et al.Measurement of C-reactive protein for the targeting of statin therapy in theprimary prevention of acute coronary events. N Engl J Med. 2001;344:1959-65.[PMID: 11430324]44. Kent SM, Taylor AJ. Usefulness of lowering low-density lipoprotein choles-terol to "70 mg/dl and usefulness of C-reactive protein in patient selection. AmJ Cardiol. 2003;92:1224-7. [PMID: 14609606]45. Ridker PM, Cannon CP, Morrow D, Rifai N, Rose LM, McCabe CH, etal. C-reactive protein levels and outcomes after statin therapy. N Engl J Med.2005;352:20-8. [PMID: 15635109]46. Nissen SE, Tuzcu EM, Schoenhagen P, Crowe T, Sasiela WJ, Tsai J, et al.Statin therapy, LDL cholesterol, C-reactive protein, and coronary artery disease.N Engl J Med. 2005;352:29-38. [PMID: 15635110]47. Boekholdt SM, Hack CE, Sandhu MS, Luben R, Bingham SA, WarehamNJ, et al. C-reactive protein levels and coronary artery disease incidence andmortality in apparently healthy men and women: The EPIC-Norfolk prospectivepopulation study 1993-2003. Atherosclerosis. 2005. [PMID: 16257408]



Current Author Addresses: Drs. Cook, Buring, and Ridker: Division ofPreventive Medicine, Brigham and Women’s Hospital, 900 Common-wealth Avenue East, Boston, MA 02215.

Author Contributions: Conception and design: N.R. Cook, P.M. Ridker.Analysis and interpretation of the data: N.R. Cook, P.M. Ridker.Drafting of the article: N.R. Cook, P.M. Ridker.Critical revision of the article for important intellectual content: N.R.Cook, J.E. Buring, P.M. Ridker.Final approval of the article: N.R. Cook, P.M. Ridker.Statistical expertise: N.R. Cook.Obtaining of funding: P.M. Ridker.Collection and assembly of data: J.E. Buring.

48. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues indeveloping models, evaluating assumptions and adequacy, and measuring andreducing errors. Stat Med. 1996;15:361-87. [PMID: 8668867]

APPENDIXComputation of 10-Year Risk

The 10-year risk for cardiovascular disease calibrated to theFramingham population may be estimated for each individualwoman using the !-coefficients in Table 1. First, multiply eachwoman’s risk factor x by the appropriate coefficient in Table 1and sum these (! "! # x). The risk may then be computedfrom the following equation:

Risk ! 1 $ (0.903)exp("! # x $8.795)

Additional Measures of Model FitThe Appendix Table presents additional measures compar-

ing the best-fitting models in the WHS data with and withouthsCRP, as well as the ATP III model with coefficients refitted tothe WHS data, with and without hsCRP. As indicated, all globalmeasures of fit showed a preference for the models that includedhsCRP. The measures are as follows:

1. The likelihood ratio chi-square provides a global test ofmodel fit. It is a function of the degrees of freedom, or number ofterms in the model. The difference between chi-square valuesprovides a test of the model improvement with hsCRP (P %0.0001 for both the WHS and ATP III models).

2. The Bayes information criterion is a function of the loglikelihood but adds a penalty for added variables based on thesample size (28). It is not influenced by the number of predictors,so models can thus be compared directly. Lower values reflectbetter fit, suggesting improvement with the addition of hsCRP.

3. The Bayes information criterion weight provides an esti-mate of the posterior probability of each model given the set ofcandidate models considered (29, 32). The weights suggest a

much higher probability that the WHS model that includeshsCRP is correct.

4. The Akaike information criterion is a function of the loglikelihood that adds a penalty of 2 for each added variable (32),less extreme than the penalty used in the Bayes information cri-terion. Lower values are better, again suggesting improvementwith hsCRP.

5. The Akaike information criterion weights reflect the rel-ative likelihood of a model given the data and the set of models(32). These weights display a clear preference for the models withhsCRP.

6. Nagelkerke’s generalized model R2 (33, 34) is a measureof the fraction of the $2 log likelihood explained by the predic-tors, analogous to the percentage of variance explained in a linearmodel. It is adjusted to a range of 0 to 1 and is higher for modelswith hsCRP, both in the original data and after adjustment foroptimism using the bootstrap (31, 48).

7. The D-statistic of Royston and Sauerbrei (35) measuresthe separation of survival curves across levels of the predictorvariables, analogous to distance between Kaplan–Meier curves.This is higher for models that included hsCRP, even after adjust-ment for optimism, suggesting better prediction for these models.

8. The Brier score (28) computes the sum of squared differ-ences between the observed outcome and the fitted probability. Itis lower for models that included hsCRP, indicating that thepredicted probabilities are closer to the observed outcomes.

9. The c-index represents the area under the receiver-oper-ating characteristic curve (30), allowing for censored data. This isa measure of discrimination based on ranks and is similar butslightly higher for models that included hsCRP, even after ad-justment for optimism. The c-statistic is the probability that, fora randomly selected pair of subjects, one diseased and the othernondiseased, the person with disease will have the higher esti-mated disease probability according to the model.

10. The Hosmer–Lemeshow calibration statistic (36) classi-fies predicted probabilities into categories and compares themean predicted probability with the observed risk within eachcategory. A P value representing a significant difference indicatesa lack of fit. When decile categories are used, the predicted prob-ability is less than 5% for the first 9 of 10 categories. Calibrationis adequate for all models that use this measure and is somewhatbetter for models without hsCRP. The calibration statistic basedon risk percentage compares observed and predicted risk by using10 categories based on 2–percentage point increments in pre-dicted risk, from 0% to 2% risk to 18% or greater risk. Thisstatistic indicates significant deviation of observed and predictedvalues in models without hsCRP, suggesting a lack of fit in high-er-risk categories.

Annals of Internal Medicine

W-12 4 July 2006 Annals of Internal Medicine Volume 145 • Number 1 www.annals.org

Appendix Table. Comparison of Discrimination and Calibration for Global Risk Prediction Models with and withoutHigh-Sensitivity C-Reactive Protein*

Variable WHS Model ATP III Model

With hsCRP Without hsCRP With hsCRP Without hsCRP

Global measuresLR chi-square (df) 556.75 (8)† 541.44 (7) 558.69 (9)† 542.54 (8)BIC 6960.26† 6969.60 6964.28† 6974.46BIC weight 0.874† 0.0082 0.117† 0.0007AIC 6928.53† 6941.84 6928.58† 6942.74AIC weight 0.506† 0.0007 0.492† 0.0004R2, % 9.28† 9.03 9.31† 9.05Adjusted R2, %‡ 9.05† 8.92 8.97† 8.84D-statistic 1.948† 1.922 1.951† 1.918Adjusted D-statistic‡ 1.914† 1.893 1.913† 1.884Brier score 0.01960† 0.01965 0.01959† 0.01964

DiscriminationC-index 0.815† 0.813 0.814† 0.812Adjusted c-index‡ 0.813† 0.811 0.810† 0.809

CalibrationHosmer–Lemeshow statistic

P value, deciles 0.19 0.59† 0.71 0.79†P value, risk percentage§ 0.23† 0.039 0.25† 0.008

* Lower values of the BIC, AIC, and Brier score and higher values of all other statistics, including the calibration P values, indicate better fit. See text for descriptions. AIC !Akaike information criterion; ATP ! Adult Treatment Panel; BIC ! Bayes information criterion; df ! degrees of freedom; hsCRP ! high-sensitivity C-reactive protein;LR chi-square ! multi–degree-of-freedom likelihood ratio chi-square statistic for fit of the entire model; WHS ! Women’s Health Study.† Superior fit in comparison of models with and without hsCRP.‡ Adjusted for optimism using bootstrap resampling. Tests for improvement in the D-statistic with the addition of hsCRP to the ATP III model reached statisticalsignificance (P ! 0.03).§ Based on 10 categories defined by 2–percentage point increments in predicted risk.

www.annals.org 4 July 2006 Annals of Internal Medicine Volume 145 • Number 1 W-13

the effect of include crp

Documents