© 2009, kaiser permanente center for health research receiver operating characteristic (roc) curves...

© 2009, KAISER PERMANENTE CENTER FOR HEALTH RESEARCH

Receiver Operating Characteristic (ROC) Curves

Assessing the predictive properties of a test statistic – Decision Theory


Binary Prediction ProblemConceptual Framework

Suppose we have a test statistic for predicting the presence or absence of disease.

True Disease Status

Pos Neg

Test

Criterion

Pos

Neg


Binary Prediction Problem Conceptual Framework


True Disease Status

Pos Neg

Test

Criterion

Pos

Neg




True Disease Status

Pos Neg

Test

Criterion

Pos TP Neg

TP = True Positive




True Disease Status

Pos Neg

Test

Criterion

Pos

Neg




True Disease Status

Pos Neg

Test

Criterion

Pos FP Neg

FP = False Positive




True Disease Status

Pos Neg

Test

Criterion

Pos

Neg




True Disease Status

Pos Neg

Test

Criterion

Pos

Neg FN

FN = False Negative




True Disease Status

Pos Neg

Test

Criterion

Pos

Neg




True Disease Status

Pos Neg

Test

Criterion

Pos

Neg TN

TN = True Negative



True Disease Status

Pos Neg

Test

Criterion

Pos TP FPNeg FN TN

P N P+ N



Binary Prediction ProblemTest Properties

True Disease Status

Pos Neg

Test

Criterion

Pos TP FPNeg FN TN

P N P+ NAccuracy = Probability that the test yields a

correct result.= (TP+TN) / (P+N)



True Disease Status

Pos Neg

Test

Criterion

Pos TP FPNeg FN TN

P N P+ NSensitivity = Probability that a true case will test positive

= TP / PAlso referred to as True Positive Rate (TPR)

or True Positive Fraction (TPF).



True Disease Status

Pos Neg

Test

Criterion

Pos TP FPNeg FN TN

P N P+ NSpecificity = Probability that a true negative will test negative

= TN / NAlso referred to as True Negative Rate (TNR)

or True Negative Fraction (TNF).



True Disease Status

Pos Neg

Test

Criterion

Pos TP FPNeg FN TN

P N P+ N1-Specificity = Prob that a true negative will test positive

= FP / NAlso referred to as False Positive Rate (FPR)

or False Positive Fraction (FPF).



True Disease Status

Pos Neg

Test

Criterion

Pos TP FPNeg FN TN

P N P+ NPositive Predictive Value (PPV)

= Probability that a positive test will truly have disease

= TP / (TP+FP)



True Disease Status

Pos Neg

Test

Criterion

Pos TP FPNeg FN TN

P N P+ NNegative Predictive Value (NPV)

= Probability that a negative test will truly be disease free

= TN / (TN+FN)


Binary Prediction ProblemExample

True Disease Status

Pos Neg

Test

Criterion

Pos 27 173 200Neg 73 727 800

100 900 1000

27/100 = .27 Se =

Sp = 727/900 = .81

FPF = 1- Sp = .19

Acc = (27+727)/1000 = .75 PPV = 27/200 = .14 NPV = 727/800 = .91



Of these properties, only Se and Sp (and hence FPR) are considered invariant test characteristics.

Accuracy, PPV, and NPV will vary according to the underlying prevalence of disease.

Se and Sp are thus “fundamental” test properties and hence are the most useful measures for comparing different test criteria, even though PPV and NPV are probably the most clinically relevant properties.


ROC Curves

Now assume that our test statistic is no longer binary, but takes on a series of values (for instance how many of five distinct risk factors a person exhibits).

Clinically we make a rule that says the test is positive if the number of risk factors meets or exceeds some threshold (#RF > x)

Suppose our previous table resulted from using x = 4.

Let’s see what happens as we vary x.


ROC CurvesImpact of using a threshold of 3 or more RFs

True Disease Status

Pos Neg

Test

Criterion

Pos 45 200 245Neg 55 700 755

100 900 1000 27/100 = .45 Se =

Sp = 727/900 = .78

FPF = 1- Sp = .22

Acc = (27+727)/1000 = .75

PPV = 27/200 = .18

NPV = 727/800 = .93

Se , Sp , and interestingly both PPV and NPV

.27

.81 .14

.91

.75

200

800


ROC CurvesSummary of all possible options

Threshold TPR FPR

6 0.00 0.005 0.10 0.114 0.27 0.193 0.45 0.222 0.73 0.271 0.98 0.800 1.00 1.00

As we relax our threshold for defining “disease,” our

true positive rate (sensitivity) increases, but so does the false positive

rate (FPR).

The ROC curve is a way to visually display this

information.


ROC CurvesSummary of all possible options

Threshold TPR FPR

6 0.00 0.005 0.10 0.114 0.27 0.193 0.45 0.222 0.73 0.271 0.98 0.800 1.00 1.00

x=5

x=4

x=2

The diagonal line shows what we would expect from simple guessing (i.e., pure chance).

What might an even better ROC curve look like?


ROC CurvesSummary of a more optimal curve

Threshold TPR FPR

6 0.00 0.005 0.10 0.014 0.77 0.023 0.90 0.032 0.95 0.041 0.99 0.400 1.00 1.00

Note the immediate sharp rise in sensitivity. Perfect accuracy is represented by upper left corner.


ROC CurvesUse and interpretation

The ROC curve allows us to see, in a simple visual display, how sensitivity and specificity vary as our threshold varies.

The shape of the curve also gives us some visual clues about the overall strength of association between the underlying test statistic (in this case #RFs that are present) and disease status.



The ROC methodology easily generalizes to test statistics that are continuous (such as lung function or a blood gas). We simply fit a smoothed ROC curve through all observed data points.



See demo from www.anaesthetist.com/mnm/stats/roc/index.htm


ROC CurvesArea under the curve (AUC)

The total area of the grid represented by an ROC curve is 1, since both TPR and FPR range from 0 to 1.

The portion of this total area that falls below the ROC curve is known as the area under the curve, or AUC.


Area Under the Curve (AUC)Interpretation

The AUC serves as a quantitative summary of the strength of association between the underlying test statistic and disease status.

An AUC of 1.0 would mean that the test statistic could be used to perfectly discriminate between cases and controls.

An AUC of 0.5 (reflected by the diagonal 45° line) is equivalent to simply guessing.



The AUC can be shown to equal the Mann-Whitney U statistic, or equivalently the Wilcoxon rank statistic, for testing whether the test measure differs for individuals with and without disease.

It also equals the probability that the value of our test measure would be higher for a randomly chosen case than for a randomly chosen control.



FPR

TPR

1

0 1

ROC Curve

AUC

~ 0.540

casescontrols


AUC

~ .95

TPR

1

0 1FPR

ROC Curve


casescontrols



What defines a “good” AUC? Opinions vary Probably context specific

What may be a good AUC for predicting COPD may be very different than what is a good AUC for predicting prostate cancer



http://gim.unmc.edu/dxtests/roc3.htm .90-1.0 = excellent .80-.90 = good .70-.80 = fair .60-.70 = poor .50-.60 = failRemember that <.50 is worse than guessing!



www.childrens-mercy.org/stats/ask/roc.asp .97-1.0 = excellent .92-.97 = very good .75-.92 = good .50-.75 = fair


ROC CurvesComparing multiple ROC curves

Suppose we have two candidate test statistics to use to create a binary decision rule. Can we use ROC curves to choose an optimal one?



Adapted from curves at: http://gim.unmc.edu/dxtests/roc3.htm



http://en.wikipedia.org/wiki/Receiver_operating_characteristic



We can formally compare AUCs for two competing test statistics, but does this answer our question?

AUC speaks to which measure, as a continuous variable, best discriminates between cases and controls?

It does not tell us which specific cutpoint to use, or even which test statistic will ultimately provide the “best” cutpoint.


ROC CurvesChoosing an optimal cutpoint

The choice of a particular Se and Sp should reflect the relative costs of FP and FN results.

What if a positive test triggers an invasive procedure? What if the disease is life threatening and I have an

inexpensive and effective treatment? How do you balance these and other competing factors? See excellent discussion of these issues at

www.anaesthetist.com/mnm/stats/roc/index.htm


ROC CurvesGeneralizations

These techniques can be applied to any binary outcome. It doesn’t have to be disease status. In fact, the use of ROC curves was first introduced during

WWII in response to the challenge of how to accurately identify enemy planes on radar screens.


ROC CurvesFinal cautionary notes

We assume throughout the existence of a gold standard for measuring “disease,” when in practice no such gold standard exists. COPD, asthma, even cancer (can we truly rule out the absence of

cancer in a given patient?) As a result, even Se and Sp may not be inherently

stable test characteristics, but may vary depending on how we define disease and the clinical context in which it is measured. Are we evaluating the test in the general population or only among

patients referred to a specialty clinic? Incorrect specification of P and N will vary in these two settings.

© 2009, kaiser permanente center for health research receiver operating characteristic (roc) curves...

Documents

absence of disease

kaiser permanente center

true negative slide

true positive slide

health research receiver

false negative slide

false positive slide

tp tn p n slide