roc curves analysis

7/30/2019 ROC Curves Analysis

1/20

1

ROCCurvesAnalysis

Introduction

Receiveroperatingcharacteristic(ROC)curvesareusedinmedicinetodetermineacutoffvaluefora

clinicaltest. Forexample,thecutoffvalueof4.0ng/mlwasdeterminedfortheprostatespecificantigen

(PSA)testforprostatecancer. Atestvaluebelow4.0isconsideredtobenormalandabove4.0tobe

abnormal. ClearlytherewillbepatientswithPSAvaluesbelow4.0thatareabnormal(falsenegative)and

thoseabove4.0thatarenormal(falsepositive). ThegoalofanROCcurveanalysisistodeterminethe

cutoffvalue.

Assumethattherearetwogroupsofmenandbyusingagoldstandardtechniqueonegroupisknownto

benormal(negative),nothaveprostatecancer,andtheotherisknowntohaveprostatecancer(positive).

Abloodmeasurementofprostatespecificantigenismadeinallmenandusedtotestforthedisease.

Thetestwillfindsome,butnotall,abnormalstohavethedisease. Theratiooftheabnormalsfoundby

thetesttothetotalnumberofabnormalsknowntohavethediseaseisthetruepositiverate(alsoknown

assensitivity). Thetestwillfindsome,butnotall,normalstonothavethedisease. Theratioofthe

normalsfoundbythetesttothetotalnumberofnormals(knownfromthegoldstandardtechnique)is

thetruenegativerate(alsoknownasspecificity). ThehopeisthattheROCcurveanalysisofthePSAtest

willfindacutoffvaluethatwill,insomeway,minimizethenumberoffalsepositivesandfalsenegatives.

Minimizingthefalsepositivesandfalsenegativesisthesameasmaximizingthesensitivityandspecificity.

For

the

PSA

test

abnormal

values

are

large

(>

4)

and

normal

values

are

small

(


2/20

2

1 - Specificity

0.0 0.2 0.4 0.6 0.8 1.0

Sensitivity

0.0

0.2

0.4

0.6

0.8

1.0

Albumin, A = 0.72

Figure1:AnexampleROCcurve.

AnimportantmeasureoftheaccuracyoftheclinicaltestistheareaundertheROCcurve. Ifthisareais

equalto1.0thentheROCcurveconsistsoftwostraightlines,oneverticalfrom0,0to0,1andthenext

horizontalfrom0,1to1,1. Thistestis100%accuratebecauseboththesensitivityandspecificityare1.0

sotherearenofalsepositivesandnofalsenegatives. Ontheotherhandatestthatcannotdiscriminate

betweennormalandabnormalcorrespondstoanROCcurvethatisthediagonallinefrom0,0to1,1. The

ROCareaforthislineis0.5. ROCcurveareasaretypicallybetween0.5and1.0likeshowninFigure1.

TwoormoretestscanbecomparedbystatisticallycomparingtheROCareasforeachtest. Thetestsmay

becorrelatedbecausetheyoccurredfrommultiplemeasurementsonthesameindividual. Ortheymay

beuncorrelatedbecausetheyresultedfrommeasurementsondifferentindividuals. TheROCCurves

AnalysisModulereferstothisasPairedandUnpaired,respectively,andcananalyzeeithersituation.

Thetestmeasurementsmaycontainmissingvaluesandtwomethodsareprovidedtohandlemissing

valueswhencomparingROCareaspairwisedeletionandcasewisedeletion. Thisisdescribedindetail

later.

Givenavaluefortheprobabilitythatthepatienthasthedisease(pretestprobability)theprobabilitythat

thepatienthasthedisease,giventhevalueofthetestmeasurement,canbecomputed. Also,givena

valueforthefalsepositive/falsenegativecostratio(forthescreeningexampleabove,thefalsenegative

costwould

be

greater

than

the

false

positive

cost),

an

optimal

test

value

cutoff

can

be

computed.

The

presentprogramallowsentryofthepretestprobabilityandthefalsepositive/falsenegativecostratio.


3/20

3

DataEntry

Datacan

be

entered

in

two

formats

in

SigmaPlot

Indexed

and

Grouped.

IndexedDataFormat

ThisistheformatfoundinstatisticsprogramssuchasSYSTATandSigmaStat. Indexedisthe

terminologyusedinSigmaStat. Ithasonecolumnthatindexesanothercolumn(orothercolumns). Itis

alsotheformatoftheoutputoflogisticregressionwhereROCcurvesareusedtodeterminetheabilityof

differentlogisticmodelstodiscriminatenegativefrompositivetestresults(normalsfromabnormals).

Eachdatasetconsistsofapairofcolumnsaclassificationvariableandatestvariable. Theclassification

variablehasabinarystatethatiseithernegative(normal)orpositive(abnormal).Manyprogramsusea

valueof1forpositiveand0fornegative. Theclassificationvariableisrequiredtobelocatedincolumn1

oftheworksheet. Thetestvariableisacontinuousnumericvariableandcontainsthetestresults. A

singletest

variable

will

be

located

in

column

2.

Multiple

test

variables

will

be

located

in

multiple

columns

startingincolumn2. Thereisnobuiltinlimitforthenumberoftestvariables. Thereisonlyone

classificationvariableformultipletestvariablesanditislocatedincolumn1. Thetestvariablecolumns

mustbeleftjustifiedandcontiguous. Thereforenoemptycolumnstotheleftoforwithinthedataare

allowed.

Thefollowingexampleshowsafewrowsofdatafortwodatasets. Thefirstcolumnistheclassification

variable. ItcontainsacolumntitleThyroidFunctionwhichistheclassificationvariablename. Italso

containsthetwoclassificationstatesHypothyroidandEuthyroid(normalthyroidfunction).

HypothyroidandEuthyroidaretheabnormalandnormalclassificationstates,respectively. T4andT5are

thenamesofdifferentbloodteststhatwillbeusedintheROCanalysistodiscriminatebetweennormal

andabnormalandthencomparedtodeterminewhichisthebettertest. Theclassificationvariablemust

be

in

column

1

and

the

two

test

variables

in

the

two

columns

adjacent

to

it

Theclassificationvariablenamewillbeobtainedfromthecolumn1columntitleifitexists. Thetest

nameswillbeobtainedfromthecolumntitlesofthetestvariablecolumnsiftheyexist. Theclassification

statenameswillbeobtainedfromtheentriesinthecellsofcolumn1. Ifnocolumntitleshavebeen

enteredforthetestvariablesthendefaultnamesforthetests,Test1,Test2,etc.,willbeusedand

displayedinthegraphsandreports. Thetestvariablenamesshouldbeuniquebuttheprogramwill

subscriptanyidenticalnamesthatarenot.


4/20

4

Figure2:

Indexed

data

format

for

two

tests.

The

test

names

are

T4

and

T5,

the

classificationstatesareEuthyroidandHypothyroidandtheClassificationvariable

nameisThyroidFunction. Theindexcolumnisalwayscolumn1anddata

columnsmustbeleftadjusted.

Theremustbetwoormorenonmissingdatapointsforeachtestforeachclassificationstate.Missing

valuesarehandledautomaticallybytheanalysis. Fordatacolumns,missingvaluesareeverythingbut

numericvalues(blankcells,theSigmaPlotdoubledashmissingvaluesymbol,+inf,inf, NaN,etc.).

MissingvaluesareignoredforallcomputationsexceptthePairedareacomparison(seetheMissingValue

Methodsection)wheretheyarehandledusingoneoftwopossiblealgorithms.

GroupedData

Format

Thegroupeddataformatconsistsofpairsofdatacolumnsonepairforeachtest. Onecolumninadata

pairconsistsofthenegative(normal)datavaluesandtheothercolumnforpositive(abnormal)values.

So,forexample,iftwotestsaretobecompared,theworksheetwillcontainfourcolumnsofdatathe

firsttwocolumnsforthefirsttestandthethirdandfourthcolumnforthesecondtest.

Aspecificcolumntitleformatisusedtoidentifythetestassociatedwiththedatacolumnpairandthe

classificationstateswithineachpair. Theuserisencouragedtousethisformatsinceitclearlyidentifies

thedatainthedataworksheetandwillannotateallthegraphsandreportsgenerated. Itisnotnecessary

tousecolumntitlesastheprogramwillidentifycolumnpairsstartingincolumn1withthegeneratedtest

namesTest1,Test2,etc.,andwillarbitrarilyassign1and0classificationstatenamestothefirst

andsecond

columns,

respectively,

but

this

is

clearly

not

the

best

way

to

organize

the

data.

Since

the

test

namesandclassificationstatesarenumericalitisalsomoredifficulttointerprettheresults.

ColumnTitleConventionforGroupedData

ThiscolumntitleconventionisasimplewaytoidentifyworksheetdatafortheGroupeddataformat. The

followingexampleshowsafewrowsfortwodatasets. ThefirsttwocolumnscontainthedatafortheT4


5/20

5

test. ThefirstcolumnT4 EuthyroidisthecolumnwiththenormaldatafortestT4. Thecolumntitle

consistsofthetestnamefollowedbyaminussignfollowedbytheclassificationstate. Spacesoneither

sideoftheminussignareignored. ThesecondcolumnT4 Hypothyroidisthecolumnwiththe

abnormaldatafortestT4. Thethirdandfourthcolumntitlesarethesameasthefirsttwoexceptthe

secondtestnameT5isused.

Figure3. Groupeddataformatfortwotests. ThisisthesamedataasinFigure

1. TherearetwotestsT4andT5. Eachtestconsistsofapairofdatacolumns. In

thiscaseT4isincolumns1and2andT5incolumns3and4. TheTestState

columntitleformatisusedtoidentifythetwotestsandthenormal(Euthyroid)

andabnormal(Hypothyroid)states.

Thetestnamesinbothcolumnsofacolumnpairmustbethesame. Alsotheremustbeexactlytwo

classificationstatesinthecolumntitles.

Likethe

Indexed

format,

missing

values

in

the

worksheet

cells

are

ignored

except

for

special

handling

whencomparingROCareas(seetheMissingValueMethodsection).

ProgramOptions

SelectingROCCurvesfromtheSigmaPlotToolboxmenuopensthedialog


6/20

6

TestandclassificationstatenamesfromtheindexeddatashowninFigure2oftheDataEntrysectionare

displayedinthisdialog.

DataSelectionOptions

DataFormat(AutomaticDetermination)

Inmostcasetheprogramwillidentifythedataformatfromtheinformationinthedataworksheet. Inthe

dialogabovetheformatwasidentifiedasIndexed. Youmayselectfromthetwoformats Indexedand

Grouped.

AvailableDataSetsSelectedDataSets

SelectoneormoreoftheavailabledatasetsbyclickingonthemintheAvailableDataSetswindowand

thenclickingontheAddbutton. Ifdesired,youmaythenselectatestnameintheSelectedDataSets

windowandclickRemovetodeselectthetest.


7/20

7

DataType

IftwoormoredatasetsareselectedthentheDataTypeoptionforcorrelatedtestsismadeavailable

YoumayselecteitherPaired,forcorrelatedtests,orUnpaired. IfPairedisselectedtheROCareasand

areacomparisonsaredeterminedusingtheDeLong,DelongandClarkePearsonmethod(2). IfUnpairedis

selectedtheareasarecomputedusingtheHanleyandMcNeilmethod(3)

andtheareasarecompared

usingaZtest.

MissingValueMethod

IfmissingvaluesexistthentwooptionsareavailableforthepairwisecomparisonofROCareasPairwise

DeletionandCasewiseDeletion. Thisoptionisnotavailableifnomissingvaluesexist.

Pairwisedeletiononlydeletesrowscontainingmissingvaluesfortheparticularpairbeinganalyzednot

foranentirerowofdata. Fewerdatavaluesaredeletedusingthismethod. Therearesituationswhen

pairwisedeletionwillfailbutthisistheoptiontousewhenitispossible. Casewisedeletiondeletesall

cellsinanyrowofdatacontainingamissingvalue.Muchmoredatamaybedeletedusingthisoption. To

betterunderstandthedifference,considerasimpleexampleoftwodatacolumnsofequallengthoneof

whichhasnomissingvaluesandtheotherhasonemissingvalue.WhenROCareasarebeingcompared,

certaincomputationsonthesetwocolumnswillbedonepairwisethefirstcolumnwithitself,thefirst

columnwiththesecondcolumnandthesecondcolumnwithitself.Whenthecolumnwithoutamissingvalueisbeingcomparedwithitselfnorowdeletionsoccurforpairwisedeletion. Forcasewisedeletion,

however,therowthatcontainsthemissingvaluewillbedeletedfrombothdatasets. So,forcasewise

deletion,thecomputationinvolvingthecolumnwithoutamissingvaluewithitselfwillbedonewithone

rowdeleted(therowcorrespondingtothemissingvalueintheotherdataset). Theprogramdetermines

whenpairwisedeletionisnotvalidandinformstheuserwhenthisisthecase.


8/20

8

PositiveStateOptions ClassificationStateandDirection

ThetwoclassificationstatesarereferredtoasNegative(normal)orPositive(abnormal). TheROC

analysissoftwaremustbeinformedwhichstateisPositiveandwhetherthetestmeasurementvalues

forthepositivestateareHigh,meaninghigherthanthoseofthenegativestate,orLow,meaning

lowerthanthoseofthenegativestate.

AcceptednormalvaluesforthePSA(prostatespecificantigen)testarelessthan4ng/mlandabnormal

valuesarehigherthanthis. Thusifthetwoclassificationstatesnamesarepositiveandnegativethen

thePositivestateispositiveandthePositiveDirectionisHigh. Inthiscaseyouwouldselecttheradio

buttonnexttopositiveandHigh.

Ontheotherhand,fortheT4(thyroxine)testforhypothyroidismtheT4valuesarelowerintheabnormal

statethanforthenormalstate. InthiscasetheabnormalPositiveStateisHypothyroidandthePositive

DirectionisLow. SoyouwouldselecttheradiobuttonnexttoHypothyroidandLow.

Whathappensifyouselecttheincorrectoption? Sensitivity(specificity)isdefinedintermsofthepositive

(negative)state. Soifthepositivestateisincorrectlyselectedthensensitivityandspecificitywillbe

incorrectlydefined(switched)andtheROCcurvewillhavetheXandYaxesswitched. Thiswillresultinan

ROCcurvethatappearsbelowthediagonalunityline. Itwillhaveanarealessthan0.5. Theprogramwill

detectthisandgiveyoutheoptions


9/20

9

ItispossiblethatthereissomethingwrongwiththedatasoyoucanAborttheanalysisandcorrectthe

problem.MorelikelyyouhaveselectedtheincorrectpositivestateordirectionsoyoucanRetrythe

analysiswithcorrectselections. Inrareoccasionsformultipletestssometestswillhaveareasgreater

than0.5andoneormorewillhaveareaslessthan0.5. InthiscaseyoucanIgnorethiswarningand

continuewiththeanalysis.

ReportOptions

ConfidenceIntervals

ConfidenceintervalsarecomputedforstatisticsinboththeSensitivity&SpecificityandAreaComparison

reports. Youcangenerate90,95and99%confidenceintervals.

CreateSensitivityandSpecificityReport

Cutoffvaluesarecreatedbetweeneachtestdatavalueinthe(sorted)dataset. Iftherearealarge

numberof

data

points

and

several

tests

then

there

will

be

alarge

number

of

cutoff

values

and

the

Sensitivity&SpecificityReportcanbeverylong. Thecheckbox

allowsyoutoturnoffthisreport. Ifyouturnoffthisreportthenallreportoptionsinthedialogbelowthis

arenotrequiredandaredisabled.

Fractions/Percents

Youmaydisplaysensitivities,specificitiesandprobabilitiesineitherfractionorpercentformat. Selecting

Percentsalso

requires

the

pre

test

probability

to

be

entered

as

apercent.

CreatePostTestResults

Selectingthisoptionallowsentryofthepretestprobability. Italsoenablesthepossibleentryofthe

falsepositive/falsenegativecostratio. Givenapretestprobabilitytheprogramwillcreateposttest

probabilities,boththepositivepredictivevalue(PV+=probabilityofdiseasegivenapositivetestresult)

andthenegativepredictivevalue(PV =probabilityofnodiseasegivenanegativetestresult),foreach

cutoffvalue. Ifthecostratiooptionisselectedthentheoptimalcutoffvaluewillbecomputed. Allof

theseresultsaredisplayedforeachtestintheSensitivity&Specificityreport.

ROCGraph

Options

AllofthegraphoptionsinthedialogapplytotheROCgraph. Theyallowyoutoaddadiagonallinetothe

graph,addgridlines,addsymbolsforsensitivityandspecificityateachcutoffpointandchangetheROC

plotlinesfromsolidtodifferentlinestyles.

AnalysisResults


10/20

10

Introduction

TypicalresultsoftheROCanalysisareshowninthefollowingexamplefromtheNotebookManager.

ThefirstsectionentitledOvarianCancercontainstheworksheetcontainingtherawdata. Theprogram

createdthenextthreesectionsthatcontaintwographsandtworeports. Thecontentsofthetwographs

ROCCurves

DotHistogram

andthetworeports

Sensitivity&Specificity

ROCAreas

aredescribedinthenextsections.

ROCCurvesGraph

TheROCcurvesgraphforthreedatasetsisshowninFigure4. Thesegraphsarederivedfromnumerical

resultsintheworksheetentitledGraphData. Thegraphtitleisobtainedfromthesectionname

containingtherawdata. ThelegendshowsthetestnamesandtheROCareasforeachcurve. The

diagonallineandgridsoptionswereselectedforthisgraph.


11/20

11

Ovarian Cancer ROC Curves

1 - Specificity

0.0 0.2 0.4 0.6 0.8 1.0

Sensitivity

0.0

0.2

0.4

0.6

0.8

1.0

US, A = 0.85

CT, A = 0.93

MR, A = 0.99

Figure4. TheROCcurvesgraphforthreetests.

Ofcoursethisgraphcanbeeditedinanywayyouwish. Youmightwanttochangethestartingcolorof

thecolorschemeusedforthelinecolors. YoucandothisbydoubleclickingononeoftheROCplotlines

andthenrightclickingontheLineColorlistboxasshownnext.

DotHistogramGraph

DothistogramsforthedataassociatedwiththeROCcurvesinFigure4areshowninFigure5below.


12/20

12

Ovarian Cancer Data

US-normal

US-abnormal

CT-norm

al

CT-abnormal

MR-norm

al

MR-abno

rmal

TestValue

0

2

4

6

8

10

12

14

Cutoff < 7.13

Sens = 0.78

Spec = 0.85

Cutoff < 7.14

Sens = 0.84

Spec = 0.97

Cutoff < 8.34

Sens = 0.94

Spec = 0.97

Figure5: Dothistogrampairsforeachtest. Thehorizontallinesandthetables

belowthegraphshowtheoptimalcutoffvaluesdeterminedfromthepretest

probabilityandcostratio.

Thegraphtitleisobtainedfromthetitleofthesectioncontainingtherawdata. Thexaxisticklabelsare

obtainedfromthetestnamesandtheclassificationstatenames. Theticklabelswillrotateiftheyaretoo

longto

fit

horizontally.

The

symbol

layout

design

allows

for

symbols

to

touch

horizontally

and

nest

vertically.

Ifvaluesforpretestprobabilityandfalsepositive/falsenegativecostratioareenteredthentheoptimal

cutoffvaluesforeachtestarecomputedandrepresentedasahorizontallineacrossthetwodot

histogramsforeachtest. Thenumericvaluesfortheoptimalcutoffparametersareshownastables

belowthexaxis.

Sensitivity&SpecificityReport

Thesensitivity&Specificityreportcontainsresultsforalltestswithadditionaltestsresultsplacedin

reportrows

below

those

of

prior

tests.

The

results

for

each

test

can

be

separated

into

three

parts:

1)

optimalcutoffvalue,2)sensitivityandspecificityversuscutoffvaluesand3)likelihoodratiosandpost

testprobabilities.

Ifvaluesforbothpretestprobabilityandcostratiohavebeenenteredthentheoptimalcutoffis

calculated. AslopeofthetangenttotheROCcurvemisdefinedintermsofthetwoenteredvalues(P=

pretestprobability)(1)


13/20

13

1false positive c os t Pm

false negative cos t P

=

(1)

Theoptimalcutoffvalueiscomputedfromsensitivityandspecificityusingtheslopembyfindingthe

cutoffthatmaximizesthefunction(1)

( )1Sensitivity m Specificity (2)

TheresultsofthiscomputationintheSensitivity&SpecificityreportareshowninTable1.

Table1: OptimalcutoffresultsintheSensitivity&Specificityreport.

Forthisdataset,theoptimalcutoffis7.125forapretestprobabilityof0.5andcostratioof1.0.

Sensitivities,specificitiesandtheirconfidenceintervalsarelistedasafunctionofcutoffvalueinthe

secondpartofthereport. AportionoftheseresultsisshowninTable2. Theseresultscanbeexpressed

asfractionsorpercentsbyusingtheFractions/Percentsoption.


14/20

14

Table2: SensitivityandspecificityresultsintheSensitivity&Specificityreport.

ThethirdpartoftheSensitivity&Specificityreportcontainsthelikelihoodratiosandposttest

probabilities.

Thepositiveandnegativelikelihoodratiosaredefinedrespectivelyas

1

Pr obability of a positive test given the presenceof disease SensitivityLR

Pr obability of a positivetest given the absenceof disease Specificity+ = =

(3)

1Pr obability of a negativetest given the presence of disease SensitivityLR

Pr obability of a negative test given the absenceof disease Specificity

= = (4)

Theposttestprobabilitiesaretheprobabilityofdiseasegivenapositivetest(PV+)andtheprobabilityof

nodiseasegivenanegativetest(PV). Thesewillbecomputedwhenapretestprobabilityhasbeen

entered.Using

P=pre

test

probability,

the

equations

used

for

these

probabilities

are

( ) ( )1 1Sensitivity x P

PVSensitivity x P Specificity x P

+ =+

(5)

( )

( ) ( )

1

1 1

Specificity x PPV

Specificity x P Sensitivity x P

=

+ (6)

AportionofthereportshowingthelikelihoodandposttestprobabilitiesresultsisshowninTable3.


15/20

15

Table3: Positiveandnegativelikelihoodratios,LR+andLR,andposttest

probabilities,PV+andPV,intheSensitivity&Specificityreport.

Thepositivelikelihoodratioisnotdefinedforsomecutoffvaluessincespecificity=1.

ROCAreas

Report

TheROCAreareportconsistsoftwoparts:1)ROCareasandtheirassociatedstatisticsand2)pairwise

comparisonofROCareas. AnexampleofareportisshowninTable4.


16/20

16

Table4: AnexampleROCAreasreport. Fromtoptobottomitshowsthetypeof

analysisusedtogetherwiththemissingvaluemethod,theROCareasand

associatedstatisticsandapairwisecomparisonofROCareas.

Inthis

case

there

are

three

correlated

tests.

Row

two

of

the

report

shows

that

aPaired

Analysis

was

performedand,sincethereweremissingvaluesinthedata,PairwiseDeletionofmissingvalueswas

selectedtocomparetheareas.

ThefirstsectionofthereportshowstheROCcurveareasforthethreetests. Thisisfollowedbythe

standarderroroftheareaestimate,the95%confidenceinterval(90%and99%arealsoavailable)andthe

Pvaluethatdeterminesiftheareavalueissignificantlydifferentfrom0.5. Thesamplesizeandthe

numberofmissingvaluesforeachclassificationstatearegiven. Thenumberofmissingvaluesreflects

onlywhatisseeninthedataanddoesnotgivethenumberusedforeachcomputationpairinthe

pairwisedeletedcomparisonofareas.

Thesecondsectionshowstheresultsofthepairwisecomparisonofareas. ThemethodofDeLong,

DeLongandClarkePearson(2)

isusedtocompareareaswhenthePaireddatatypeoptionisselected.

Whenthe

Unpaired

data

type

is

selected,

areas

are

compared

using

aZtest.

The

report

shows

results

for

allpairsofdatasets. Thedifferenceofeachareapairanditsstandarderrorand95%confidenceinterval

arecomputed. Thisisfollowedbythechisquarestatisticfortheareacomparison(orZstatisticif

Unpairedisselected)anditsassociatedPvalue.


17/20

17

FormattedFullPrecisionDisplay

Thisreportpresentsthenumericresultsinafoursignificantdigitformatwithfullprecisionavailable.

Doubleclickonanycell(excepttheconfidenceintervals)todisplaythenumberatfullprecision.

AdditionalGraphs

Resultsdatainbothreportscanbeusedtocreateadditionalgraphs. Someexamplesseeninthe

literatureareshownhere.

SensitivityandSpecificityvs.Cutoff

ThedataforthegraphinFigure6isfromtheSensitivity&Specificityreportincolumns1,2and4. Use

theDataSamplingoptioninGraphProperties,Plots,Datatospecifytherowrangeforthegraph(youcan

alsodragselecttherowsintheworksheettodothis).

Cutoff

0 2 4 6 8 10 12 14

Sensitivity

0.0

0.2

0.4

0.6

0.8

1.0

S

pecificity

0.0

0.2

0.4

0.6

0.8

1.0

Figure6:Graphofsensitivityandspecificityvs.cutoffforonetestusingdata

fromcolumns1,2and4oftheSensitivity&Specificityreport.

LikelihoodRatios

ThepositiveandnegativelikelihoodratiosforthreedifferentimagingmodalitiesareshowninFigure7

(thedataisartificial). Thedataisincolumns1,6and7oftheSensitivity&Specificityreport. Thevalues

associatedwiththeoptimalcutoffareshownassolidsymbols. Thelargestpositivelikelihoodand

smallestnegativelikelihoodattheoptimalcutoffisassociatedwithmagneticresonanceimaging(MR).


18/20

18

Likelihood Ratio of a Positive Test

Cutoff

4 5 6 7 8 9 10

LR

+

0

20

40

60

80

US

CT

MR

Likelihood Ratio of a Negative Test

Cutoff

4 5 6 7 8 9 10

LR-

0.0

0.2

0.4

0.6

0.8

1.0

1.2

US

CT

MR

Figure7: Positiveandnegativelikelihoodratiosgraphedfromdatainthe

Sensitivity&Specificityreportfromcolumns1,6and7. Theresultsforthree

testsareshowntogetherwithvaluesassociatedwiththeoptimalcutoff(solid

symbols).

OptimalCutoffvs.CostRatio

Frequentlyitcanbedifficulttodetermineavalueforthefalsepositive/falsenegativecostratio. Soitis

worthperformingasensitivityanalysis(sensitivityheremeanshowmuchonevariablechangeswith

changesinasecondvariable)toseewhetherthecutoffvaluechangessignificantlyintherangeofcost

ratiovaluesofinterest. TheROCCurvesModulewasrunmultipletimesfordifferentcostratiosanda

graphofoptimalcutoffvs.costratioforthethreeimagingmodalitytestsisshownbelow.

Cost Ratio

0.1 1 10

OptimalCutoff

4

6

8

10

12

14

US

CTMR

Figure8: Optimalcutoffvaluesobtainedfrommultiplerunsoftheprogram.

Regionsofinsensitivity,orstrongsensitivity,tocostratiocanbeidentified.

Iftherelativecostofafalsepositiveismuchgreaterthanthatofafalsenegativethenthecostratiois

greaterthan1. Butletsassumethatwedontknowexactlyhowmuchgreateritisbuthavesomeidea

thatitshouldbeintherangeof2to5,say. Lookingattheoptimalcutoffforthebestimagingmodality


19/20

19

(MR,greenline)wefindthatitdoesntchangeforcostratiosfrom2to20. Sotheoptimalcutoffis

insensitivetocostratioand,inthiscase,itisnotimportanttoknowaprecisevalueforcostratio.

PostTestProbabilityvs.PreTestProbability

Givenvaluesofsensitivityandspecificityassociatedwiththeoptimalcutoffagraphofposttest

probabilitiesasafunctionofpretestprobabilitycanbecreatedusingequations(5)and(6). Theposttest

probabilityofdiseasewhenthetestispositive,bluelinesinFigure9,wasobtainedfromequation(5)and

theposttestprobabilityofdiseasewhenthetestwasnegative,redlines,wasobtainedfrom1.0minus

equation(6). AtransformwaswritteninSigmaPlotimplementingthesetwoequationsthatgeneratedthe

posttestprobabilitiesforarangeofpretestprobabilities. Theresultsforthebesttest,MR,andworst

test,US,areshown. TheMRtestisclearlybettersincetheposttestprobabilityrange,fromnegativetest

topositivetest,islarger. Thusgivenapositivetestthepatientismorelikelytohavethediseaseusingthe

MRtestratherthantheUStest. Similarly,givenanegativetestitislesslikelythatthepatienthasthe

diseaseusingtheMRtest.

Imaging Modalities

Pre-Test Probability

0.0 0.2 0.4 0.6 0.8 1.0

Post-TestProbability

0.0

0.2

0.4

0.6

0.8

1.0

US

MR

Test Positive

Test Negative

Figure9: Posttestprobabilitiesofdiseasegivenpositiveandnegativetest

results. TheMRtestisbasedonsensitivity=0.94andspecificity=0.97whereas

theUStestusedsensitivity=0.78andspecificity=0.85.


20/20

20

References

1. Zweig,MH,Campbell,G.Receiveroperatingcharacteristic(ROC)plots:Afundamentalevaluation

toolinclinicalmedicine. ClinChem1993;39/4,56577.

2. DeLong,ER,DeLong,DM,ClarkePearson,DL. Comparingtheareasundertwoormore

correlated

receiver

operating

characteristic

curves:

a

nonparametric

approach.

Biometrics

1988;44,837845.

3. Hanley,JA,McNeil,BJ. Themeaninganduseoftheareaunderareceiveroperating

characteristic(ROC)curve. Radiology1982,143,2936.

roc curves analysis

Documents