appraising a diagnostic test
DESCRIPTION
Appraising A Diagnostic Test. Clinical Epidemiology and Evidence-based Medicine Unit FKUI-RSCM. What is diagnosis ?. Increase certainty about presence/absence of disease Disease severity Monitor clinical course Assess prognosis – risk/stage within diagnosis Plan treatment e.g., location - PowerPoint PPT PresentationTRANSCRIPT
Appraising A Diagnostic Test
Clinical Epidemiology and Evidence-based Medicine Unit
FKUI-RSCM
What is diagnosis ?Increase certainty about presence/absence of diseaseDisease severityMonitor clinical courseAssess prognosis – risk/stage within diagnosisPlan treatment e.g., location Stalling for time!
Knottnerus, BMJ 2002
Key Concept Pre-test Probability– The probability of the target
condition being present before the results of a diagnostic test are available.
Post-test Probability– The probability of the target
condition being present after the results of a diagnostic test are available.
Key ConceptPre-test Probability– The probability of the target condition
being present before the results of a diagnostic test are available.
Post-test Probability– The probability of the target condition
being present after the results of a diagnostic test are available.
Basic Principles (1)Ideal diagnostic tests – right answers:(+) results in everyone with the
disease and( - ) results in everyone elseUsual clinical practice:–The test be studied in the same
way it would be used in the clinical setting
Observational study, and consists of:–Predictor variable (test result)–Outcome variable (presence /
absence of the disease)
Basic Principles (2)Sensitivity, specificityPrevalence, prior probability, predictive valuesLikelihood ratiosDichotomous scale, cutoff points (continuous scale)Positive (true and false), negative (true & false)ROC (receiver operator characteristic) curve
General structure : 2 X 2 table
Target disorderPositive(disease)
Target disorderNegative (normal)
PredictorTest
positive
True positiveTPa
False positiveFPb
PredictorTest
negative
False negative
FNc
True negativeTNd
Disease(+)
Disease(-) Total
Test (+) True posa
False posb
a+b
Test (-)False neg
cTrue neg
d c+d
Total a+c b+d a+b+c+d a+ca+b+c+dPrevalence Pretest probability
Sensitivity
The proportion of people who truly have a designated disorder who are so identified by the test.Sensitive tests have few false negatives. When a test with a high Sensitivity is Negative, it effectively rules out the diagnosis of disease. SnNout
Specificity
The proportion of people who are truly free of a designated disorder who are so identified by the test. Specific tests have few false positivesWhen a test is highly specific, a positive result can rule in the diagnosis. SpPin
Disease(+)
Disease(-) Totals
Test (+) a b a+b
Test (-) c d c+d
Totals a+c b+d a+b+c+d a/a+c d/b+d
Probability of positive test result in patients with the disease
Probability of negative test result
in patients without the disease
Sensitivity Specificity
SnNOut SpPIn
SnNOutThe sensitivity of dyspnea on exertion for the diagnosis of CHF is 100% (41/(41+0)), and the specificity 17% (35/(183+35)). If DOE, it is very unlikely that they have CHF (0 out of 41 patients with CHF did not have this symptom)."SnNOut", which is taken from the phrase: "Sensitive test when Negative rules Out disease".
SpPinConversely, a very specific test, when positive, rules in disease. "SpPIn"!
The sensitivity of gallop for CHF is only 24% (10/41), but the specificity is 99% (215/218). Thus, if a patient has a gallop murmur, they probably have CHF (10 out of 13).
Iron deficiency anemiaTotals
Present Absent
Diagnostic
test result (Serum ferritin)
(+)<65
mmol/L731
a270b
1001 a+b
(-)>65
mmol/L78c
1500d
1578 c+d
Totals 809 a+c
1770 b+d
2579
a+b+c+d
Sensitivity=a/a+c=90%Specificity =d/b+d=85%
Pos predictive value=a/a+b=73%Neg predictive value=d/c+d=95%
LR + = sn/(1-sp)=90/15=6
Prevalence=(a+c)/(a+b+c+d)= 32%
Posttest odd =Pretest odd xLikelihood Ratio
PredictorOutcome
Odds = ratio of two probabilities Odds = p/1-p Probability = odds/1+odds
Likelihood ratio (+):Prob (+) result in people with the
diseaseProb (+) result in people w/out the
disease
Pretest Odds X LR = Posttest Odds
Key Concept Likelihood Ratio– Relative likelihood that a given test
would be expected in a patient with (as opposed to one without) a disorder of interest.
probability of the test result in pts without disease
LR=probability of a test result in pts with disease
Likelihood ratios (LR) General Rules of Thumb
LR > 10 or < 0.1 produce large changes in pre-test probabilityLR of 5 to 10 or 0.1 to 0.2 produce moderate changesLR of 1 to 2 or 0.5 to 1 produce small changes in pre-test probability
Test
CA B
pretest probability
0 .10 .20 .30 .40 .50 .60 .70 .80 .90 1
do not test
do nottreat
do not test
get on with treatment
Likelihood ratio
posttest probability
Test
+ = Sn/(1-Sp)(1-Sn)/Sp= -
PreTest odds x LR
pretest probability
Serum ferritin (mmol/L)
Iron def positive Iron def negative Likelih
ood ratio
Diagnostic impactNo % No %
Very positive <15 474 59(474/809) 20 1.1
(20/1770) 52 Rule inSpPin
Moderately positive 15-34 175 22
(175/809) 79 4.5(79/1770) 4.8 Intermed
High
Neutral 35-64 82 10(82/809) 171 10
(171/1770) 1 Indeter mine
Moderately negative 65-94 30 3.7
(30/809) 168 9.5(168/1770) 0.39 Intermed
low
Extremely negative >95 48 5.9
(48/809) 1332 75(1332/1770) 0.08 Rule out
SnNout
809 100(809/809) 1770 100
(1770/1770)
The usefulness of five levels of a diagnostic test result
Pretest probability
Likelihood ratio
Posttest probability
T4 value Hypothyroid
Euthyroid
5 or less 18 15.1 – 7.0 7 177.1 – 9.0 4 369 or more 3 39Totals 32 93
T4 value Hypothyroid
Euthyroid
≤ 5 18 1
> 5 14 92
Totals 32 93
T4 value Hypothyroid
Euthyroid
≤ 7 25 18> 7 7 75
Totals 32 93
T4 value Hypothyroid
Euthyroid
≤ 9 29 54
> 9 3 39
Totals 32 93
Cutoff point
Sens Spec
5 0.56 0.997 0.78 0.819 0.91 0.42
T4 level in suspected hypo-thyroidism in children
For tests / predictors with continuous values result , cutoff points should be determine to choose the best value to use in distinguishing those with and without the target disorder
Cutoff point
Sens Spec
5 0.56 0.997 0.78 0.819 0.91 0.42
Cutoff point
SensTP
1-SpecFP
5 0.56 0.017 0.78 0.199 0.91 0.58
Accuracy of the testThe accuracy of the test depends on how well the test separates the group being tested into those with and without the disease in questionAccuracy is measured by the area under the ROC curve. An area of 1 represents a perfect test; an area of 0.5 represents a worthless test (AUC)
• 0.90-1.00 = excellent (A)
• 0.80-0.90 = good (B)
• 0.70-0.80 = fair (C) • 0.60-0.70 = poor (D) • 0.50-0.60 = fail (F)
An ROC curve demonstrates several things:
It shows the tradeoff between sensitivity and specificity
• any increase in sensitivity will be accompanied by a decrease in specificity
The closer the curve follows the left-hand border and then the top border of the ROC space, the more accurate the test. The closer the curve comes to the 45-degree diagonal of the ROC space, the less accurate the test. The slope of the tangent line at a cutoff point gives the likelihood ratio (LR) for that value of the test.
Appraising DxTestIs the evidence valid? (V)
• Was there an independent, blinded comparison with a gold standard?
• Was the test evaluated in an appropriate spectrum of patients?
• Was the reference standard applied regardless of the test result?
• Was the test validated in a second, independent group of patients?
Can I trust the accuracy data?
RAMMbo
Recruitment: Was an appropriate spectrum of patients included? (Spectrum Bias)Maintainence: All patients subjected to a Gold Standard? (Verification Bias)Measurements: Was there an independent, blind or objective comparison with a Gold standard? (Observer Bias; Differential
Reference Bias)
Guyatt. JAMA, 1993
Critical AppraisalIs this valid test important? (I)
• Distinguish between patients with and those without the disease
• Two by two tables• Sensitivity and Specificity
–SnNOut–SpPIn
• ROC curves• Likelihood Ratios
Critical Appraisal
Can I apply this test to my patient (A)
• Similarity to our patient• Is it available• Is it affordable• Is it accurate• Is it precise
Critical AppraisalCan I apply this test to my patient?
• Can I generate a sensible pre-test probability–Personal experience–Practice database–Assume prevalence in the study
Critical AppraisalDiagnosis– Can I apply this test to a specific patient
• Will the post-test probability affect management
– Movement above treatment threshold– Patient willing to undergo testing
Thank You