1 performance of a diagnostic test based on the lecture of 2011 by steen ethelberg dagmar rimek...
TRANSCRIPT
1
Performance of a diagnostic test
Based on the Lecture of 2011 by Steen Ethelberg
Dagmar Rimek
EPIET-EUPHEM Introductory Course 2012
Lazareto, Menorca, Spain
2
Outline
1. Performance characteristics of a test– Sensitivity– Specificity– Choice of a threshold
2. Performance of a test in a population – Positive predictive value of a test (PPV)– Negative predictive value of a test (NPV)– Impact of disease prevalence, sensitivity and
specificity on predictive values
3
1. Performance characteristics of a test in a laboratory
setting
4
Population with affected and non-affected individuals
Affected
Non-affected
5
A perfect diagnostic test identifies the affected individuals only
Affected
Non-affected
6
In reality, tests are not perfect
Affected
Non-affected
Sensitivity of a test
The sensitivity of a test is the ability of the test to identify correctly the affected individuals
Proportion of persons testing positive among affected individuals
Affected persons
Test result+-
True positive (TP)
False negative (FN)
Sensitivity (Se) = TP / (TP + FN)
7
Estimating the sensitivity of a test
• Identify affected individuals with a gold standard
• Obtain a wide panel of samples that are representative of the population of affected individuals– Recent and old cases
– Severe and mild cases
– Various ages and sexes
• Test the affected individuals
• Estimate the proportion of affected individuals that are positive with the test
8
Example: Estimating the sensitivity of a new ELISA IgM test for acute Q-fever
• Identify persons with acute Q-fever with a gold standard (IgM Immunofluorescence Assay)
• Obtain a wide panel of samples that are representative of the population of individuals with acute Q-fever– Recent and old cases
– Severe and asymptomatic cases
– Various ages and sexes
• Test the persons with acute Q-fever
• Estimate the proportion of persons with acute Q-fever that are positive with the ELISA IgM test
9
Example: Sensitivity a new ELISA IgM test for acute Q-fever
Patients with acute Q-fever
ELISA IgM test result+ True positive (TP) 148
- False negative (FN) 2
150
Sensitivity =TP / (TP + FN)
148 / 150 = 98.7%
10
What factors influence the sensitivityof a test?
• Characteristics of the affected persons? YES: Antigenic characteristics of the pathogen in the area
(e.g., if the test was not prepared with antigens reflecting the population of pathogens in the area, it will not pick up infected persons in the area)
• Characteristics of the non-affected persons? NO: The sensitivity is estimated on a population of affected
persons
• Prevalence of the disease? NO: The sensitivity is estimated on a population of affected
persons
Sensitivity is an INTRINSIC characteristic of the test
11
Specificity of a test
Specificity (Sp) = TN / (TN + FP)
The specificity of a test is the ability of the test to identify correctly non-affected individuals
Proportion of persons testing negative among non-affected individuals
12
Non-affected persons
Test result+-
False positive (FP)
True negative (TN)
Estimating the specificity of a test
• Identify non-affected individuals
– Negative with a gold standard
– Unlikely to be infected
• Obtain a wide panel of samples that are representative of the population of non-affected individuals
• Test the non-affected individuals
• Estimate the proportion of non-affected individuals that are negative with the test
13
Example: Estimating the specificity of a new ELISA IgM test for acute Q-fever
• Identify persons without Q-fever– Persons without sign and symptoms of the infection
– Persons at low risk of infection, negative with gold standard (IgM Immunofluorescence Assay)
• Obtain a wide panel of samples that are representative of the population of individuals without Q-fever
• Test the persons without Q-fever
• Estimate the proportion of persons without Q-fever that are negative with the new ELISA IgM test
14
Specificity of a new ELISA IgM testfor acute Q-fever
Persons without acute Q-fever
ELISA IgM test result+ False positive (FP) 10
- True negative (TN) 190
200
Specificity =TN / (TN + FP)
190 / 200 = 95%
15
What factors influence the specificity of a test?
• Characteristics of the affected persons? NO: The specificity is estimated on a population of non-
affected persons
• Characteristics of the non-affected persons? YES: The diversity of antibodies to various other antigens in
the population may affect cross reactivity or polyclonal hypergammaglobulinemia may increase the proportion of false positives
• Prevalence of the disease? NO: The specificity is estimated on a population of non-
affected persons
Specificity is an INTRINSIC characteristic of the test
16
17
Disease
TNSp = TN + FP
Performance of a test
FP
TN
No
TPSe = TP + FN
Test
TP
FN
Yes
+
-
To whom sensitivity and specificity matters most?
INTRINSIC characteristics of the test
► To laboratory specialists!
18
19
0 5 10 15 20
Quantitative result of the test
Distribution of quantitative test results among affected and non-affected people
TN
Non-affected:
Affected:
TP
Nu
mb
er
of
peop
le t
este
d Threshold forpositive result
Ideal situation
20
0 5 10 15 20
TN TP
FN FP
Distribution of quantitative results among affected and non-affected people
Non-affected:Threshold forpositive result
Quantitative result of the test
Nu
mb
er
of
peop
le t
este
d Affected:
Realistic situation
21
TN TP
FN
FP
Non-affected:
Affected:Threshold forpositive result
Effect of Decreasing the ThresholdN
um
ber
of
peop
le t
este
d
Quantitative result of the test0 5 10 15 20
22
Disease
FP
TN
No
Effect of Decreasing the Threshold
TNSp = TN + FP
TPSe = TP + FN
TestTP
FN
Yes
+
-
23
0 5 10 15 20
TNTP
FN
FP
Non-affected:
Affected:Threshold forpositive result
Nu
mb
er
of
peop
le t
este
d
Quantitative result of the test
Effect of Increasing the Threshold
24
Effect of Increasing the Threshold
Disease
FP
TN
No
TNSp = TN + FP
TPSe = TP + FN
Test
TP
FN
Yes
+
-
25
Performance of a test and threshold
• Sensitivity and specificity vary in opposite directions when changing the threshold (e.g. the cut-off in an ELISA)
• The choice of a threshold is a compromise to best reach the objectives of the test– consequences of having false negatives?– consequences of having false positives?
26
Using several tests
• One way out of the dilemma is to use several tests that complement each other
• First use test with a high sensitivity(e.g. screening for HIV by ELISA, or for syphilis by TPHA)
• Second use test with a high specificity(e.g. confirmation of HIV or syphilis by western blot)
27
ROC curves
• Receiver Operating Characteristics curve
• Representation of relationship between sensitivity and specificity for a test
• Simple tool to:– Help define best cut-off value of a test
– Compare performance of two tests
28
Prevention of blood transfusion malaria:Choice of an indirect IFA threshold
00 20 40 60 80 100
20
40
60
80
100
IFA Dilutions
1/101/201/40
1/80
1/160
1/320
1/640
100 - Specificity (%): Proportion of false positives
Sensitivity (%)
29
0 25 50
75 100
Comparison of performance of IFA and ELISA IgM tests for detection of acute Q-fever
IFAELISA
0
20
40
60
80
100
100 - Specificity (%)
Sensitivity (%)
Area under the ROC curve (AUC)
30
2. Performance of a test in a population
How well does the test perform in a real population?
Status of persons
Affected Non-affected
TestPositive True + False + A+B
Negative False - True - C+D
A+C B+D A+C+B+D
• The test is now used in a real population
• This population is made of – Affected individuals
– Non-affected individuals
• The proportion of affected individuals is the prevalence
31
Predictive value of a positive test
The predictive value of a positive test is the probability that an individual testing positive is
truly affected
Proportion of affected persons among those testing positive
32
Positive predictive value (PPV) of a test
Status of persons
Affected Non-affected
TestPositive A B A+B
Negative C D C+D
A + C B+D A+C+B+D
PPV = A / (A+B)
This is only valid for the sample of specimens tested
33
What factors influence the positive predictive value of a test?
• Sensitivity?YES: To some extend.
• Specificity?YES: The more the test is specific, the more it will be negative
for non-affected persons (less false-positive results).
• Prevalence of the disease?YES: Low prevalence: Low pre-test probability for positives.
The test will pick up more false positives.YES: High prevalence: High pre-test probability for positives.
The test will pick up more true positives.
Status of persons
Affected Non-affected
TestPositive A B A+B
Negative C D C+D
A + C B+D A+C+B+D
34
Positive predictive value of a test according to prevalence and specificity
0102030405060708090
100
0 10 20 30 40 50 60 70 80 90 100
Prevalence (%)
PVP % 70%80%90%95%
Specificity
PPV (%)
35
Predictive value of a negative test
The predictive value of a negative test is the probability that an individual testing negative
is truly non-affected
Proportion of non-affected persons among those testing negative
36
Negative predictive value (NPV) of a test
Status of persons
Affected Non-affected
TestPositive A B A+B
Negative C D C+D
A+C B+D A+C+B+D
NPV = D / (C+D)
This is only valid for the sample of specimens tested
37
What factors influence the negative predictive value of a test?
• Sensitivity?YES:
The more the test issensitive, the more it captures affected persons (less false negatives).
• Specificity?YES: But to a lesser extend.
• Prevalence of the disease?YES: Low prevalence: High pre-test probability for negatives.
The test will pick up more true negatives.YES: High prevalence: Low pre-test probability for negatives.
The test will pick up more false negatives.38
Status of persons
Affected Non-affected
TestPositive A B A+B
Negative C D C+D
A+C B+D A+C+B+D
Negative predictive value of a test according to prevalence and sensitivity
Sensitivity
0
10
20
30
40
50
60
70
80
90
100
Prevalence (%)
PVN %
70%80%90%95%
NPV (%)
39
Relation between predictive values and sensitivity (Se), specificity (Sp), prevalence (Pr)
(1-Se)Pr + Sp(1-Pr)
Disease
(1-Sp)(1-Pr)Se Pr
NoYes
Se Pr + (1-Sp)(1-Pr)
Pr 1-Pr
Sp(1-Pr)(1-Se)Pr
Test
+
-
40
41
Calculate PPV and NPV
Pr) Sp)(1 (1 Pr Se
Pr Se PPV
Pr Se)(1Pr)-Sp(1
Pr)-Sp(1 NPV
42
Relation between predictive values and sensitivity / specificity
Increasing specificity increasing PPV
Increasing sensitivity increasing NPV
Pr) Sp)(1 (1 Pr Se
Pr Se PPV
Pr Se)(1Pr)-Sp(1
Pr)-Sp(1 NPV
43
Relation between predictive values and prevalence
Increasing prevalence increasing PPV
Decreasing prevalence increasing NPV
Pr) Sp)(1 (1 Pr Se
Pr Se PPV
Pr Se)(1Pr)-Sp(1
Pr)-Sp(1 NPV
44
• ELISA IgM test– Sensitivity = 98%– Specificity = 95%
• Population in low endemic area– Prevalence = 0.5%
• Patients with atypical pneumonia– Prevalence = 20%
• 10,000 tests performed in each group
Example: Screening for acute Q-fever in two settings
45
Example: Screening for acute Q-fever in a population in a low endemic area
Prevalence = 0.5%
PPV = 8.97%NPV = 99.98%
IgM ELISA test sensitivity = 98% IgM ELISA test specificity = 95%
Q-fever
Yes No Total
IgM ELISA
+ 49 497 546
- 1 9,453 9,454
50 9,950 10,000
46
Example: Screening for acute Q-fever in patients with atypical pneumonia
Prevalence = 20%
PPV = 83.05%NPV = 99.48%
IgM ELISA test sensitivity = 98% IgM ELISA test specificity = 95%
Q-fever
Yes No Total
IgM ELISA
+ 1,960 400 2,360
- 40 7,600 7,640
2,000 8,000 10,000
To whom predictive values matters most?
• Look at denominators!– Persons testing positive
– Persons testing negative
► To clinicians – probability that a individual with a positive test is really sick?– probability that a individual with a negative test is really healthy?
► To epidemiologists!– proportion of positive tests corresponding to true patients?– proportion of negative tests corresponding to healthy subjects?
47
48
• Sensitivity and specificity matter to laboratory specialists– Studied on panels of positives and negatives – Intrinsic characteristics of a test
• Capacity to identify the affected• Capacity to identify the non-affected
• Predictive values matter to clinicians and epidemiologists– Studied on homogeneous populations– Dependent on the disease prevalence– Performance of a test in real life
• How to interpret a positive test• How to interpret a negative test
Summary
Where will you do your rain dance?
Here?
There?