This document summarizes key points from a presentation on assessing the performance of diagnostic tests:
1. Screening tests are used to distinguish healthy from infected animals for disease surveillance and certification of disease-free herds. Issues include false positives and false negatives.
2. The accuracy, sensitivity, specificity, and predictive values of diagnostic tests are important metrics to consider. Sensitivity measures the ability to detect true infections, while specificity measures ability to detect true non-infections.
3. Testing multiple animals in parallel or series can impact overall test sensitivity and specificity. Testing in parallel increases sensitivity but decreases specificity, while testing in series has the opposite effect.
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
Assessing the performance of diagnostic tests
1. Assessing the performance of diagnostic tests
Johanna Lindahl
Laboratory review meeting
Ouagadougou, Burkina Faso
19 December 2019
2. Screening tests
• To distinguish apparently healthy animals from
infected animals
• For disease surveillance
– To measure disease burden in animal
populations
– To certify that an animal herd or region is free
from a specific disease
– For early detection of infection or sub-clinical
disease in animals
– For making management decisions
Issues
• False positives
• False negatives
9. True prevalence
• If all you have is your test result and estimates
for Se/Sp
True Prevalence = AP*Se+(1-AP)(1-Sp)
10. Accuracy
• Proportion of infected and non-infected
animals correctly classified by the test
TP + TN / N
(a+d)/(a+b+c+d)
11. Sensitivity
• Ability of a test to detect infected animals
• Proportion of infected animals that test
positive
TP / Infected
Infected = TP + FN
a/(a+c)
13. Specificity
• Ability of a test to detect non-infected
animals
• Proportion of non-infected animals that
test negative
TN / Not infected
Not Infected = TN + FP
d/(b+d)
15. Picking a test using Se/Sp
If the outcome will be expensive or catastrophic:
Minimize false positives
• Test with high specificity
If the penalty for missing a case is high (e.g., the
disease is fatal, or disease easily spreads):
Maximize true positives
• Test with high sensitivity
16. Predictive values
• It reflects the way test results are used in
the clinic, hospital, free-living population
• If the test result is negative, what is the
probability that this patient is non-infected
• If the test result is positive, what is the
probability that this patient is infected
• It is used as a method for test selection
• It is affected by the SE and SP of the test,
as well as disease prevalence (I+)
17. Negative predictive value
• Proportion of non-infected animals among
those that test negative
TN / Test negative
Test negative = TN + FN
c/(c+d)
19. Positive predictive value
• If the test result is positive, what’s the probability
that this patient is infected?
• If we screen a population, what’s the proportion
of animals who have the infection will be
correctly identified?
• Proportion of infected animals among those that
test positive
TP / Test positive
Test positive = TP +
FP
a/(a+b)
21. Selection of diagnostic tests
• High SE and Negative Predictive Value
when it is important to reduce the number
of FN
• Avoid introduction of disease
22. Selection of diagnostic tests
• High SP and Positive Predictive Value
when it is important to reduce the number
of FP
• To confirm a diagnosis
• Avoid unnecessary elimination of animals
24. Testing in parallel
• The results of two or more tests must be
negative
• To increase SE and Negative Predictive Value
• The goal is to maximize the probability that
subjects with the disease (true positives) are
identified (increase sensitivity)
• Consequently, more false positives are also
identified (decrease specificity)
• Net sensitivity = Se1 + Se2 –(Se1*Se2)
• Net specificity = Sp1*Sp2
25. Testing in parallel
Infected Non-infected Total Test 1
Test positive 7 9 16 SE = 70%
Test negative 3 139 142 SP = 94%
10 148 158
Test 2
Test positive 8 1 9 SE = 80%
Test negative 2 147 149 SP = 99%
10 148 158
Net sensitivity = 0.7+ 0.8–(0.7*0.8) = 0.94
Net specificity = 0.94*0.99=0.93
26. Testing in series
• The results of two tests must be positive
• Only use second test when the first test is
positive
• To increase SP and Positive Predictive
Value
• Testing in series leads to a net loss in
sensitivity and a net gain in specificity
27. Testing in series
Infected Non-
infected
Total Test 1
Test
positive
SE = 70%
Test
negative
SP = 94%
10 148 158
28. Testing in series
Infected Non-
infected
Total Test 1
Test
positive
7 9 16 SE = 70%
Test
negative
3 139 142 SP = 94%
10 148 158
29. Testing in series
Infected Non-
infected
Total Test 1
Test positive 7 9 16 SE = 70%
Test negative 3 139 142 SP = 94%
10 148 158
Test 2
Test positive SE = 80%
Test negative SP = 99%
7 9 16
30. Testing in series
Infected Non-infected Total Test 1
Test positive 7 9 16 SE = 70%
Test negative 3 139 142 SP = 94%
10 148 158
Test 2
Test positive 6 0 6 SE = 80%
Test negative 1 9 10 SP = 99%
7 9 16
Overall SE = (10 - 3 - 1 = 6) / 10 = 60%
Overall SP = (139 + 9 = 148) / 148 = 100%
Overall positive predictive value = 6 / 6 = 100%
31. Testing in series – negative sample
Infected Non-infected Total Test 1
Test positive 70 90 160 SE = 70%
Test negative 30 1390 1420 SP = 94%
100 1480 1580
Test 2
Test positive 24 14 38 SE = 80%
Test negative 6 1376 1382 SP = 99%
30 1390 1420
Overall SE = (70+24 = 94) / 100 = 94%
Overall SP = (1480-90-14 = 1376) / 1480 = 93%
33. Policy implications –testing in series
Infected Non-
infected
Total RBPT
Test positive 40 20 60 SE = 100%
Test negative 0 1940 1940 SP = 99%
40 1960 2000
C-ELISA
Test positive 40 0 40 SE = 100%
Test negative 0 20 20 SP = 99.9%
40 20 60
Overall SE = (40 - 0 = 40) / 40 = 100%
Overall SP = (1940 + 20 = 1960) / 1960 = 100%
Overall positive predictive value = 40 / 40 = 100%
34. Policy implications
Plan A
C-ELISA
2,000 samples x $5 = $10,000
TOTAL: $10,000
Plan B
RBPT
2,000 samples x $1 = $2,000
C-ELISA
60 samples x $5 = $300
TOTAL: $2,300
35. Relationship between disease prevalence and
positive predictive value
• SE=99% and SP=95%
Disease No
Disease
Total PPV
1% + 99 495 594 17%
- 1 9405 9406
Total 100 9900 10000
10% + 990 450 1440 69%
- 10 8550 8560
Total 1000 9000 10000
36. Measuring diagnostic test performance
C-ELISA
Positive Negative Total
RBPT Positive 35 6 41
Negative 0 1976 1976
Total 35 1982 2017
Kappa = 0.92
40. @OneHealthHORNHORN@liverpool.ac.ukwww.OneHealthHORN.net
Factors affecting antimicrobial susceptibility
measurements
• Medium type (Mueller-Hinton, Iso-sensitest, Sensitest
medium)
• Medium manufacturer
• Lot-to-lot variation for both medium and disks
• Effect of additives (e.g. blood)
• Inoculum size and concentration
• Incubation conditions (temperature and duration)
• Human factors (e.g. preparation of dilutions)
Quality control is
essential
41. How is quality control done?
<Udfyld sidefod-oplysninger her>
• Reference strains of different species should routinely be
included in the testing
• The MIC (or inhibition zone diameter) of the reference strain
has to fall within a given range to validate the test
• If not, the test is not validated and should be repeated
Send also
samples
regularly to
diagnostic labs
to validate
results
42. @OneHealthHORNHORN@liverpool.ac.ukwww.OneHealthHORN.net
QC Antimicrobial Susceptibility Testing - Module 8 42
AST Methods Interpretation
• agar disk diffusion method provides qualitative interpretive category
results of susceptible, intermediate, and resistant
• microdilution and agar gradient diffusion methods provide a
quantitative result, a minimum inhibitory concentration
44. QC Antimicrobial Susceptibility Testing -
Module 8
44
44
Clinical and Laboratory Standards Institute
French Society of Microbiology
British Society for Antimicrobial Chemotherapy
References
45. QC Antimicrobial Susceptibility Testing -
Module 8
45
45
Where errors can occur in susceptibility
testing
•media
•antimicrobials
•inoculum
•incubation
•equipment
•interpretation
46. QC Antimicrobial Susceptibility Testing -
Module 8
46
46
Reference Strains
E. coli ATCC 25922
S. aureus ATCC 25923
P. aeruginosa ATCC 27853
QC organisms must be obtained from
reputable source
Use specific QC organisms to test different
groups of “drug-bug” combinations
50. QC Antimicrobial Susceptibility Testing -
Module 8
50
50
Measuring Conditions
RulerCalipers
read with good light, and from the back of the plate
zone size reading is drug specific
magnification may help
millimeters matter
52. @OneHealthHORNHORN@liverpool.ac.ukwww.OneHealthHORN.net
QC Antimicrobial Susceptibility Testing - Module 8 52
Patient results may be incorrect if:
•the organism was misidentified
•a clerical error was made
•inappropriate choice of antimicrobials were tested
and reported
•the wrong patient’s sample was examined
•the wrong test was ordered
•the sample was not preserved properly
53. This presentation is licensed for use under the Creative Commons Attribution 4.0 International Licence.
better lives through livestock
ilri.org
ILRI thanks all donors and organizations which globally support its work through their contributions
to the CGIAR Trust Fund