2. Lesson objectives
At the end of the session the participants would
be able to
A.Enlist the indicators for evaluation of a
diagnostic test
B.Describe and calculate sensitivity, specificity &
predictive power
C.Describe the utility of ROC curve
D.Describe the utility of Kappa Statistics
8. Test
Result
Disease Tot
Yes No
Positive a b a + b
Negative c d c + d
Total a + c b+ d n
Sensitivity
Ability of test to
detect disease
in those who
actually have it
= a / (a+ c)
= True + ve / Having
Disease
A test that yields
minimum
False Negatives is
most seNsitive
9. Test Disease Total
Yes No
+ ve 90 10 100
-ve 20 180 200
Total 110 190 300
Example
Sensitivity
= 90 / 110
= 0.8182
=81.82 %
10. Test
Result
Disease Tot
Yes No
Positive a b a + b
Negative c d c + d
Total a + c b+ d n
Specificity
Ability of test to
detect absence of
disease
in those who
actually do not
have it
= d / (b + d)
= True – ve / Not
having disease
A test that yields
minimum
False Positives is
most sPecific test
11. Test Disease Total
Yes No
+ ve 90 10 100
-ve 20 180 200
Total 110 190 300
Example
Specificity
= 180 / 190
= 0.9473
= 94.73 %
12. Test
Result
Disease Tot
Yes No
Positive a b a + b
Negative c d c + d
Total a + c b+ d n
Predictive Power of
Positive Test
Probability that a
person will have
disease if test is
positive
= a / ( a + b )
= True + ve / Test + ve
13. Test Disease Total
Yes No
+ ve 90 10 100
-ve 20 180 200
Total 110 190 300
Example
Pr. Power
(+ve Test)
= 90 / 100
= 0.9000
= 90.00 %
14. Test
Result
Disease Tot
Yes No
Positive a b a + b
Negative c d c + d
Total a + c b+ d n
Predictive Power of
Negative Test Probability that a
person will NOT have
disease if test is
negative
= d/ ( c + d )
= True -ve / Test - ve
Predictive power also
depends on prevalence of
disease
15.
16. Test Disease Total
Yes No
+ ve 90 10 100
-ve 20 180 200
Total 110 190 300
Example
Pr. Power
(-ve Test)
= 180 / 200
= 0.9000
= 90.00 %
17. Reproducibility
• Ability of test to give consistent results when
repeated under similar conditions.
Observer errors
Errors in instrument / procedures
Biological variation
Improvement in reproducibility
Training of personnel
Standardization of procedures & instruments
Multiple testing and averaging
18. Points To Remember
• New test is always compared with established
test (Golden Standard)
• All measures are relative to that golden
standard and would change if golden standard
is changed
• If different cut-offs are set, sensitivity /
specificity changes ( if in one increases, the
other decreases)
• Observed indicators are subject to sampling
variation
19. Points to remember
1. Validity( Syn.= accuracy): ability of measurement
to be correct on an average.
2. Reproducibility( Syn.= precision, repeatability,
reliability): ability of measurement to give same
or similar result with repeated measurement
3. Economicity( Syn.= efficiency): extent to which
the expenditures on the test in clinical and public
health practice commensurate with the results
4. Acceptability, cost, ease of administration, technical
ease.
20. Points to remember
• Accuracy: How close the estimates of new test
are to the truth. Truth is what GOLD
STANDARD says. So, for accuracy, Gold
standard is a must.
• Reproducibility: How close are the repeated
estimates of the new test to each other. So,
for reproducibility, we must repeat the test in
similar circumstances. Gold standard is NOT
required here.
22. ROC Curve
• Receiver Operating
Characteristic Curves
Uses
1. Deciding cut-off points
2. Comparing two tests
Example
• If fasting blood sugar
(FBS) after 2 hrs is taken
as a NEW test and its
sensitivity / specificity is
compared against GTT
( the golden test)
• The sensitivity /
specificity would differ
at different cut-offs of
FBS levels
23. ROC Curve-
Example
• Y axis : Sensitivity
• X axis : (1- Specificity )
• Maximizing sensitivity
corresponds to some
large value on y-axis
• Maximizing specificity
corresponds to some
small value on x-axis
• Good first choice will be
the one corresponding
to upper left corner
24. ROC Curve- Area
Under Curve (AUC)
• An important measure
of accuracy of the test
• If AUC = 1, then curve
consists of two lines
Vertical: 0,0 to 0,1 and
horizontal : 0,1 to 1,1
(Indicated by bold blue
line) : The best, ideal
• If AUC =0.5: A diagonal
line results (0,0 to 1,1
Indicates a test that
cannot discriminate
between normal &
abnormal
25. ROC Curve- Area
Under Curve (AUC)
• Softwares can calculate
area under curve in a
given case
• Two or more tests can
be compared
statistically
• May consist of
measurements on same
individuals (Paired Test)
or on different
individuals (Un-paired
Test)
• Test with higher AUC
will be preferred .
26. Which Test Would You Select ?
AreaUC = 0.918
95 % CI: 0.878, 0.958
AreaUC = 0.803
95% CI: 0.737, 0.870
30. Kappa Statistics
• Is the agreement between two or test a
chance occurrence or otherwise
• “Significant agreement” means that element
is ruled out
31. O=Observed frequency of agreement = a+d = 90
E1=Expected agreement (a) = (a+b)x((a+c)/N=12.24
E2=Expected agreement(d) =(c+d)x(b+d)/N= 42.64
E= E1+E2=54.88
New Test
Gold Standard Total
Disease
Present
Disease
Absent
Disease Present a (30) b (6) a+b (36)
Disease Absent c (4) d (60) c+d (64)
Total a+c (34) b+d (66) N (100)
32. New Test
Gold Standard Total
Disease Present Disease
Absent
Disease Present a (30) b (6) a+b (36)
Disease Absent c (4) d (60) c+d (64)
Total a+c (34) b+d (66) N (100)
Kappa = [Obsv Agree – Exp Agree] / [Total-Exp Agree]
= [O-E ] / [N-E]
= [90-54.88] / [100-54.88] = 35.12 / 45.12
= 0.780
33. Interpretation of Kappa values
K Interpretation
< 0 No agreement
0 – 0.19 Negligible
0.20 – 0.39 Minimal agreement
0.40 – 0.59 Fair agreement
0.60 – 0.79 Good agreement
0.80 – 1.00 Excellent agreement
0.7783
34.
35. Further Evaluation
1. Likelihood Ratio
2. Post Test Odds and
Post Test
Probability
3. ROC Curves
• Predictive power of test is
affected by prevalence of
the disease
• Can be used for
combination of tests
• Can be used for several
levels of test
36. Further Evaluation
• Pre Test Probability: Prevalence rate :
Probability that a person will have the target
disorder before the test is carried out
• Pre Test Odds: The odds that the patient has
the target disorder before the test is carried
out (pre-test probability/ [1 - pre-test
probability]).
37. Further Evaluation
• Pre Test Probability:
Prevalence rate :
Probability that a
person will have the
target disorder before
the test is carried out
• Pre Test Odds: The odds
that the patient has the
target disorder before
the test is carried out
• Post Test Probability:
The proportion of
patients with that
particular test result
who have the target
disorder
• Post Test Odds: The
odds that the patient is
declared to have the
target disorder after the
test is carried out
38. Further Evaluation
• Pre Test Probability:
(PrTP) Prevalence rate :
Persons with disorder ÷
Persons without
disorder
[Range: 0-1]
• Pre Test Odds: (PrTO)
= PrTP / (1- PrTP)
[Range 0 to Infinity]
• Post Test Probability:
(PoTP)
• Post Test Odds: (PoTO)
Requires calculation of
Likelihood Ratio
39. Further Evaluation
• Likelihood Ratio Positive Test (LRP)
LRP = Sensitivity / (1- Specificity)
• Likelihood Ratio Negative Test (LRN)
LRN = (1- Sensitivity) / Specificity
• Post Test Probability Positive Test (PoTPP)
(Probability that test +ve will have target disease)
PoTPP = PrTP x LRP
• Post Test Probability Negative Test (PoTPN)
Probability that a test –ve will have target disease
PoTPN = PrTP x LRN
40. Likelihood Ratio & Post Test
Probability
Test Disease Tota
l
Yes No
+ ve 90 10 100
-ve 20 180 200
Tota
l
110 190 300
Likelihood Ratio (+ve test)
= Sensitivity / (1-Specificity)
= 0.8182 / (1- 0.9493)
= 16.13
Post Test Probability + test
= LHR+ / (1+LHR+)
= 16.17 / (1+16.17)
=0.9417 ( = 94.17 %)
Probability that test positive
will have disease is @ 94%
Pre Test Probability (Prevalence)
= 110 / 300 = 0.3666 ( 36.66%)
Pretest Odds
= 0.3666 / (1- 0.3666)
= 0.5789
41. Likelihood Ratio & Post Test
Probability
Test Disease Tota
l
Yes No
+ ve 90 10 100
-ve 20 180 200
Tota
l
110 190 300
Likelihood Ratio (-ve test)
= (1-Sensitivity )/ Specificity
= (1-0.8182) / (0.9493)
= 0.1915
Post Test Probability -ve test
= LHR v-ve / (1+LHR-ve)
= 0.1915 / (1+0.1915)
= 0.1607 (= 16.07%)
Probability that test –ve will
have disease is @ 16%
( Ideally this should be =0.0 %
Pre Test Probability (Prevalence)
= 110 / 300 = 0.3666 ( 36.66%)
Pretest Odds
= 0.3666 / (1- 0.3666)
= 0.5789