Epcm l18-19 assessing tests

EPIDEMIOLOGY & COMMUNITY MEDICINE
WRITTEN AND COMPILED BY DR. SAMER RASTAM MD, PHD
Third year
Level 6
Lectures 17, 18
Assessing the Validity and Reliability
of Diagnostic and Screening Tests

• How to define, calculate, and use sensitivity,
specificity, predictive values of positive and
negative test for a diagnostic test
• How predictive values differ from sensitivity and
specificity
• How to apply basic test characteristics to solve a
clinical diagnostic problem

• To provide appropriate and effective health care, it is necessary
to distinguish between those who have the disease and those
who do not, and that what tests do.
• Medical test is any medical procedure used to detect and
monitor diseases. It may be:
– History & physical examination
– Imaging tests
– Physiological (ECG, EEG,)
– Clinical pathology tests
– Histopathology tests
– Example: In cardiac stress test, ST segment depression for more
than 1 mm is a test to check angina pectoris with a sensitivity of
65% and specificity of 89%

1. Diagnosis (Establishing, differential diagnosis, evaluating): in
people suspected to have a disease.
2. Screening: in healthy people to check the possibility of a
subclinical disease.
3. Monitoring: in patients to monitor the development of a
disease.

• For patients: painless, non-invasive, fast, ...
• For labs: easy, fast, don't need extensive experience, don't
need special instruments, cost-effective, ….
• For doctors:
– Positive results mean presence of the disease
– Negative means absence of the disease
• Do not exist

• The treatment threshold is the probability above which the
diagnosis is so likely you would treat the patient without
further testing.
• The test threshold is the probability below which the diagnosis
is so unlikely it is excluded without further testing.

• Diagnostic tests are necessary when the pretest
probability of disease is in the middle. A really useful
test shifts the probability of disease so much that the
posttest probability (the probability of disease after
the test is done) crosses one of the thresholds

• Examples:
– Ms. A, a 19-year-old woman, who complains of 30 seconds of
sharp right-sided chest pain after lifting a heavy box. The pretest
probability of cardiac ischemia is so low that no further testing is
necessary
– Mr. B, a 60-year-old man who smokes and has diabetes,
hypertension, and had a 15 minutes of crushing substernal chest
pain accompanied by nausea and diaphoresis, with an ECG
showing ST-segment elevations in the anterior leads. The pretest
probability of an acute MI is so high you would treat without
further testing, such as cardiac enzymes

• Thresholds: are dependent on
–What is the harm or risk of treating someone who
doesn't have the disease?
–What is the harm or risk of not treating someone
who actually does have the disease?
–What are the harms or risks of doing the test?

• Test’s accuracy is considered in relation to the best way of
knowing whether the diseases is truly present or not. This is
usually called gold standard or criterion standard or reference
standard
– It may be simple and easy like throat culture for group A streptocooci.
– Or it may be expensive or risky such as biopsy, surgical exploration,
and even autopsy.
• For diseases that are not self-limited and become overt in
years , such as cancers , the gold standard is the results of
follow-up.

• Reliability (precision):
– The extent to which multiple measurements (questionnaire/
instrument) of a characteristic are in agreement.
– For reliable test, multiple measurements will give the same results.
– Non- reliable test: a test with a random error
• Accuracy (unbiased) :
– The ability of a test to produce a true value for the measurements.
– Biased test (Diminished accuracy): a test with a non-random
(systematic) error.

• Validity:
–The extent to which a test measures what it was
design to measure.
–However, a reliable and unbiased test or
measurement is not necessarily valid or
accurate.
–For example, it would be possible to measure the
circumference of a person’s skull with great
reliability and precision, but this would not be a
valid assessment of intelligence!

Non- reliable Reliable Non- reliable Reliable
Unbiased Biased Biased Unbiased

• Even a high-quality, valid, and reliable measurement
does not in itself permit a diagnosis to be made.
• Doing this requires knowing the measurement’s
range of values among normal, healthy people. This
range is called the normal range or reference range,
and the limits of this range are the reference values.
• The normal range of a biomedical variable is often
arbitrarily defined as the middle 95% of the normal
distribution—in other words, the population mean
plus or minus two standard deviations.

• Ideally: positive results means presence of a the disease, and
negative results means absence of the disease.
• The results of diagnostic accuracy studies are commonly
summarized in two by two (2 × 2) tables

• True positive: Patients who test positive on the index test and
who have the target disorder.
• False positive: Patients who test positive on the index test but
do not have the target disorder
• False negative: Patients who test negative on the index test
and who have the target disorder.
• True negative: Patients who test negative on the index test but
do not have the target disorder

• Sensitivity is the probability of a positive (abnormal) test result
in individuals with the target disorder.
• Sensitivity = a / (a+c)
• Specificity is the probability of a negative (normal) test result in
participants without the target disorder.
• Specificity = d / (b+d)

• Sensitivity and specificity tell you how many of the subjects
with or without the target disorder a test can identify.
• Denominators: are people with disease or without the disease
(we already know that someone have/haven't the disease)
• SnNout: High sensitivity allows ruling out.
• If the sensitivity is 100% → no false negative results → negative
test means absence of the disease.
– The complaint of "dyspnea on exertion" in the diagnosis of congestive
heart failure (CHF) (sensitivity 100% , spec 17%)

• SpPin: High specificity allows ruling in.
• If the specificity is 100% → no false positive results → positive
test means presence of the disease
– Gallop murmur (3rd heart sound or S3) in the diagnosis of CHF
(Specificity 99% , sensitivity 24%)
• In a screening program for HIV in residents; what is more
appropriate: a test with high sensitivity, or a test with high
specificity ?

• Use of sensitive test
– When there is an important penalty for missing a diagnosis; i.e.
dangerous (to the patient or the community) , but treatable
conditions. Examples include Tuberculosis, HIV, Hodgkin’s lymphoma
– During the early stage of a diagnostic workup to rule out diseases
with a negative result of highly sensitive test.
– Usually these tests are more useful when negative.
• Use of specific tests
– Use to confirm (to rule in) a diagnosis that has been suggested by
other data
– Used when false positive results can harm the patient, i.e. before
chemotherapy
– More useful when positive

• In clinical practice, sensitivity and specificity are not useful,
because generally you don’t know if the patient does or does
not have the condition.
• Positive predictive value (PPV) is the probability of the target
disorder in participants with a positive (abnormal) test result.
• PPV = a / (a+b)
• Negative predictive value (NPV) is the probability of absence
of the target disorder in participants with a negative (normal)
test result.
• NPV = d /(c+d)
• Denominators: are people with a positive or negative test
results

• Sensitivity = 950 / 1000 = 95% (95% of people with disease will have
positive results).
• Specificity = 8100/9000 = 90% (90% of people without the disease
will have negative results).
• Positive predictive value = 950/1850 = 51.4% (a positive result means
that the probability of having the disease is 51.4%
• Negative predictive value = 8100/8150 = 99.4% (a negative results
means that the probability of not having the disease is 99.4%
• Prevalence of disease = 1000/10000 = 10%
• Positive results % = 1850/10000 = 18.5%

• A new test for a disease that affects 10% of population was
developed. This test is positive in 95% of patients with the
disease (sensitivity) , and negative in 90% of patients without
the disease (specificity)
• If a patient test was positive, what is the probability that this
patient is really have the disease (PPV) , and if the test was
negative, what is the probability that this patient do not have
the disease (NPV)

• Prevalence (pre-test probability - prior probability)
– The prevalence of the target disorder in the sample is the number of
patients with the target condition in the entire sample
– = (a+c) / (a+b+c+d)
• The probability of disease given a specific result is called Post-
test probability (posterior probability) (PPV and NPV)
• In clinical practice, predictive values are more useful, because
generally you don’t know if the patient does or does not have
the condition
• predictive values are directly dependent on the prevalence

• Neuroblastoma is the most common form of malignant solid
tumor during childhood.
• A program for screening for neuroblastoma has been adopted
as a national policy, and the program has been conducted since
1984
• Urine were tested for homovanillic acid and vanilmandelic acid
(metabolites of catecholamines produced by neuroblastoma) by
high-performance liquid chromatography. A test with about
73% sensitivity and 90% specificity.
• 2003 review: There was sufficient evidence that the current
method of screening led to overdiagnosis of neuroblastoma
and that there was insufficient evidence that the program
reduced the rate of death from the disease

• Only 7.3 of 10,000 positive results are for patients with
neuroblastoma.
• Why this test is still useful in pediatrics ??!!

• PPV (probability of a disease after a positive result) is
dependent on the prevalence of the disease.
• The same test may be useful in clinical situation, but useless in
screening

• There is a trade-off between sensitivity and specificity
• Cutoff point: is the point that differentiate between normal
and abnormal. The location of this point will affect sensitivity
and specificity in opposite directions.
• Example: using of prostate specific antigen
(PSA) in screening for prostate cancer.
• If we use very low cut-off point, e.g. 3 ng/ml
and above: sensitivity will be 100% , but
specificity is just 60%
• If we use a high cut-off point, e.g. 10 ng/ml
an above: specificity will be 93% , but
sensitivity is just 54 %

• ROC curve: receiver operating characteristic curve
• A way to express the relationship between sensitivity and
specificity.
• Constructed by plotting the true positive rate (sensitivity)
against the false positive rate (1-specificity) over the range of
cut-off points.
• Discriminative tests crowd toward the upper left corner of the
ROC curve, tests that’s perform less well have curves closer to
the diagonal running from the lower left to the upper right.
• Best cut-off points is at or near the shoulder of the ROC curve,
unless there are clinical reasons for minimizing either false
negative or false positive results.

ROC Curve: for PSA
levels in screening
prostate cancer

• Overlap between normal and abnormal values:
– In many cases there is no clear cut point between normal and
abnormal values
– Cross-reactivity between normal and abnormal antigens.
• Biological variability:
– Most time we measure not disease directly, but something that is
associated with, but the association is not perfect.
– In neuroblastoma, only 90% of them secrete catecholamine, so
whatever the test to measure them in blood or urine will not be 100%
sensitive
• Measurement errors

1. EPIDEMIOLOGY; Leon Gordis; 5th Ed; 2013
2. Basic epidemiology; R Bonita, R Beaglehole, and T Kjellström;
2nd Ed; 2006
3. Clinical Epidemiology- The Essentials, Robert Fletcher (2005)
4. Symptom to Diagnosis- An Evidence Based Guide, Scott Stern
(2009)
5. Diagnostic Tests Toolkit, Matthew Thompson, BMJ Books (2011)

Epcm l18-19 assessing tests

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Epcm l18-19 assessing tests

Similar to Epcm l18-19 assessing tests (20)

More from Dr Ghaiath Hussein

More from Dr Ghaiath Hussein (20)

Recently uploaded

Recently uploaded (20)

Epcm l18-19 assessing tests