This presentation from RSNA explains how their similarities and differences have an impact on assessment, quality assurance and training in radiography. Read the blog at http://www.carestream.com/blog/2016/06/07/differences-between-computer-aided-diagnosis-and-quantitative-image-analysis/
Chandigarh Call Girls đ 7001035870 đ Genuine WhatsApp Number for Real Meet
Â
The Complementary Roles of Computer-Aided Diagnosis and Quantitative Image Analysis
1. The Complementary Roles of
Computer-Aided Diagnosis and
Quantitative Image Analysis:
Similarities and Differences in
Assessment, Quality Assurance
and Training
2. Berkman Sahiner1, PhD
Samuel G. Armato III2, PhD
Zhimin Huo3, PhD
Heang-Ping Chan4, PhD
Ronald M. Summers5, MD, PhD
Nicholas Petrick1, PhD
1: US Food and Drug Administration, Center for Devices and Radiological Health
2: University of Chicago, Department of Radiology
3: Carestream Inc.
4: University of Michigan, Department of Radiology
5: National Institutes of Health, Clinical Center
3. Introduction
ī§ Quantitative image analysis (QIA) and computer-aided
diagnosis (CAD) are closely linked
ī§ This exhibit reviews the methods developed for assessment,
quality assurance, and user training for CAD, and highlights
parallels and distinctions between CAD and QIA
ī§ Comparison between CAD and QIA in this computer exhibit
is performed through several example applications
developed by the authors or that are publicly available
4. What is CAD?
ī§ CAD systems incorporate pattern recognition and data
analysis capabilities* and are intended to
ī§ Mark regions of an image that may reveal specific abnormalities
and alert the clinician to these regions during image interpretation
(computer-aided detection (CADe) systems)**
ī§ Provide to the clinician an assessment of disease, disease type,
severity, stage, progression (computer-aided diagnosis (CADx)
systems)**
*Guidance for industry and FDA staff: âClinical performance assessment: Considerations for computer-assisted
detection devices applied to radiology images and radiology device data - premarket approval (PMA) and
premarket notification [510(k)] submissions," (FDA, 2012).
**N. Petrick et al., âEvaluation of computer-aided detection and diagnosis systems,â Med. Phys. 2013, 40:087001.
5. What is QIA?
ī§ QIA is the process of extraction of quantitative imaging
biomarkers from medical images, typically involving
computerized tools
ī§ A quantitative imaging biomarker is an objectively measured
characteristic derived from an in vivo image as an indicator of
normal biological processes, pathogenic processes, or response to a
therapeutic intervention*
*Sullivan DC et al., âMetrology standards for quantitative Imaging biomarkers,â Radiology 2015 Aug 12:142202.
(Epub ahead of print).
6. What is Quantitative Imaging?
ī§ Both QIA and quantitative imaging biomarkers are within
the larger context of quantitative imaging, defined as
ī§ the extraction of quantifiable features from medical images for the
assessment of normal [findings] or the severity, degree of change,
or status of a disease, injury, or chronic condition relative to normal
[findings]
ī§ Quantitative imaging includes the development,
standardization, and optimization of anatomical, functional,
and molecular imaging acquisition protocols, data analyses,
display methods, and reporting structures
*Quantitative Imaging Biomarkers Alliance. http://rsna.org/QIBA.aspx. Accessed Oct. 15, 2015.
7. How are CAD and QIA Similar?
ī§ Both aim at aiding clinicians through advanced image
analysis techniques
ī§ Both commonly use computer methods to extract features
ī§ CAD may use other patient-related information, e.g., age, gender,
risk factors, or non-image biomarkers
ī§ CAD features may include quantitative imaging biomarkers, but can
also be relative, ordinal features
ī§ User interaction may be somewhat more prevalent in QIA (e.g.,
semi-automated segmentation)
ī§ Both emphasize appropriate image acquisition protocols,
display methods, training, and reporting
8. How are CAD and QIA Different?
ī§ In CAD, the emphasis is on how the CAD output can provide
decision support to clinicians
ī§ In full assessment, it is required to demonstrate that a CAD system
aids the clinician
ī§ May integrate imaging and non-imaging information or biomarkers
ī§ Not necessary to establish a direct link between individual CAD
features and a specific disease condition
ī§ Standalone testing (performance of the CAD system alone) is
nonetheless important
ī§ In QIA, the emphasis is on the extraction of biomarkers
ī§ Bias and variability across devices, patients and time are major
considerations
9. Assessment, Quality Assurance,
and Training
ī§ The differences and similarities between CAD and QIA have
a number of implications for assessment, quality assurance
(QA), and training
ī§ We briefly summarize our work in CAD in these areas and
draw parallels with QIA
10. Assessment - CAD
ī§ CAD systems are typically assessed by evaluating the effect
of the system on clinicians
ī§ Reader performance assessment: Evaluate performance of a
clinician using CAD as part of the decision making process
ī§ Retrospective multiple-reader multiple case (MRMC) studies
ī§ Prospective field trials
ī§ Standalone performance of a CAD system is also useful as a
performance indicator, both in
ī§ System design phase and
ī§ Final system assessment
11. Assessment - CAD
ī§ A recent paper by AAPM outlined important considerations in CAD
assessment including*:
ī§ Data set selection
ī§ Representativeness of the test set for the targeted population
ī§ Reference standard and mark labeling
ī§ Disease status of a case (sometimes includes location information)
ī§ Standalone assessment metrics
ī§ True-positive fraction, false-positive fraction pairs
ī§ Receiver operating characteristic (ROC) curves
ī§ Location-specific ROC
ī§ Reader study design and analysis
ī§ Prospective versus retrospective studies
ī§ MRMC study design
ī§ Reader training for reader studies
*N. Petrick et al., âEvaluation of computer-aided detection and diagnosis systems,â Med. Phys. 2013, 40:087001.
12. Assessment - Quantitative Imaging
ī§ Most groups working in quantitative imaging are interested
in technical performance assessment
ī§ How a test performs in reference objects or subjects under
controlled conditions
ī§ Other quantitative imaging assessment
ī§ Clinical assessment
ī§ Clinical impact
13. Technical Assessment - Quantitative Imaging
ī§ Technical assessment can be performed using
ī§ Anthropomorphic phantoms
ī§ Large number of replicate measurements easily obtained
ī§ The same phantoms can be used repeatedly
ī§ Reference standard easier to obtain
ī§ Phantom realism may be limited
ī§ Generalizability to human data may be limited because of limited
abnormality variation and complexity
ī§ Patient data
ī§ More realistic
ī§ Number of replicate measurements may be limited
ī§ Reference standard more difficult to obtain
ī§ Number of available patients usually limited
14. Technical Assessment - Quantitative Imaging
ī§ Important considerations include*
ī§ Measurand/Reference
ī§ True value of the quantity intended to be measured
ī§ Bias
ī§ Difference between the expected value of the biomarker and the measurand
ī§ Linearity
ī§ Is a change in the measurand reflected as a proportional change in the biomarker?
ī§ Repeatability
ī§ Ability to repeatedly measure the same feature under identical or near-identical
conditions
ī§ Reproducibility
ī§ Ability to measure the same feature under different conditions expected in
ī§ A preclinical study
ī§ Clinical trial
ī§ Clinical practice
D.L. Raunig et al., âQuantitative imaging biomarkers: a review of statistical methods for technical performance
assessment,â Stat Methods Med Res. 2015, 24:27-67.
15. Clinical Assessment - Quantitative Imaging
ī§ Clinical assessment of a QIA method
ī§ How does the method help clinicians in diagnosis or treatment?
ī§ Recommendations developed for clinical assessment of CAD
may be useful for the clinical assessment of quantitative
imaging biomarkers after proper modifications
ī§ Methodology for prospective and retrospective studies
ī§ MRMC methods
ī§ Assessment metrics
16. Quality Assurance - CAD
ī§ CAD systems are medical devices, and can benefit from QA
procedures like all medical devices
ī§ Clinical image acquisition:
ī§ Follow manufacturerâs QA procedure for imaging device
ī§ Additional QA for CAD:
ī§ Assure functionality and performance of CAD device according to
vendorâs specifications
ī§ Assure consistency of CAD device performance over time
ī§ A recent paper by AAPM outlined important considerations
in CAD quality assurance*
*Z. Huo, et al., âQuality assurance and training procedures for computer-aided detection and diagnosis systems,â
Med. Phys. 2013, 40:077001.
17. Quality Assurance - Quantitative Imaging
ī§ Clinical image acquisition:
ī§ Follow manufacturerâs QA procedure for imaging device
ī§ Image acquisition for quantitative imaging:
ī§ QA procedures to reduce variability across devices, patients and
time
ī§ QA procedures to ensure that acquired images are of quantitative
quality
ī§ Calibration methods and phantom tests in addition to those
recommended by manufacturers for clinical image quality
18. Clinician Training - CAD
ī§ Inform clinicians about
ī§ Intended use of the CAD system
ī§ CAD systemâs limitations
ī§ Particular strengths and weaknesses of different CAD systems
ī§ E.g., with respect to lesion type, acquisition parameters
ī§ Although many clinicians and researchers have underlined
the need for clinician training for CAD
ī§ Additional awareness needs to be raised for proper use and
potential adverse effects of improper use
ī§ Best practices for CAD training are yet to be established
19. Clinician Training - Quantitative Imaging
ī§ Clinicians using the quantitative imaging tool in the clinic
should be qualified in accordance with local and national
requirements and standards
ī§ Training of operators in image acquisition and analysis can
influence the value of the extracted imaging biomarker
ī§ e.g., training of an operator in a semi-automated segmentation task
for a lesion volume biomarker
ī§ There is a small number of studies on clinician training and
its effect on the extracted quantitative imaging biomarker
20. âĸ The rest of the computer exhibit consists of a summary
and example applications developed by the authors or
that are publicly available
âĸ Click to go to the slide of your choice
âĸ Summary
âĸ Example applications
âĸ Click on the E on lower right from any slide to go to the list of
example applications
E
21. Summary
ī§ CAD and QIA fields have a number of shared attributes as
well as some differences
ī§ Methodologies developed in one field are likely to be
applicable in the other after careful consideration of the
differences
ī§ We expect that the development of standardization,
assessment and clinical use recommendations in these two
fields will strengthen each other
E
22. Example Applications
ī§ Example 1
ī§ CADe assessment: A
commercial CAD system
ī§ Example 2
ī§ Quantitative imaging
technical assessment: A
phantom study
ī§ Example 3
ī§ Quantitative volume change
analysis for treatment
response assessment of
head and neck neoplasms
on CT scans
ī§ Example 4
ī§ Computer-aided analysis of treatment
response of bladder cancers on CT scans
ī§ Example 5
ī§ Radiomic biomarkers and decision
support system
ī§ Example 6
ī§ Measurement of pleural effusion in
thoracic CT volumes
ī§ Example 7
ī§ Radiomics-based assessment of normal
lung tissue damage in radiation therapy
24. A Commercial CADe System
ī§ M-Vu Algorithm Engine
ī§ CADe device intended to aid radiologists in reading
screening mammograms
ī§ Mark areas for review by a radiologist, who
ī§ First reviews each case in the conventional manner,
ī§ then re-examines regions marked by the M-Vu system before
making a BI-RADS assessment
ī§ Pre-market approval (PMA) by the FDA in 2012
E
25. Test Data Set Selection
ī§ 140 positive and 140 negative cases
ī§ 11 sites in the United States
ī§ Academic, specialty, and community clinics.
ī§ Pre-defined inclusion and exclusion criteria
ī§ Sequential, eligible positive mammograms
ī§ A mammography exam for which a biopsy-proven breast cancer was
found within 15 months following the exam date
ī§ Eligible negative mammograms
ī§ A mammography exam for which
ī§ Breast cancer was not found within 15 months prior or 15 months after the exam
date
ī§ At least one associated subsequent negative mammogram was acquired at least 11
months after the exam date.
E
26. Reference Standard
ī§ Positive cases:
ī§ Pathology reports from biopsies or surgeries
ī§ Mammograms and radiology reports for lesion location
ī§ Each biopsy-proven lesion is outlined by site investigators
ī§ Negative cases
ī§ Radiology reports
E
27. Standalone Assessment: Sensitivity
ī§ A malignant region was considered to be detected if
ī§ A CAD mark centroid was inside the region, or
ī§ if the region centroid was inside a CAD mark
ī§ Having the cancer within the CAD mark, as above, can lead to an
optimistic estimate of sensitivity
ī§ A large mark may cover the lesion, but may not attract the attention of
the radiologist to the abnormality
ī§ When this is used as a detection criterion, other additional data are
evaluated to check if sensitivity is overestimated
ī§ A case was considered a true positive if
ī§ At least one malignant region was detected by CAD
E
28. Standalone Assessment: Sensitivity
Cases Sensitivity 95% Conf. Interval
Overall 140 79.3% (72.6% 86.0%)
Microcalcification 69 79.7% (70.2%, 89.2%)
Mass 86 81.4% (73.2%, 89.6%)
ī§ Sensitivity reported for two important sub-groups: Masses and
microcalcifications
ī§ Additional data on sensitivity with respect to
ī§ Lesion size
ī§ Lesion pathology (invasive, DCIS, other)
ī§ Breast density E
29. Standalone Performance: False
Positives
ī§ Additional data on
ī§ False-positives by breast density
ī§ Specificity (percentage of mammograms with no FP marks)
False-positives per mammo. 95% Conf. Interval
Total 0.418 (0.346, 0.490)
Microcalcification 0.088 (0.042,0.133)
Mass 0.330 (0.275, 0.385)
E
30. Pivotal Study
ī§ Multi-reader multi-case (MRMC) study to investigate the
effect of the CADe device on radiologistsâ performance
ī§ 21 radiologists from a variety of academic, specialty, and
community clinics located across the US
ī§ For each case:
ī§ Evaluate the case without CAD and record an assessment
ī§ View CAD marks
ī§ Record a "with CAD" assessment
E
31. Pivotal Study
ī§ With and without CAD assessments
ī§ Recall/do not recall patient
ī§ Screening BI-RADS (0, 1, 2, 3, 4a, 4b, 4c, 5)
ī§ Forced BI-RADS (1, 2, 3, 4a, 4b, 4c, 5)
ī§ Lesion findings
ī§ Lesion findings:
ī§ Laterality
ī§ Type (Mass, Architectural Distortion, Asymmetry,
microcalcification)
ī§ BI-RADS (1, 2, 3, 4a, 4b, 4c, 5)
ī§ Probability of Malignancy (POM) (0-100%) E
32. Pivotal Study - ROC Analysis
ī§ POM data analyzed using ROC
methodology
ī§ Area under the ROC curve (AUC)
ī§ AUC w/0 CAD = 0.885
ī§ AUC with CAD = 0.902
ī§ Difference statistically significant
with p=0.013 (DBM MRMC method)
E
33. Pivotal Study â Sens., Spec.
ī§ Radiologist per-case sensitivity and specificity based on
decision to recall
W/o CAD With CAD Difference P-value
Sensitivity 0.865 0.901 0.036 (0.014, 0.058) 0.002
Specificity 0.649 0.623 -0.026 (-0.039, -0.013) < 0.001
ī§ Sensitivity increased and specificity decreased with CAD
ī§ Typical in reader studies for most second-read CADe systems
ī§ Lack of CADe mark for a lesion does not dissuade a reader from recall
ī§ Marks for missed lesions and some false-positive marks add to the
number of recalled cases E
34. Summary
ī§ Components of CAD assessment
ī§ Test data set selection
ī§ Representative of the targeted population
ī§ Reference standard
ī§ Clear and accurate definitions of actual positives and negatives
ī§ Standalone performance
ī§ How does the CAD system alone perform in the task for which it is intended to help
the radiologist ?
ī§ Performance metrics, confidence intervals
ī§ Reader performance
ī§ What is the effect of the CAD system on readers?
ī§ Performance metrics
ī§ Performance with and without CAD
ī§ Difference, confidence intervals E
36. Designing an Analysis Plan*
Step 1: Define the quantitative imaging biomarker and its relationship to quantity to
be measured (measurand)
Step 2: Define question to be addressed
ī§ Hypothesis or bounds on performance
Step 3: Define the experimental unit
ī§ Lesion-level, patient-level
Step 4: Define statistical measures of performance
ī§ What are the metrics for bias, repeatability, reproducibility?
Step 5: Specify elements of statistical design
ī§ Reference, reproducibility conditions, etc.
Step 6: Determine the data requirements
ī§ Patient population, type of images, sample size, etc.
Step 7: Collect data and perform statistical analysis
*Raunig et al., Stat Methods
Med Res, 2014 E
39. Purpose of the Study
ī§ Evaluate technical performance of a nodule volume
estimation tool
E
40. Study Design
ī§ Phantom
ī§ Thorax phantom with
vascular insert
ī§ Synthetic nodules
ī§ 4 spherical nodules
ī§ 5, 8, 9, 10 mm
âĸ +100 HU
E
41. Study Design
ī§ Image collection protocol
ī§ 10 repeat acquisitions
Phantom
16Ã0.75 mm
100 mAs
0.75 mm 1.5 mm
Reconstruction
Acquisition
Detailed Detailed
E
42. Bias/Linearity
ī§ Visually assess data to define limits of quantitation
ī§ Evaluate means/variances
ī§ Is data transformation necessary/appropriate to stabilize the variance?
-60
-40
-20
0
20
40
60
5 mm 8 mm 9 mm 10 mm
Nodule Size
50
150
250
350
450
550
0.75 1.5 0.75 1.5 0.75 1.5 0.75 1.5
Slice Thickness (mm)
MeasuredValue(mm3
)
5 mm 8 mm 9 mm 10 mm
Nodule Size
MeasuredValue-Median(mm3) E
47. Summary
ī§ Main components of quantitative imaging biomarker technical
assessment
ī§ Bias/linearity analysis
ī§ Repeatability analysis
ī§ Reproducibility analysis
ī§ Others
ī§ Identification of significant factors/subgroups
ī§ âĻ.
ī§ Challenging to maintain consistency across studies
ī§ Phantom/clinical data
ī§ Transformation of data
ī§ Reference standard
ī§ Is test-retest data available?
ī§ Reproducibility conditions E
48. Example 3
Quantitative Volume Change Analysis
for Treatment Response Assessment of
Head and Neck Neoplasms on CT Scans
Ref. Hadjiiski L, Mukherji SK, Gujar SK, Sahiner B, Ibrahim M, Street E, Moyer J, Worden FP, Chan HP.
Treatment response assessment of head and neck cancers on CT using computerized volume analysis.
American Journal of Neuroradiology (AJNR) 2010;31(9):1744-1751.
E
59. Example 4
Computer-aided Analysis of Treatment
Response of Bladder Cancers on CT Scans
Ref. Hadjiiski L, Weizer AZ, Alva A, Caoili EM, Cohan RH, Cha K, Chan HP. Bladder cancers on CT: preliminary study
of treatment response assessment based on computerized volume analysis, WHO and RECIST Criteria.
American Journal of Roentgenology 2015, 205(2) pp 348-352.
E
62. Feature Descriptors
ī§ Descriptors automatically extracted from the segmented
lesions:
- 15 radiomic features (RF) based on pre- and post-treatment
changes in:
- volume (V)
- 5 gray level descriptors (GL)
- 9 shape descriptors (S)
- Selected features merged into a Combined Response Index (CRI)
E
64. Example 5
Radiomic Biomarkers and
Decision Support System
for Multiple Myeloma
Ref. Zhou C, Chan HP, Dong Q, Couriel DR, Pawarode A, Hadjiiski LM, Wei J. Quantitative analysis of MR
imaging to assess trestment response for patients with multiple myeloma by using dynamic intensity
entropy transformation 2015, Ahead of Print, 10.1148/radiol.2015142804
E
65. īž Multiple Myeloma (MM) â a cancer formed by
malignant plasma cells in the bone marrow
o MR radiomic biomarkers (+ clinical biomarkers)
- Staging
- Treatment response assessment
- Prognosis prediction
- Recurrence â early detection
E
66. Radiomic Biomarkers for MM Treatment Response
T1W sagittal views
VOI of vertebral bodies and intervertebral discs
3D Dynamic Intensity Energy Transformation (DIET)
Quantitative Energy Enhancement Value:
Response Index DIET response map E
p-qEEVvbr and m-DqEEVvbr
67. T1-weighted MR DIET response map
Pre-Bone Marrow
Transplant
55 days post-BMT Pre-BMT 55 days post-BMT
BMT Treatment Response Assessment
Responder
E
68. 56 days post-BMT Pre-BMT 56 days post-BMT
T1-weighted MR DIET response map
Non-responder
BMT Treatment Response Assessment
Pre-Bone Marrow
Transplant
E
69. Kaplan-Meier Survival Curve
predicted by p-qEEV biomarker
0 5 10 15 20 25 30 35
0.2
0.4
0.6
0.8
1
Time Since BMT (Months)
Survivalprobability
p-qEEV >= 10%
p-qEEV < 10%
E
Prediction of treatment response by
qEEV-based response index
Computer-Aided Decision Support
for Multiple Myeloma
70. Example 6
Measurement of Pleural Effusion
in Thoracic CT Volumes
Ref. Yao J, Bliton J, Summers RM. Automatic segmentation and measurement of pleural effusions on CT. IEEE
Transactions on Biomedical Engineering 2013, 60(7) pp 1834-1840.
E
71. Significance and Workflow
ī§ Pleural effusion:
ī§ Size, location and temporal change
can be significant for diagnosis and
patient care
ī§ Very time consuming to manually
measure in 3D chest CT
E
72. Segmentation and Surface Modeling
Segmentation
before
deformable
surface
modeling
Segmentation
after
deformable
surface
modeling
Segmented PE
surface before
deformable
surface modeling
Segmented PE
surface after
deformable
surface modeling
E
75. Example 7
Radiomics-Based Assessment of
Normal Lung Tissue Damage in
Radiation Therapy
E
Ref. Cunliffe AR, et al.: Lung texture in serial thoracic CT scans: Correlation with radiologist-defined severity of
acute changes following radiation therapy. Physics in Medicine and Biology 59: 5387â5398, 2014.
Ref. Cunliffe AR, et al.: Lung texture in serial thoracic computed tomography scans: Correlation of radiomics-based
features with radiation therapy dose and radiation pneumonitis development. International Journal of
Radiation Oncology âĸ Biology âĸ Physics 91: 1048â1056, 2015.
76. Purpose
ī§ Develop quantitative methods to measure dose-
dependent normal lung tissue damage in patients
who receive thoracic radiation therapy
ī§ Characterize CT scan appearance based on pixel values and
spatial relationship between pixels
ī§ Use these quantitative techniques to distinguish
between patients with and without radiation
pneumonitis E
78. Texture Analysis
E
Mean -785 -385
Median -820 -348
St. dev. 148 234
IQR 90 296
Fractal dimension 3.0 2.7
Entropy of ripple-filtered region 4.0 7.1
Follow-upBaseline
Identified using
deformable registration
79. Results
ī§ Extent of texture change between scans increases with
severity of RT-induced damage E
80. Texture Analysis
ī§ Image texture change related to dose through the dose map
of the treatment planning scan
ETreatment planning dose mapBaseline
deformable registration
82. Patient Classification
E
ī§ Area under the ROC curve for the ability of texture features
to differentiate patients with and without radiation
pneumonitis
Mean AUC Across 20 Features
Low Dose
Medium
Dose
High Dose Fitted Slope
0.64 0.68 0.71 0.71
83. Summary
ī§ Identified a set of reliable texture features for use in lung
texture analysis applications
ī§ Fully automated method for quantitative analysis of
changes in lung parenchyma between serial CT scans
ī§ Demonstrated relationship between texture change and
ī§ Radiation dose
ī§ Severity of radiation therapy-induced damage
ī§ Radiation pneumonitis status
E