Motivation, tools and methods analysis of digital pathology imagery, integration with "omics" and Radiology, use in Precision Medicine. Presentation at the Early Detection Research Network meeting, April 2015, Atlanta GA

1. Generation and Use of Quantitative Pathology
Phenotype
Joel Saltz MD,PhD
Chair Biomedical Informatics, Stony Brook
Associate Director Stony Brook Cancer Center
NCI Early Detection Research Network - Atlanta April 1, 2015

2. Computational Pathology: High Dimensional Fused-Informatics
• Anatomic/functional characterization at
fine and gross level
• Integrate of anatomic/functional
characterization, multiple types of
“omic” information, outcome
• Predict treatment outcome, select,
monitor treatments
• High throughput tissue classification
• Computer assisted exploration of new
classification schemes
• Integrated analysis and presentation of
observations, features analytical results
– human and machine generated
Pathology
Patient
Outcome
Radiology
“Omic”
Data

3. Johns Hopkins School of Medicine
Virtual Microscope
1997 Proceedings AMIA Annual Meeting

4. Information Technology for Cancer Research : NCI U24
Stony Brook: Joel Saltz, Tahsin Kurc, Yi Gao, Allen
Tannenbaum, Fusheng Wang, Liangjia Zhu, Ivan
Kolesov, Romeil Sandhu, Erich Bremer
Emory: Adam Marcus, Lee Cooper, Dan Brat, Fadlo
Khuri Lee Cooper, Ashish Sharma, Rick Cummings,
Roberd Bostick
Oak Ridge National Lab: Scott Klasky, Dave Pugmire
Yale: Michael Krauthammer
Tools to Analyze Morphology and Spatially Mapped Molecular Data

5. Pathology Analytical Imaging
• Provide rich information about morphological and functional characteristics
• Image analysis, feature extraction on multiple scales
• Spatially mapped “omics”
• Multiple microscopy modalities
Glass Slides Scanning Whole Slide Images Image Analysis

6. Tumor Heterogeneity
Marusyk 2012

7. Report: Imaging-Genomics Workshop June 2013

8. Correlating Imaging Phenotypes with Genomic Signatures: Scientific Opportunities
Clinical Approach and Use
• Development of imaging+analysis methods to characterize heterogeneity
• within a tumor at one time point
• evolution over time
• among different tumor types
• Development of imaging metrics that:
• can predict and detect emergence of resistance?
• correlates with genomic heterogeneity?
• correlates with habitat heterogeneity?
• can identify more homogeneous sub-types
Imaging Genomics Workshop NCI June 2013

9. Direct Study of Relationship Between Image Features vs Clinical
Outcome, Response to Treatment, Molecular Information
Lee Cooper,
Carlos Moreno

10. Integrative
Morphology/”omics”
Quantitative Feature Analysis in Pathology: Emory
In Silico Center for Brain Tumor Research (PI =
Dan Brat, PD= Joel Saltz)
NLM/NCI: Integrative Analysis/Digital Pathology
R01LM011119, R01LM009239 (Dual PIs Joel
Saltz, David Foran)
Marcus Foundation Grant – Ari Kaufman, Joel
Saltz

11. Associations

12. Gene Expression Correlates of GBM with High Oligo-Astro Ratio
Oligo Related Genes
Myelin Basic Protein
Proteolipoprotein
HoxD1
Nuclear features most
Associated with Oligo
Signature Genes:
Circularity (high)
Eccentricity (low)

13. Pathology Computer Assisted Classification
Gurcan, Shamada, Kong, Saltz
Hiro Shimada, Metin Gurcan, Jun Kong, Lee Cooper Joel Saltz

14. Neuroblastoma Classification
FH: favorable histology UH: unfavorable histology
CANCER 2003; 98:2274-81
<5 yr
Schwannian
Development
≥50%
Grossly visible Nodule(s)
absent
present
Microscopic
Neuroblastic
foci
absent
present
Ganglioneuroma
(Schwannian stroma-dominant)
Maturing subtype
Mature subtype
Ganglioneuroblastoma, Intermixed
(Schwannian stroma-rich)
FH
FH
Ganglioneuroblastoma, Nodular
(composite, Schwannian stroma-rich/
stroma-dominant and stroma-poor) UH/FH*
Variant forms*
None to <50%
Neuroblastoma
(Schwannian stroma-poor)
Poorly differentiated
subtype
Undifferentiated
subtype
Differentiating
subtype
Any age UH
≥200/5,000 cells
Mitotic & karyorrhectic cells
100-200/5,000 cells
<100/5,000 cells
Any age
≥1.5 yr
<1.5 yr
UH
UH
FH
≥200/5,000 cells
100-200/5,000 cells
<100/5,000 cells
Any age UH
≥1.5 yr
<1.5 yr
≥5 yr
UH
FH
UH
FH

15. Computerized Classification System for Grading Neuroblastoma
• Background Identification
• Image Decomposition (Multi-resolution
levels)
• Image Segmentation (EMLDA)
• Feature Construction (2nd order statistics,
Tonal Features)
• Feature Extraction (LDA) + Classification
(Bayesian)
• Multi-resolution Layer Controller
(Confidence Region)
No
Yes
Image Tile
Initialization
I = L
Background? Label
Create Image I(L)
Segmentation
Feature Construction
Feature Extraction
Classification
Segmentation
Feature Construction
Feature Extraction
Classifier Training
Down-sampling
Training Tiles
Within Confidence
Region ?
I = I -1
I > 1?
Yes
Yes
No
No
TRAINING
TESTING

17. Large Scale Data Management
Data model capturing multi-faceted information including markups,
annotations, algorithm provenance, specimen, etc.
Support for complex relationships and spatial query: multi-level
granularities, relationships between markups and annotations, spatial
and nested relationships
Highly optimized spatial query and analyses
Implemented in a variety of ways including optimized CPU/GPU,
Hadoop/HDFS and IBM DB2 (Wang, Saltz, Kurc)
NLM/NCI: Integrative Analysis/Digital Pathology R01LM011119, R01LM009239 (Dual PIs Joel Saltz, David Foran)

18. Spatial Centric – Pathology Imaging “GIS”
Point query: human marked point
inside a nucleus
.
Window query: return markups
contained in a rectangle
Spatial join query: algorithm
validation/comparison
Containment query: nuclear feature
aggregation in tumor regions
Fusheng Wang

19. HPC Pipelines (Segmentation and Feature Extraction)
Tony Pan, George Teodoro,
Tahsin Kurc and Scott Klasky

20. Tools to Analyze Morphology and
Spatially Mapped Molecular Data
1U24CA180924-01
SPECIFIC AIMS

21. Pathology Image Analysis
• Specific Aim - Develop, deploy, and evaluate robust and scalable
methods and analysis pipelines for multi- scale, integrative image
analysis.
• Aim 1a Develop methods to segment micro-anatomic objects and extract
and classify features from whole slide tissue images.
• Aim 1b Develop analytic methods to carry out spatially mapped molecular
tissue characterizations.
• Aim 1c Develop methods to create 3-D reconstructions of multi- scale micro-
anatomic features and to characterize changes in morphology over time.

22. Pipelines, Database, Data modeling, Visualization
• Specific Aim 2: Develop database infrastructure to manage and query
image data, image analysis results.
• Specific Aim 3: Develop high performance software that targets
clusters, cloud computing, and leadership scale systems.
• Specific Aim 4: Develop visualization middleware for 2D/3D image
and feature data and for integrated image and “omic” data.

23. caMicroscope - A Digital Pathology Integrative Query System – PI Ashish Sharma -
CTIIP IIWG
• Interoperate with U24 MongoDB feature
database to allow annotation and
markup view and generation
• Extensions to support creation and
display of annotations (free form pencil
tool, polygon tool, rectangle & ellipse)
• Measurement tool & magnifying glass
(magnify by 200% a region of interest)

24. caMicroscope/MongoDB - Multiple Algorithm Comparison

25. Why we need multiple algorithm comparison

26. Hierarchical Pipeline (Slicer!)
Liangjia Zhu
Ivan Kolesov
Romeil Sandhu
Allen Tannenbaum

27. Low Power
• Fast GrowCut segmentation
• Intensity insufficient: need user
guidance
• Boundaries are most time
consuming for user

28. Medium Power
• Adaptive thresholding
segmentation
• Allow for global user input
(influence parameter settings)

29. Crypt/Nuclear Segmentation
• Variational active contour
• Context is crucial

30. Convolutional Neural Network Classification
Le Hou, Dimitris Samaras, Tahsin Kurc, Yi Gao, Liz Vanner, James Davis, Joel Saltz

32. Confocal/Super resolution nuclear morphometry (Slicer!)
Ken Shroyer, Yi Gao, Tahsin Kurc, Joel Saltz
• Pancreatic Fine Needle
Aspirate
• Correlative studies
linking fine needle
aspirate cell data,
“omic” and Radiology
imaging data
• Leverages Marcus
foundation virtual
biopsy effort

33.  Cells first prepared via Papanicolaou stain – identified as not suspicious
Preliminary Work

34.  Cells first prepared via Papanicolaou stain – identified as suspicious

35. Cytology shape analysis
volume elongation surface area roundness sp-radius
sp-area ellipse diameter axis 1 ellipse diameter axis2 ellipse diameter axis 3 flatness

36. 3D Imaging Analytics
• 3D Vessel Reconstruction with Serial Microscopy Images
• WSIs from serial sections have significant potential to enhance the study of both normal and disease
processes
• 3D reconstruction of cellular level objects is a critical step
(Yanhui Liang, Jun Kong, Fusheng Wang, Darren Treanor – Wang NSF CAREER Award)

37. 3D Imaging Analytics: Vessel Association
1
2
2 1
(a)
(b)
(a) Panoramic view of 3D reconstructed vessels; (b) two close-up views of
segment 1 and 2 indicated in the panoramic view.

38. Driving Cancer Research, Community Support, Challenges and Engagement
• Specific Aim 5: Drive continuing development of the tools using a suite of cancer
driving biomedical problem, and provide collaborative support and training to the
cancer research community.
• Aim 5a: Evaluate, demonstrate and drive continuing development of the tools using
a suite of cancer driving biomedical problems.
• Aim 5b: Provide support to the cancer research community through:
• Engagement in NCI Quantitative Imaging Network
• support of community digital Pathology image analysis “grand challenges” – initial
challenge involving analysis of TCGA whole slide image at MICCAI 2014,
• partnerships with collaborative efforts described in letters of support and collaboration,
including the National Cancer Imaging Archive, the Mayo Quantitative Imaging Network
site, the Colon Cancer Family Registry and the Polyp Prevention Study,
• partnerships with cancer microscopy/Pathology shared resources
• development of on line resources and workshops to teach users how to employ U24
tools.

39. CORRELATIVE FEATURE ANALYSIS

40. MICCAI 2014
BRAIN TUMOR
Classification and Segmentation Challenges
TCGA
TCIA
IMAGING
CHALLENGE
DIGITAL PATHOLOGY
CHALLENGE
Phase 1: Training
June 20 - July 31
Phase 2: Leader Board
Aug 1 - Aug 29
Phase 3: Test
Sept 8 - Sept 12
For more information about these challenges and a related workshop
on September 14, 2014 at MICCAI in Boston, see: cancerimagingarchive.net
MICCAI: Medical Image Computing and Computer Aided Interventions - MICCAI2014.org
TCGA: The Cancer Genome Atlas - cancergenome.nih.gov
TCIA: The Cancer Image Archive - cancerimagingarchive.net

41. Digital Pathology/Brain Tumor Image Segmentation (BRATS)
• Used data currently available through data archive resources of the
National Institutes of Health (NIH), namely, the Cancer Genome Atlas
(TCGA) and the Cancer Image Archive (TCIA)
• Digital Pathology challenge used digital slides related to patients whose
genomics data are available from TCGA. Similarly, BRATS 2014 Challenge
used clinical MRI image data, also from the TCGA study subjects.
• Sub-Challenge 1: Classification - Automated classification of LGG and
GBM from a collection of 30+ high-resolution digital pathology slides.
• Sub-Challenge 2: Segmentation – Automated segmentation of
necrotic and normal brain regions on regions of digital pathology
slides from a collection of 20+ GBM cases.

42. Organizers and Major Contributors
• Daniel J. Brat, Emory University
• Larry Clarke, National Cancer Institute
• James Davis, Stony Brook Cancer Center
• Keyvan Farahani, National Cancer Institute
• John Freymann, Leidos Biomedical Res, Inc.
• Carl Jaffe, Boston University
• Justin Kirby, Leidos Biomedical Res., Inc.
• Tahsin Kurc, Stony Brook Cancer Center
• Miguel Ossandon, National Cancer Institute
• Joel Saltz, Stony Brook Cancer Center
• Roberta Seidman, Stony Brook Cancer Center

43. Example of a “good match”

44. Example of a problematic match

45. Thanks!