This document describes research on automatically classifying confocal laser endomicroscopy (CLE) images of the colon. The researchers built a system to 1) characterize crypt geometry in CLE images to help diagnosis, 2) automatically diagnose images as healthy, hyperplastic, or neoplastic, and 3) catalog and allow search of a CLE image database. The system achieved 100% accuracy on a test set of 137 images in automatically distinguishing healthy from unhealthy tissue, and in distinguishing hyperplastic from neoplastic tissue within unhealthy images. The researchers believe the system could help standardize CLE diagnosis and training for endoscopists.

1. Introduction
Materials and Methods
Results
Discussion
Automatic Classification of Optical Biopsy
Images
Ruben Tous1 & Prof. Dr. Ferrer Roca
2
1Universitat
Politècnica de Catalunya. Departament d’Arquitectura de
Computadors
2 Unesco
Chair of Telemedicine. Canary Islands. Spain
July 2013
Tous
Automatic Classification of Optical Biopsy Images

2. Introduction
Materials and Methods
Results
Discussion
Índex
1
Introduction
2
Materials and Methods
3
Results
4
Discussion
Tous
Automatic Classification of Optical Biopsy Images

3. Introduction
Materials and Methods
Results
Discussion
Índex
1
Introduction
2
Materials and Methods
3
Results
4
Discussion
Tous
Automatic Classification of Optical Biopsy Images

4. Introduction
Materials and Methods
Results
Discussion
Introduction
Problem statement
Confocal laser endomicroscopy (CLE) has revolutionized
gastrointestinal endoscopy by providing microscopic
visualization on a cellular basis in vivo.
However, most gastroenterologists are not trained to
interpret mucosal pathology, and histopathologists are
usually not available in the endoscopy suite.
Tous
Automatic Classification of Optical Biopsy Images

5. Introduction
Materials and Methods
Results
Discussion
Introduction
General project goal
To build a set of computer-aided diagnosis tools that assist
endoscopists in the interpretation of optical biopsies obtained
through CLE.
Real-time functionalities for supporting diagnosis in vivo.
Tools for cataloguing and searching CLE databases.
Tous
Automatic Classification of Optical Biopsy Images

6. Introduction
Materials and Methods
Results
Discussion
Introduction
Project subgoals
Real-time functionalities for supporting diagnosis in vivo.
Automated diagnosis. (normal vs. abnormal mucosa,
differentiation between different pathologies).
Overprint visual markers highlighting the geometry of the
crypts.
Tools for cataloguing and searching CLE databases.
Automated cataloguing of CLE databases.
Search engine with query-by-example functionality.
Tous
Automatic Classification of Optical Biopsy Images

7. Introduction
Materials and Methods
Results
Discussion
Introduction. Colon microarchitecture
Tous
Automatic Classification of Optical Biopsy Images

8. Introduction
Materials and Methods
Results
Discussion
Introduction. Colouring agents
Left: acrifliavine hydrochloride. Right: fluorescein (used for our
sample). Red arrows: goblet cells.
Tous
Automatic Classification of Optical Biopsy Images

9. Introduction
Materials and Methods
Results
Discussion
Índex
1
Introduction
2
Materials and Methods
3
Results
4
Discussion
Tous
Automatic Classification of Optical Biopsy Images

10. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. General workflow
Tous
Automatic Classification of Optical Biopsy Images

11. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. General workflow
In a first step, the system’s database was standardized
using image sets based on the Mainz confocal
classification for colonic optical biopsies.
This step was used to provide an optical biopsy search
engine based on extracting simplified features of crypt and
mucosal patterns.
The second step aims at providing an interrogation
platform to provide on-site assistance during CLE, mainly
by automated detection of crypt geometry.
Tous
Automatic Classification of Optical Biopsy Images

12. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. Image set description
137 black and white instances
10 are labelled as "healthy"
12 as "hyperplastic"
115 as neoplastic"
Instances have 1024x1024 pixels, and are 8-bit grayscale,
JPEG compressed and 548KB in average.
Many of the instances were taken from same patients, and
some of them only capture slight variations of a same
tissue.
Tous
Automatic Classification of Optical Biopsy Images

13. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. Class description
Each image instance is tagged according to medical
judgement, the three main categories being:
Healthy
Hyperplastic
Neoplastic
Neoplastic comprises Adenomous and Cancerous, each of
them in several degrees of seriousness and malignity,
namely normalör low gradefor adenomas, and normaländ
Type 1for carcinomas.
Sample instances can be categorised via visual inspection,
at least into the three main categories.
Tous
Automatic Classification of Optical Biopsy Images

14. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. Healthy tissue
Healthy tissue is characterised by regular
crypts in a honeycomb-like structure that
present small size and shape variance.
In general, healthy crypts are recognisable
as darker, circle-shaped areas in a
generally regular spatial distribution.
Due to variations in the depth or diferent
degrees of fluorescent agent penetration,
the cells’ cytoplasm may be especially
prominent (bottom). This circumstance
has proven, at later stages, to be
non-trivial to address.
Tous
Automatic Classification of Optical Biopsy Images

15. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. Hyperplastic tissue
Hyperplasia means cells have proliferated
more than usual in a tissue.
In hyperplastic colon cases we can
appreciate abnormal crypt bloating,
although their integrity is still preserved to
some extent.
Tous
Automatic Classification of Optical Biopsy Images

16. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. Hyperplastic tissue
Some milder cases present a regular crypt
structure just like in healthy tissue, except
in one or two crypts that have abnormal
growth.
In contrast with these, some others are
more dificult to recognise.
Tous
Automatic Classification of Optical Biopsy Images

17. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. Neoplastic tissue
Neoplasia is the massive and uncoordinated proliferation of
cells; this is also commonly known as tumour.
Tous
Automatic Classification of Optical Biopsy Images

18. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. Neoplastic tissue
A typical feature in them is having strange
nuclear-to-cytoplasmic ratios. However,
nuclei are not stained by fluorescein.
Fortunately, neoangiogenesis can be
observed with fluorescein, and it is a
phenomenon whose detection points
strongly towards pathogenesis.
These new blood vessels are caracterised
not only by tortuosity and high irregularity,
but also by profuse liquid leakiness, that
causes tissue staining easy to see.
Tous
Automatic Classification of Optical Biopsy Images

19. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. Algorithm outline
Automatic feature extraction in CLE images. Inferring
semantic metadata from low-level features.
Image Analysis + Pattern Recognition.
The image analysis algorithm allows identifying the
different crypts and also featuring their contours.
The extracted low-level visual features are then combined
to obtain a feature vector.
This vector is analyzed to translate the low-level details
into high-level semantic information about the images,
notably a suggested diagnosis.
Tous
Automatic Classification of Optical Biopsy Images

20. Introduction
Materials and Methods
Results
Discussion
Materials and Methods. Algorithm flowchart
Tous
Automatic Classification of Optical Biopsy Images

21. Introduction
Materials and Methods
Results
Discussion
Índex
1
Introduction
2
Materials and Methods
3
Results
4
Discussion
Tous
Automatic Classification of Optical Biopsy Images

22. Introduction
Materials and Methods
Results
Discussion
Results. Automatic crypt geometry characterization
The image analysis algorithm allows identifying the
different crypts and also featuring their contours with high
accuracy.
This information is computed to characterize the geometry
of the crypts and to overprint visual markers aiming to
facilitate diagnosis.
Tous
Automatic Classification of Optical Biopsy Images

23. Introduction
Materials and Methods
Results
Discussion
Results. Automatic crypt geometry characterization
Two watershedding flavours on a
neoplastic image
Tous
Automatic Classification of Optical Biopsy Images

24. Introduction
Materials and Methods
Results
Discussion
Results. Automated diagnosis
Our classification approach
Chaining two binary classification problems in a row. 1)
healthy vs. unhealthy; if unhealthy 2) hyperplasia vs
neoplasia.
We opted for this because the crypt shape, boundary, etc
on hyperplastic examples were not much different from
those obtained from neoplastic ones.
All classifers were evaluated using Leave One Out
Cross-Validation (LOOCV).
Tous
Automatic Classification of Optical Biopsy Images

25. Introduction
Materials and Methods
Results
Discussion
Results. Automated diagnosis
(step 1) healthy vs. unhealthy
Effectiveness (with 200x200 pixels downscaling and a
custom classifier). A 100% hit rate is attained with the
present algorithm.
The present sample (137 images) is relatively small, and
its classes are highly unbalanced in size.
Besides, they come from an even smaller number of
patients and capturing situations.
Tous
Automatic Classification of Optical Biopsy Images

26. Introduction
Materials and Methods
Results
Discussion
Results. Automated diagnosis
Confusion Matrix (custom classifer healthy vs. unhealthy)
classified as unhealthy
unhealthy
healthy
classified as healthy
127
0
0
10
Tous
Automatic Classification of Optical Biopsy Images

27. Introduction
Materials and Methods
Results
Discussion
Results. Automated diagnosis
(step 2) hyperplasia vs neoplasia
Classfication over the resulting unhealthy examples from
step 1.
We noted that cells were usually abundant and
homogeneously distributed in hyperplastic images.
We implemented a simple adhoc decision tree on the only
3 descriptors for the second decision by seeing how they
behaved. It also reached 100% prediction rate within
unhealthy images when dividing them in hyperplasia and
neoplasia.
Tous
Automatic Classification of Optical Biopsy Images

28. Introduction
Materials and Methods
Results
Discussion
Results. Automated diagnosis
Efficiency
We have studied the impact of the parameters choice on
time costs, aiming to reduce them at the least possible
accuracy expense.
With the current performance, we believe real-time
analysis of the newly acquired images is possible, thus
aiding the endoscopist in his or her exploration task.
In addition, this suggests a possible future line of progress:
work towards a video-rate (24 frames per second)
processing system, where crypt highlighting could be
especially valuable.
Tous
Automatic Classification of Optical Biopsy Images

29. Introduction
Materials and Methods
Results
Discussion
Results. Cataloguing and search
Besides supporting the diagnosis of a single image during
an ongoing session, the described method also allows
annotating a complete database of CLE images with
semantic information, thus providing a search engine with
advanced functionalities such as semantic retrieval or
query by image.
Tous
Automatic Classification of Optical Biopsy Images

30. Introduction
Materials and Methods
Results
Discussion
Índex
1
Introduction
2
Materials and Methods
3
Results
4
Discussion
Tous
Automatic Classification of Optical Biopsy Images

31. Introduction
Materials and Methods
Results
Discussion
Discussion
On-site image analysis and semantic interpretation are
able to provide a standardized diagnosis of CLE images in
real time.
This has the potential to facilitate, standardize and shorten
endoscopists’ training for CLE.
In addition, we provide a powerful tool to explore
databases of images for retrospective analyses.
This system’s geometry is not limited to CLE, but can be
extended to multiple imaging modalities.
Tous
Automatic Classification of Optical Biopsy Images