https://imatge.upc.edu/web/publications/detection-aided-liver-lesion-segmentation-using-deep-learning
A fully automatic technique for segmenting the liver and localizing its unhealthy tissues is a convenient tool in order to diagnose hepatic diseases and assess the response to the according treatments. In this work we propose a method to segment the liver and its lesions from Computed Tomography (CT) scans using Convolutional Neural Networks (CNNs), that have proven good results in a variety of computer vision tasks, including medical imaging. The network that segments the lesions consists of a cascaded architecture, which first focuses on the region of the liver in order to segment the lesions on it. Moreover, we train a detector to localize the lesions, and mask the results of the segmentation network with the positive detections. The segmentation architecture is based on DRIU, a Fully Convolutional Network (FCN) with side outputs that work on feature maps of different resolutions, to finally benefit from the multi-scale information learned by different stages of the network. The main contribution of this work is the use of a detector to localize the lesions, which we show to be beneficial to remove false positives triggered by the segmentation network.
4. Introduction
§ Contract Tomography (CT) images are an important tool in medical
imaging to diagnose several diseases or assess treatments
§ Doctors typically rely on manual or semi-automatic techniques to
study anomalies in the shape and texture of organs of CT scans
8. September 2017 4
5. Introduction
§ Contract Tomography (CT) images are an important tool in medical
imaging to diagnose several diseases or assess treatments
§ Doctors typically rely on manual or semi-automatic techniques to
study anomalies in the shape and texture of organs of CT scans
§ Time-consuming
§ Subjective
8. September 2017 5
6. Introduction
§ Contract Tomography (CT) images are an important tool in medical
imaging to diagnose several diseases or assess treatments
§ Doctors typically rely on manual or semi-automatic techniques to
study anomalies in the shape and texture of organs of CT scans
§ Time-consuming
§ Subjective
8. September 2017 6
Fully Automatic Tool ✔
7. Introduction
§ We will focus on liver and its lesions segmentation tasks for
afterwards segmenting several organs and other anatomical structures
8. September 2017 7
Lesion GT
Liver GT
8. Introduction
§ Lesion segmentation is a quite challenging task:
§ Low contrast between lesion, liver and other organs
§ Lesions are variable in terms of shape, size and texture
§ Noise in CT scans
§ Typically statistical shape and intensity distribution models
8. September 2017 8
9. Introduction
§ Lesion segmentation is a quite challenging task:
§ Low contrast between lesion, liver and other organs
§ Lesions are variable in terms of shape, size and texture
§ Noise in CT scans
§ Typically statistical shape and intensity distribution models
§ Deep Convolutional Neural Networks have demonstrated to be
successful on challenging tasks as lesion and liver segmentation
8. September 2017 9
10. Introduction
§ Our pipeline is based on the strengths of a segmentation network and
a detector:
8. September 2017 10
Segmentation
Network
Specializes on fine localization
of lesions
Detection
Network
Learns global features given a
whole liver patch
11. Introduction
§ Tasks:
§ Develop a method to segment the lesion and the liver from CT scans using
in the framework of the LiTS Challenge.
§ Prove generality of segmentation network with Visceral dataset.
8. September 2017 11
14. Segmentation
§ Fully Convolutional Networks (FCNs)
8. September 2017 14
Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation
15. Segmentation
§ Encoder-Decoder architecture
8. September 2017 15
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2015). Segnet: A deep convolutional encoder-decoder
architecture for image segmentation
16. Segmentation
§ U-net
8. September 2017 16
Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks
for biomedical image segmentation
17. Segmentation
§ Deep Retinal Image Understanding (DRIU)
8. September 2017 17
Maninis, K. K., Pont-Tuset, J., Arbeláez, P., & Van Gool, L. (2016, October). Deep retinal image understanding
Input
Fine feature maps Coarse feature maps
Vessels Optic Disc
Specialized
Layers
Image
Base Network Architecture
19. Detection
8. September 2017 19
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016). Region-based convolutional networks for
accurate object detection and segmentation
23. Architecture: Segmentation Network
§ Segmentation network based on Deep Retinal Image Understanding
(DRIU) network (*)
§ Fully Convolutional Network (FCN)
§ Base network is VGG-16 and pre-trained with Imagenet
§ Side outputs
§ Combination of multi-scale side outputs
8. September 2017 23
Liver Segmentation
1
Les
3
Bounding box sampling
2Maninis, K. K., Pont-Tuset, J., Arbeláez, P., & Van Gool, L. (2016, October). Deep retinal image understanding. In International
Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 140-148). Springer International Publishing.
25. Bounding Box Sampling for Detection
§ Example of bounding box sampling and labeling
§ Data augmentation with flips and rotations x 8
8. September 2017 25
26. Detection Model
§ Pre-trained Resnet-50 without classification
layer for Imagenet
§ A single neuron determining whether it is a
healthy tissue
8. September 2017 26
30. Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§ 1. Pre-processing
§ 2. Weighting of Binary Cross Entropy (BCE) loss
§ 3. Stacking 3 consecutive slices at the input of the network
§ 4. Using liver segmentation to segment lesion
8. September 2017 30
31. Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§ 1. Pre-processing
§ 2. Weighting of Binary Cross Entropy (BCE) loss
§ 3. Stacking 3 consecutive slices at the input of the network
§ 4. Using liver segmentation to segment lesion
8. September 2017 31
32. 1. Pre-processing
§ We clipped pixel intensity values by maximum and minimum values
that statistically belong to the liver and lesion class
8. September 2017 32
0.30
Lesion
Dice Score
33. 1. Pre-processing
§ We clipped pixel intensity values by maximum and minimum values
that statistically belong to the liver and lesion class
8. September 2017 33
0.30 0.32
Lesion
Dice Score
34. Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§ 1. Pre-processing
§ 2. Weighting of Binary Cross Entropy (BCE) loss
§ 3. Stacking 3 consecutive slices at the input of the network
§ 4. Using liver segmentation to segment lesion
8. September 2017 34
35. 2. Weighting of Binary Cross Entropy (BCE) loss
Loss objective
§ Binary Cross Entropy (BCE) Loss
§ Weighted Binary Cross Entropy (BCE) Loss
8. September 2017 35
Balancing term
36. 2. Weighting of Binary Cross Entropy (BCE) loss
Balancing Schemes
§ Per-volume balancing
§ General balancing
8. September 2017 36
37. 2. Weighting of Binary Cross Entropy (BCE) loss
Balancing Schemes
§ Per-volume balancing
§ General balancing
8. September 2017 37
40. Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§ 1. Pre-processing
§ 2. Weighting of Binary Cross Entropy (BCE) loss
§ 3. Stacking 3 consecutive slices at the input of the network
§ 4. Using liver segmentation to segment lesion
8. September 2017 40
41. 3. Stacking 3 consecutive slices at the input of the network
§ Volumes of data that are highly redundant
§ We input a stack of slices in the network with supervision
§ Final configuration inputs 3 slices, one at each RGB channel
8. September 2017 41
Network
44. Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§ 1. Pre-processing
§ 2. Weighting of Binary Cross Entropy (BCE) loss
§ 3. Stacking 3 consecutive slices at the input of the network
§ 4. Using liver segmentation to segment lesion
8. September 2017 44
45. 4. Using liver segmentation to segment lesion
§ We worked with 2 different strategies:
1. Back-propagation (BP) through liver: Only back-propagation through liver
2. Multitask: Segmenting the liver and the lesion simultaneously
8. September 2017 45
46. 4. Using liver segmentation to segment lesion
§ Features of Back-propagation (BP) through liver strategy
§ The predicted liver mask is fixed
§ The liver mask multiplies to the lesion mask during training and testing
§ The weighting term in the BCE loss now just considers liver pixels
8. September 2017 46
Network 1 ∘Network 2
47. 4. Using liver segmentation to segment lesion
§ Pros of BP through liver ✔
§ Network learns from relevant pixels
§ Positive/Negative pixels are more balanced
§ Cons of BP through liver ✖
§ If there is a mistake in the liver mask, this mistake will affect the lesion
segmentation
8. September 2017 47
48. 4. Using liver segmentation to segment lesion
§ Features of Multi-task liver strategy
§ The output of the network is a channel for the liver and another for the
lesion.
§ The loss is the sum of the two BCE losses . Both classes can happen at the
same time. No competition among classes.
8. September 2017 48
Network 1 ∘Network 2
Network 1 Network 2
49. 4. Using liver segmentation to segment lesion
§ Pros of Multi-task ✔
§ The gradients of the network have information of both the liver and lesion
§ Efficient and Scalable
§ Cons of Multi-task ✖
§ Difficult to weight each task properly
8. September 2017 49
50. 4. Using liver segmentation to segment lesion
8. September 2017 50
Difference in masking during testing or testing + training
Performance of various strategies
Lesion
Dice Score 0.30 0.32 0.34 0.36
51. 4. Using liver segmentation to segment lesion
8. September 2017 51
Difference in masking during testing or testing + training
Performance of various strategies
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39
54. Detection
8. September 2017 54
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Precision-Recall for detection of lesion
55. Detection
§ Examples of detected positive windows
8. September 2017 55
Lesion GT
Lesion Pred.
Liver GT
Liver Pred.
56. Detection
§ Examples of detected positive windows
8. September 2017 56
Before Detection After Detection
57. Detection
8. September 2017 57
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Multitask
BP through liver
Multitask + Det
BP through liver + Det
Lesion Precision – Recall Curve
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39
58. Detection
8. September 2017 58
Lesion Precision – Recall Curve
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Multitask
BP through liver
Multitask + Det
BP through liver + Det
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39 0.41
59. Post- processing: 3D – Conditional Random Fields
§ It will add spatial coherence in the 3 dimensions, based on space and
appearance
8. September 2017 59
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39 0.41
60. Post- processing: 3D – Conditional Random Fields
§ It will add spatial coherence in the 3 dimensions, based on space and
appearance
8. September 2017 60
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39 0.41 0.43
70. Conclusions
§ We improved the baseline we worked on developing a set of strategies
applicable to other pipelines.
§ Detection + Segmentation improved over only Segmentation.
§ Using the liver for the lesion segmentation boosted the performance.
Learning from relevant samples seems a good practice.
§ Segmentation network has a lot of potential to segment different kinds of
structures in a single forward pass.
8. September 2017 70
85. Bounding Box Sampling for Detection
§ Single-scale 2D windows of 50x50 with stride 50
§ A window is placed if it overlaps > 25% with the liver
§ A windows is considered as positive if inside it there are >50 lesion
pixels
§ A margin of 15 pixels is added to all windows to have more context
31. July 2017 85
50
50 80
80
86. Image Classification
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Precision-Recall for classification of lesion
Image Classification with VGG-16
Image Classification with ResNet-50
31. July 2017 86
• As proof of concept we first
trained a classifier that
distinguished between healthy
image / unhealthy image
• Resnet – 50 layers really
improved compared to VGG-16
architecture