https://imatge-upc.github.io/rsis/ We present a recurrent model for semantic instance segmentation that sequentially generates pairs of masks and their associated class probabilities for every object in an image. Our proposed system is trainable end-to-end, does not require post-processing steps on its output and is conceptually simpler than current methods relying on object proposals. We observe that our model learns to follow a consistent pattern to generate object sequences, which correlates with the activations learned in the encoder part of our network. We achieve competitive results on three different instance segmentation benchmarks (Pascal VOC 2012, Cityscapes and CVPPP Plant Leaf Segmentation).

1. Recurrent Neural Networks for
Semantic Instance Segmentation
Amaia Salvador Jordi Torres Xavier Giró-i-Nieto Ferran MarquésManel BaradadMíriam Bellver

2. Motivation
2
Semantic Instance Segmentation
- Proposal-based solutions:
- Hundreds/Thousands of redundant predictions
- Post-processing needed (NMS)
- Holistic & class-agnostic methods
- Reduced set of predictions
- Separate network for semantics

3. Motivation
3
- Our solution:
- Recurrent model that sequentially predicts binary masks and
categorical labels for each object in an image.
- Learns to stop once all objects have been found.
- Does not need post-processing on its output.
Semantic Instance Segmentation

4. Model
4

5. Model
5

6. Pascal VOC
6
Average Precision at different IoU thresholds

7. Pascal VOC
7
Average Recall at different IoU thresholds (class-agnostic evaluation)

8. Pascal VOC
8

9. CVPPP Leaves Segmentation
9

10. Cityscapes
10

11. Object Sorting Patterns
11

12. Encoder Activations
Correlation with convolutional activations in the encoder before and after training

13. Encoder Activations
Activations at the end of the encoder before and after training