An Efficient Transformer-based Approach for Joint Nuclei Detection and Segmentation in Whole Slide Tissue Images

Abstract

The detection and segmentation of cell nuclei in whole slide tissue images plays an important role in disease diagnosis and treatment. Automatic detection and segmentation of nuclei is very challenging due to high nuclei density, low contrast, overlapping and adhesion between cells. Recently, Transformer-based object detection and instance segmentation methods have made great progress on traditional computer vision datasets. These Transformer-based approaches are effective by removing the need for many hand-crafted components in the network, and post-processing steps like non-maximum suppression (NMS). However, such approaches consume tremendous amount of memory in detection and segmentation of cell nuclei in whole slide tissue images due to large number of cells in the images. Also, those methods may suffer from inferior performance on small cell instances. Inspired by Deformable DETR which makes use of small set of key sampling points in the attention module to reduce the computation and the usage of multi-scale feature maps, we propose an efficient Transformer-based approach for joint nuclei detection and segmentation in whole slide tissue images. Specifically, In Transformer decoder, it directly outputs the object detection results and instance segmentation masks. We evaluate our proposed method on the public MoNuSeg dataset, and the experimental results show that our method achieves promising performance on nuclei detection and segmentation.