Adaptive epithelial cytoplasm segmentation and epithelial unit separation in immunoflurorescent images

Tissue segmentation is one of the key preliminary steps in the morphometric analysis of tissue architecture. In multichannel immunoflurorescent biomarker images, the primary segmentation steps consist of segmenting the nuclei (epithelial and stromal) and epithelial cytoplasm from 4′,6-diamidino-2- phenylindole (DAPI) and cytokeratin 18 (CK18) biomarker images respectively. The epithelial cytoplasm segmentation can be very challenging due to variability in cytoplasm morphology and image staining. A robust and adaptive segmentation algorithm was developed for the purpose of both delineating the boundaries and separating thin gaps that separate the epithelial unit structures. This paper discusses novel methods that were developed for adaptive segmentation of epithelial cytoplasm and separation of epithelial units. The adaptive segmentation was performed by computing the non-epithelial background texture of every CK18 biomarker image. The epithelial unit separation was performed using two complementary techniques: a marker based, center-initialized watershed transform and a boundary initialized fast marching-watershed segmentation. The adaptive segmentation algorithm was tested on 926 CK18 biomarker biopsy images (326 patients) with limited background noise and 1030 prostatectomy images (374 patients) with noisy to very noisy background. The segmentation performance was measured using two different methods, namely; stability and background textural metrics. It was observed that the database of 1030 noisy prostatectomy images had a lower mean value (using stability and three background texture performance metrics) compared to the biopsy dataset of 926 images that had limited background noise. The average of all four performance metrics yielded 94.32% accuracy for prostatectomy images compared to 99.40% accuracy for biopsy images.