Stability based validation of cellular segmentation algorithms

Performance assessment of segmentation algorithms compares segmentation outputs to a handful of manually obtained ground-truth. This assumes that the ground-truth images are accurate, reliable and representative of the entire image set. In image cytometry, few ground-truth images are typically used because of the difficulty of manually segmenting images with large numbers of small objects. This violates the aforementioned assumptions. Automated methods of segmentation evaluation without ground-truth are needed. We describe a stable and reliable method for evaluating segmentation performance without ground-truth. Segmentation errors are either statistical or structural. Statistical errors reflect failure to account for random variations in pixel values while structural errors result from inadequate image description models. As statistical errors predominate image cytometry, our method focuses on statistical stability assessment. For any image-algorithm pair, we obtain multiple perturbed variants of the image by applying slight linear blur. We segment the image and its variants with the algorithm and determine the match between the output from the image and the output from its variants. We utilized 48 realistic phantom images with known ground-truth and four segmentation algorithms with large performance differences to assess the efficacy of the method. For each algorithm-image pair, we obtained a ground truth match score and four different statistical validation scores. Analyses show that statistical validation and ground-truth validation scores correlate in over 96% of cases. The statistical validation approach reduces segmentation review time and effort by over 99% and enables assessment of segmentation quality long after an algorithm has been deployed.