The impact of pulmonary nodule size estimation accuracy on the measured performance of automated nodule detection systems

The performance of automated pulmonary nodule detection systems is typically qualified with respect to some minimum size of nodule to be detected. Also, an evaluation dataset is typically constructed by expert radiologists with all nodules larger than the minimum size being designated as true positives while all other smaller detected "nodules" are considered to be false positives. In this paper, we consider the negative impact that size estimation error, either in the establishment of ground truth for the evaluation dataset or by the automated detection method for the size estimate of nodule candidates, has on the measured performance of the detection system. Furthermore, we propose a modified evaluation procedure that addresses the size estimation error issue. The impact of the size measurement error was estimated for a documented research image database consisting of whole-lung CT scans for 509 cases in which 690 nodules have been documented. We compute FROC curves both with and without size error compensation and we found that for a minimum size limit of 4 mm the performance of the system is underestimated by a sensitivity reduction of 5% and a false positive rate increase of 0.25 per case. Therefore, error in nodule size estimation should be considered in the evaluation of automated detection systems.