Accuracy evaluation of an automatic landmarking method based on 3D segmented CT scans of full lower limbs with knee osteoarthritis

Abstract

In preoperative planning, a number of techniques for registration of anatomical landmarks on bones, or “bones landmarking”, have been developed for automatic measurement of lower limbs. However, the evaluation of these techniques is often neglected, reducing confidence in results and limiting clinical applications. We propose a protocol to evaluate landmarking techniques applied to thirty 3D images of osteoarthritic lower limbs. The protocol includes the evaluation of the ground truth based on manual landmarking performed by two experienced clinicians. Intra- and inter-operator variability were quantified. Automatic landmarking was evaluated using the same procedure and compared to manual landmarking by computing distances with bounding boxes including manual landmarks. The method was applied to 87 landmarks on the femur, tibia, fibula, and patella. For intra-operator variability, the mean median error was 2.0 mm, with a maximum of 42.1 mm. For inter-operator variability, errors ranged from 0.0 to 43.5 mm, with a mean median error of 2.3 mm across all landmarks. The mean median error between manual and automatic landmarking methods was 2.4 mm, with a range from 0.0 mm to 22.8 mm. Confidence intervals of automatic landmarking errors were consistent with intra- and inter-operator variability confidence intervals. Approximately 42 % of the automatic landmarks were inside their bounding boxes, with a mean distance of 1.4 mm from the box for landmarks on the outside. Results showed different errors depending on landmark location and coordinates. Intra- and inter-operator variability shed light on the difficulty of defining ground truth based on manual segmentation. This finding emphasizes the need for a rigorous evaluation protocol for landmarking validation, including the verification of statistical test hypotheses and population size influence.