Real-World Evaluation of an AI Auto-contouring System in Radiotherapy: A Multi-Domain Assessment

 

Background:

AI-based auto-contouring is increasingly used in radiotherapy (RT) planning. Most evaluations rely solely on geometric evaluation or time-saving metrics obtained under controlled conditions. These do not reflect real-world operational or patient impact. NICE guidance recommends multi-domain evaluation. This study provides a real-world, multi-domain assessment of Limbus AI® for Urology RT and examines the strengths and limitations of evaluation methods.

Methods:

1. Clinician usability: Two oncologists scored contours for 10 patients using a 5-point Likert scale.
2. Geometric accuracy: Dice Similarity Coefficient (DSC) and Hausdorff Distance (HD) were calculated for AI and clinician contours. Median DSC/HD were correlated with clinician scores.
3. Dosimetry: For 32 patients, plans were re-optimised using AI-generated organs at risk (OAR) while retaining clinician targets. Mandatory constraint violations and optimal variations were assessed using clinician contours as reference.
4. Workflow: RT pathway timings for all urology patients treated between Sept–Nov 2023 (pre-AI) and Sept–Nov 2024 (with AI) were compared.
5. Economics: A cost model estimated the contouring time reduction needed for cost-neutrality.

Results:

1. Median clinician usability was 3/5. 
2. Median DSC was 0.71 (IQR 0.64–0.75) and median HD 30.6 mm (IQR 23.4-34.62). Correlation between geometric and subjective ratings was weak (DSC Spearman 0.46 p=0.18; HD S=-0.02 p=0.957). 
3. Across 32 AI-optimised plans, 75% showed identical constraint attainment to clinical plans. All rectum, bladder and femoral head constraints were met. A small increase in bowel violations was observed (5 AI vs 0 clinical, p=-0.052).
4. Workflow analysis demonstrated reduced referral-to-RT (63→40.5 days, p=0.009), CT-to-RT (24→19 days, p<0.001) and contour-to-RT intervals (15→12 days, p<0.001). 
5. Economic modelling showed that a 6.1% reduction in contouring time was required for cost-neutrality; observed savings (~45 minutes per case) exceeded this threshold.

Conclusion:

Each method demonstrated the safety and benefit of Limbus AI but had its strengths and limitations:

• Clinician usability is quick and highlights structure-specific errors but is subjective.
• Geometric metrics are objective but did not correlate with clinical scores.
• Dosimetric evaluation reflects clinical safety not just stylistic differences.
• Workflow analysis and economic evaluation capture operational impact but are influenced by local practice.

Although some workflow and economic gains were observed, this study highlighted opportunities to improve efficiency by adjusting workflows or expanding automation for high-contrast pelvic organs and it confirms the need for continued oversight of bowel segmentation.