Purpose 

To assess the clinical accuracy of nine AI devices in the detection of pneumothorax on a unified dataset of chest X-rays (CXR), in addition to evaluating the impact of different demographic, clinical and imaging factors on AI performance.    

Methods and Materials  

We conducted a retrospective evaluation of nine CE-marked AI tools capable of PTX detection using a curated dataset of 394 CXRs obtained from an NHS hospital. Ground truth for PTX presence/absence on each CXR was established by two radiologists, with disagreement arbitrated by a third. Sensitivity, specificity, and accuracy were calculated for each vendor. Receiver Operating Characteristic (ROC) curve was calculated where vendors were able to provide variable threshold scores.  

Results  

A dataset of 394 CXRs (47.2% positive for PTX, 39.6% female, mean age 56 years) was inferenced by nine AI devices. Accuracy values ranged from 76.1% (95% confidence interval 71.6 - 80.3) to 93.9% (91.1-96.1). All specificity values were greater than 79.5%, with the highest at 97.1% (93.8-98.9). Overall, sensitivity was lower across the group and had a higher spread of results ranging from 54.3% (46.9-61.6) to 92.5% (87.7-95.8).  

Conclusions 

Significant variation in performance is demonstrated for pneumothorax between different commercially available AI devices. Future in-depth comparative analyses are critical for healthcare providers and developers alike to guide the development, adoption and deployment of CE-approved devices.