Study TItle: Automating the Management of Extra-Spinal Findings in MRI Spine Studies Using a Privacy-Preserving Large Language Model: A Single-Institution Feasibility Study

Background: MRI spine studies frequently reveal extra-spinal findings (ESFs) that require further evaluation, yet the current process of manually reviewing radiology reports and navigating electronic medical records (EMRs) is time-consuming, labour-intensive and prone to human error. 

Objective: To address this challenge, we propose using a privacy-preserving large language model (PP-LLM) to automate the identification, classification, and referral assessment of extra-spinal findings (ESFs).

Methods: A retrospective analysis of 405 consecutive MRI spine reports from the National University Hospital (NUH) database, covering February to June 2024, was conducted. Two independent clinicians reviewed the reports and cross-referenced them with EMRs to identify ESFs from the imaging reports. The CT Extracolonic Findings Reporting and Data System (C-RADS), was adapted to dxetermine the clinical significance of ESFs and whether specialty referral was required. The PP-LLM was designed to extract these findings, differentiate between new and pre-existing conditions, classify their clinical significance, and generate appropriate referrals.

Results: A total of 400 MRI reports belonging to 395 patients (5 MRI reports excluded due to no relevant EMR).Among 395 patients (48.1% male, 51.9% female, mean age 54.7 years  ±SD 16.7, range 17 – 89 years), 163 patients (41.2%) had no ESFs and 232 patients (58.5%) were reported to have had at least one ESF. A total of 401 ESFs were identified, with the most common findings being renal (31.9%), gynaecological (27.4%) and endocrine-related (12.0%). The PP-LLM correctly detected 99.7% of all ESFs, and correctly identified all clinically urgent findings (3.0% of the total, e.g., aortic dissection). It misclassified 2.0% of cases into lower C-RADS categories, potentially downgrading clinically significant findings (e.g. paranasal sinus mucosal thickening), and 1.2% into higher C-RADS categories, upgrading clinically insignificant findings (e.g. dependent changes in the lungs). Additionally, it achieved 98.59.3% accuracy in distinguishing new from pre-existing findings and 96.2% accuracy in assigning the correct specialty referral decision (whether referral is needed, and correct subspecialty referral suggested). The PP-LLM demonstrated almost perfect agreement with the reference standard (Gwet κ = 0.959, 95% CI: 0.937–0.981), comparable to two human readers (Reader 1: κ = 0.943; Reader 2: κ = 0.934), with no statistically significant difference in C-RADS classification accuracy. Notably, it completed the analysis of each report in under five seconds. 

Conclusion: The PP-LLM demonstrated high accuracy and efficiency in automating the identification and classification of extra-spinal findings in MRI spine reports. By integrating this AI-driven automation into clinical workflows, this technology has the potential to enhance efficiency, reduce clinician administrative burden, ensure timely specialist referrals and improve patient care.