Aims/Background

Short-answer X-ray quizzes are crucial for radiology training, yet manual marking is time-consuming and inconsistent. While NLP/LLM advances aid other assessments [1, 2], radiology quizzes' unique lexical and laterality nuances remain unaddressed.

We designed a Python pipeline utilizing fuzzy string matching, a ChatGPT-assisted synonym dictionary, and rule-based checks for automated marking.

Diagnostic accuracy and agreement with expert human grading was evaluated by quantifying concordance. Secondary aims explored how answer length, laterality, and terminology drive disagreement, guiding future refinement.

Methods & Materials

We analyzed 840 anonymized responses from 28 residents across 30 FRCR-style X-ray quizzes, independently scored by an expert. Our automated pipeline standardized answers, then processed them sequentially using three Python modules:

System performance was evaluated across similarity ratios; 65% was chosen as the optimal threshold. Agreement (Cohen's κ), accuracy (diagnostic metrics), directional bias (McNemar's), and word-count prediction of disagreement (logistic regression) were quantified.

Discussion/Results

Our system achieved 97.5% overall concordance with the human grader (819/840 answers, κ=0.892, substantial agreement), demonstrated high sensitivity, specificity, and predictive values (Table 1).

Agreement was perfect (κ=1.00) for single-word answers and remained high (κ=0.927) for two-word responses but declined with increasing word count (Table 2). Logistic regression confirmed each additional word increased mismatch odds by 47.5% (OR 1.475, p < 0.001).

We identified 65% fuzzy-matching threshold as optimal, to maintain high agreement while being the strictest not statistically inferior to peak performance (e.g outperforming 70%, p = 0.0013) (Figure 1).

Most discrepancies (71.4%) clustered in the top five problematic questions. False positives often involved partially correct/generic answers. False negatives stemmed from descriptive variation, granularity issues, multi-finding answers, or complex lexically-distant correct answers.

Our hybrid pipeline rivals expert human performance, surpasses many LLM-only grading tools, and maintains transparency [4]. This empirically optimized approach provides a reproducible, stringent benchmark [5]. While accuracy declines with longer answers, highlighting lexical limitations, integrating concept-level embeddings (e.g., SapBERT [6]) and partial-credit logic should mitigate this. Discrepancy analysis also offers clear targets for dictionary expansion and rule refinement.

Limitations & Future Directions

Limitations include a single human grader (precluding inter-rater reliability), potential synonym omissions, non-deterministic ChatGPT suggestions, binary scoring (ignoring partial credit), and limited generalizability beyond X-ray quizzes to other modalities or assessment types.

Future enhancements aim for: ontology-powered named-entity recognition (RadLex), transformer encoding for semantic comparison, partial-credit rubrics, annotated-image feedback, a user-friendly GUI for management, and multi-institutional validation across imaging modalities. 

Conclusion

This novel study demonstrates the high performance and feasibility of a transparent hybrid NLP and ChatGPT framework for consistently and rapidly grading short-answer X-ray quizzes. Achieving ≥97% agreement with human grading at a 65% similarity threshold (perfect for one-word, near-perfect for two-word answers), this approach shows strong potential as a radiology education assessment tool, warranting broader implementation studies.

References