Abstract

Machine learning (ML) models have the potential to enable earlier prediction of postoperative complications in gastrointestinal surgeries, such as anastomotic leak (AL), bleeding, and sepsis. Developing ML models using retrospective electronic health record (EHR) data can be disconnected from clinical workflows and risks producing algorithms that are not actionable in practice. We present a structured, clinician-guided methodology for building binary classification risk prediction models that integrates clinical input at each stage. By emphasizing actionability, patient safety, and clinician usability alongside technical performance, it addresses key barriers to adoption of ML in clinical practice. Future work will apply this framework to multi-institutional EHR datasets for GI surgery.

1. Cohort Definition
Clinicians provide initial definitions and help adjudicate edge cases

Establishing a target cohort defines exactly who the model applies to, such as patients undergoing a specific surgery or individuals newly diagnosed with a disease. The index date is set to the date of the surgery of interest.

Methods

1. Rule-Based Design: Cohorts are assembled using concept sets built from standardized codes such as ICD-10 or CPT.
2. NLP and Text Mining: Unstructured clinical text, including progress notes, surgical notes, etc. is scanned to extract specific medical concepts. These concepts are then mapped into a standardized ontology so they can be consumed by the model
3. Probabilistic Design: A machine-driven approach in which supervised ML learns the defining attributes of a cohort from a set of labeled examples.

 

2. Model Framing

Clinicians define the scope and timing of the prediction problem

Model framing translates the clinical question into a structured prediction problem for a machine learning model. Clinicians help establish the observation window (the period of historical data used as model input) and the prediction window (the future time period in which the outcome is expected to occur).

Methods

1. Defining the Prediction Structure: Frame the question as: Among [target cohort], who will develop [outcome] within [prediction window]?
2. Selecting the Clinical Use Case: Determine whether the model predicts disease onset or progression, need for treatment/intervention, need for further workup, etc.
3. Setting Temporal Constraints: Define when the prediction window begins relative to the index date. Require a minimum observation window to ensure patients have sufficient follow-up time.
4. Setting Exit Events: Define events after which predictions are no longer valid (e.g. discharge, ICU escalation) to avoid spurious predictions during non-actionable clinical contexts.

 

3. Label Definition
Clinicians define relevant events indicative of complications

Label definition determines the outcome status for each patient within the prediction window. Patients who experience the outcome event (e.g. diagnosis, intervention) are assigned a positive label while those who do not are assigned a negative label. Multiple methods can be used in combination to cross-validate labels and strengthen confidence in their accuracy.

Methods

1. Code-Based Phenotyping: Administrative and billing codes (e.g., ICD, CPT) extracted from the EHR are used to flag the presence of an outcome.
2. NLP-based Labelling: NLP algorithms extract outcome-relevant concepts from unstructured clinical notes.
3. Chart Review: Clinical experts manually review patient charts to establish a gold-standard label.

 

4. Feature Selection

Clinicians suggest features relating to the pathophysiology of the complication

This step identifies and selects the patient-level variables that will serve as inputs to the model. The features available for selection are determined by the observation window and the clinical context defined in previous steps. Depending on the prediction tasks features may be strictly preoperative, intraoperative, postoperative, or a combination.

Methods

1. Filter and Wrapper Methods: Filter methods use statistical tests (e.g., hypothesis testing, correlation) to pre-select features based on a significance cutoff. Wrapper methods evaluate different feature subsets iteratively through forward selection or backward elimination.
2. Implicit Feature Selection: Certain algorithms perform feature selection during training. Regularized regression (e.g., LASSO) shrinks unhelpful coefficients to zero, while tree-based methods select features based on their importance in splitting the data.
3. Missingness Indicator: Missingness itself can be a clinically meaningful feature, the absence of a measurement may reflect a provider’s clinical judgement. Absence of a feature can be encoded as a predictor. When missingness is not informative, imputation methods such as mean/median replacement, k-nearest neighbors, or multiple imputation via Bayesian regression can be used to fill in gaps.