Data for: DETERIO-LLM: Enhancing traditional deterioration risk scores with clinical context and advanced reasoning
Data files
Jul 29, 2026 version files 20.03 KB
-
DETERIO-LLM_Dryad.zip
16.51 KB
-
README.md
3.52 KB
Abstract
Introduction: Accurate and timely prediction of physiological deterioration in hospitalized patients is essential for improving clinical outcomes. While traditional deep learning models have improved performance over existing early warning systems, their reliance on structured Electronic Health Record (EHR) data often limits their capacity to capture nuanced contextual information in unstructured clinical notes, leading to high false positive rates and limited clinical utility.
Objectives: We present DETERIO-LLM, a hybrid deterioration prediction model designed to selectively integrate Large Language Models (LLMs) with a pre-existing deep learning model (DETERIO) to reclassify borderline-risk alerts using narrative clinical notes.
Methods: We used a retrospective cohort of 1,000 inpatients with a 4.6% deterioration prevalence. DETERIO-LLM selectively applied LLM analysis to narrative notes only for alerts falling within an uncertainty risk range. Model performance was evaluated using Sensitivity, Positive Predictive Value (PPV), and F1 score and benchmarked against several comparators, including the best-established ML model, eCART.
Results: DETERIO-LLM significantly improved predictive precision over all comparators. At its optimal threshold (score 3.0), the model achieved a PPV of 30.6% and an F1 score of 37.3%, exceeding the performance of eCART. Overall, this selective integration reduced false positive alerts by 46.5% within the designated uncertainty range while maintaining comparable sensitivity.
Conclusion: These findings demonstrate that selective LLM integration with structured EHR models offers a practical and interpretable path to improving early warning systems in clinical care by leveraging narrative context to reduce alert fatigue.
Manuscript ID: JAMIO-2025-0376
Authors: Ghodsieh Ghanbari, Joseph C. Ahn, Eileen Kim, Ruth Laverde, Zach Pope, and Shamim Nemati
Repository Contents
This repository accompanies the above manuscript and provides the reproducibility materials for the study. The underlying patient data is protected health information (PHI) and cannot be shared (see Data Availability below); this deposit therefore contains the prompt, evaluation code, and documentation needed to understand and run the evaluation, plus a small synthetic example of the file format.
DETERIO-LLM_Dryad.zip folder containing the following files:
- prompt/final_prompt.txt — the large language model prompt used to classify borderline-risk cases.
- evaluation/evaluate_DETERIO_LLM.py — evaluation script that computes the reported metrics (sensitivity, specificity, PPV, F1) from the model outputs.
- DATA_DICTIONARY.md — definitions of all data columns and the model's input feature groups, plus the AIDE outcome criteria.
- sample_data/ — a small synthetic format example (a few rows) showing the column names and value types expected by the evaluation code. This is not a dataset and does not reproduce the published results. See
sample_data/README.txt. - Data_Availability_Statement.pdf — full data availability statement.
- requirements.txt — Python dependencies.
- LICENSE — CC0 1.0 public-domain dedication.
To run the evaluation: pip install -r requirements.txt, then
python evaluation/evaluate_DETERIO_LLM.py --data-dir sample_data.
Data Availability
This study was conducted using retrospective electronic health record (EHR) data from adult patients admitted to the University of California San Diego Health System.
The underlying patient-level data, including structured EHR variables and unstructured clinical notes, are not publicly available because they contain protected health information (PHI) and are subject to institutional policies, HIPAA regulations, and Institutional Review Board (IRB) requirements.
Requests for access to a limited de-identified dataset may be considered on a case-by-case basis, subject to approval by the University of California San Diego Institutional Review Board (IRB), institutional policies, and execution of an appropriate data use agreement.
Study Overview
The study evaluated DETERIO-LLM, a hybrid deterioration prediction framework that combines a deep learning model based on structured electronic health record data with a large language model that interprets narrative clinical notes to improve prediction of inpatient physiological deterioration.
The study cohort consisted of a stratified random sample of 1,000 adult inpatient encounters collected between January 1, 2016, and October 31, 2022.
Contact
For questions regarding the study or requests for data access, please contact:
Shamim Nemati, PhD
Department of Biomedical Informatics
University of California San Diego
Email: snemati@health.ucsd.edu
Human subjects data
The underlying electronic health record data used in this study contain information derived from human subjects and are not publicly available due to institutional, ethical, and privacy restrictions. No individual-level patient data are included in this Dryad submission.