Intelligent documentation in medical education: Can AI replace manual case logging?
Abstract
This study investigates the feasibility of using large language models (LLMs) to automate procedural case log documentation in radiology training. We evaluate whether AI can replace manual logging, identify procedure types most challenging for extraction, and assess integration into clinical workflows. We retrospectively analyzed 36 ,659 radiology reports authored by nine interventional radiology residents (2018–2024). A subset of 414 reports was manually annotated for 39 procedures spanning vascular diagnosis, vascular intervention, and non-vascular intervention. Candidate models, Qwen-2.5 and Claude-3.5, were chosen based on privacy, hardware constraints, and availability, and tested under instruction and chain-of-thought prompting. A crosswalk baseline using structured exam codes provided comparison. Performance was measured by sensitivity, specificity, and F1-score, along with inference time and token efficiency to estimate operational cost. Both local and commercial LLMs outperformed the crosswalk benchmark. Qwen-2.5 achieved sensitivities up to 94.19\% and F1-scores of 86.66 with chain-of-thought prompting, while Claude-3.5-Haiku reached an F1-score of 86.89 and specificity of 99.29\%. Errors were concentrated in ambiguous “other” procedures, whereas common procedures were reliably classified. Chain-of-thought prompting reduced false positives relative to instruction prompting. Commercial inference delivered sub-2s latency and concise outputs, while local deployment traded speed for lower recurring cost. Automation could save more than 35 hours of manual annotation per resident annually. LLMs thus offer a scalable, accurate, and cost-efficient solution for radiology case log documentation. Optimizing for procedure-specific challenges and ensuring seamless integration with existing systems will be essential. Future work should validate across larger, multi-institution datasets and explore additional prompting strategies.
This repository contains the code and prompt templates used to evaluate whether large language models (LLMs) can automate structured medical case log generation from clinical encounter notes, benchmarked against manual documentation by medical residents. Raw clinical data is not included due to patient privacy constraints.
Description of the data and file structure
PCL-Fetcher-master/
├── Dockerfile
├── requirements.txt
├── readme.md
├── code/
│ ├── _constant_func.py
│ ├── _stat_gen.py
│ ├── 00_format_prompt_appendix.py
│ ├── 00_preprocess_prompt.py
│ ├── 00_proc_desc_table.py
│ ├── 00_procedure_count_distribution.py
│ ├── 00_report_token_count_distribution.py
│ ├── 00_resident_count_distribution.py
│ ├── 00_run_check_bedrock.py
│ ├── 01_run_llm.py
│ ├── 01_run_llm.sh
│ ├── 02_run_llm_bedrock.sh
│ ├── 02_run_llm_bedrock_converse.py
│ ├── 02_run_llm_bedrock_invoke.py
│ ├── 03_run_evaluation.py
│ ├── 04_run_evaluation_crosswalk.py
│ ├── 04_run_evaluation_crosswalk.sh
│ ├── 05_performance_visualizer.py
│ ├── 05_performance_visualizer_model.py
│ ├── 06_collect_question_number.py
│ ├── 06_collect_question_numbers_graph.py
│ ├── 07_latex_table_gen.py
│ ├── 08_time_and_token_calc.py
│ ├── 09_cost_calc_usd.py
│ ├── 10_get_server_configuration.py
└── prompt-files/
├── PCL_Questions_V5.csv
├── PCL_Questions_V5_CoT.csv
└── formatted_prompts.tex
code/
Contains Python scripts and shell runners implementing the full evaluation pipeline. Files are prefixed with step numbers indicating execution order. Helper modules (_constant_func.py, _stat_gen.py) provide shared utilities used across scripts.
- Step 00 — Preprocessing and descriptive analysis:
00_preprocess_prompt.pyprepares clinical notes as prompt inputs;00_format_prompt_appendix.pyformats prompts for the manuscript appendix;00_proc_desc_table.pygenerates a procedure description summary table;00_procedure_count_distribution.py,00_report_token_count_distribution.py, and00_resident_count_distribution.pyplot descriptive distributions;00_run_check_bedrock.pyvalidates AWS Bedrock connectivity. - Step 01 — LLM inference via OpenAI-compatible API (
01_run_llm.py,01_run_llm.sh). - Step 02 — LLM inference via AWS Bedrock, using both the Converse API (
02_run_llm_bedrock_converse.py) and InvokeModel API (02_run_llm_bedrock_invoke.py). - Step 03 — Scores LLM-generated outputs against reference case logs (
03_run_evaluation.py). - Step 04 — Reconciles and maps evaluation scores across models (
04_run_evaluation_crosswalk.py,.sh). - Step 05 — Generates performance comparison figures (
05_performance_visualizer.py,05_performance_visualizer_model.py). - Step 06 — Collects and plots per-question response statistics (
06_collect_question_number.py,06_collect_question_numbers_graph.py). - Step 07 — Exports evaluation results as LaTeX tables (
07_latex_table_gen.py). - Step 08 — Computes inference time and token usage (
08_time_and_token_calc.py). - Step 09 — Estimates inference cost in USD (
09_cost_calc_usd.py). - Step 10 — Retrieves and logs model server configuration (
10_get_server_configuration.py).
prompt-files/
Contains the prompt templates used to query LLMs for each case log field.
PCL_Questions_V5.csv— Standard prompt set for case log questions.PCL_Questions_V5_CoT.csv— Chain-of-thought variants of the same prompts.formatted_prompts.tex— LaTeX-formatted version of the prompts, included as a manuscript appendix.
Dockerfile and requirements.txt
Define the software environment. requirements.txt lists all Python package dependencies. The Dockerfile containerizes the environment for full reproducibility.
Sharing/Access information
Links to other publicly accessible locations of the code:
Raw clinical case log data are not included due to patient privacy considerations. The prompt templates and scripts are sufficient to replicate the pipeline with institutional data formatted to the expected input schema.
Code/Software
Language: Python 3.10+. Shell scripts (.sh) are provided for batch execution on Linux/macOS.
Environment setup:
pip install -r requirements.txt
Docker (recommended):
docker build -t pcl-fetcher .
docker run pcl-fetcher