Data for: From study design to executable code: Automating target trial emulation with large language models
Data files
Jul 31, 2026 version files 668.20 KB
-
dataset_evaluation.zip
125.02 KB
-
README.md
3.93 KB
-
Supplementary_Material_4_-_renv.lock
539.25 KB
Abstract
Objective: Implementing target trial emulation (TTE) studies as standardized, reproducible analytic workflows is technically demanding. We developed Text-guided Health-study Estimation and Specification Engine Using Strategus (THESEUS), which uses large language models (LLMs) to translate free-text study descriptions into structured analytic specifications and Strategus R scripts within the Observational Health Data Sciences and Informatics (OHDSI) ecosystem.
Materials and Methods: THESEUS executes two steps: an LLM maps study descriptions to a JSON schema, and validated specifications are converted into Strategus R scripts through rule-based logic. For standardization evaluation, we compared specifications generated by eight LLMs using 15 OHDSI-based TTE studies and 15 non-OHDSI studies under primary-analysis and full-analysis settings.
Results: Under the primary-analysis setting, overall standardization accuracy ranged from 0.93 to 0.97 across models in OHDSI studies and from 0.82 to 0.95 in non-OHDSI studies. Gemini-3.1-Pro achieved the highest overall accuracy in OHDSI studies, while Gemini-3.1-Pro and Gpt-5.5 jointly achieved the highest overall accuracy in non-OHDSI studies. Under the full-analyses setting, field-level sensitivity ranged from 0.83 to 0.97 in OHDSI studies, with 0.07–0.80 false positives (FPs) per study, and from 0.77 to 0.89 in non-OHDSI studies, with 0.53–1.20 FPs per study. Gpt-5.5 performed best at the field level. THESEUS was implemented as a web application and coding-agent tools.
Discussion: Pairing a standardized data model with a structured analysis framework enables reliable LLM-assisted interpretation of study descriptions and deterministic workflow construction in observational research.
Conclusion: THESEUS supports translation of natural language study descriptions into executable, shareable code in standardized observational research settings.
Dataset DOI: 10.5061/dryad.qnk98sfzs
Description of the data and file structure
This dataset was used to evaluate the THESEUS framework in the THESEUS study. It includes the ground-truth datasets for both OHDSI and non-OHDSI studies, together with the supplementary environment file required to reproduce the THESEUS software environment
Files and variables
File: Supplementary_Material_4_-_renv.lock
Description: The renv.lock file contains the R package dependency information required for dr-you-group/theseus-core and dr-you-group/theseus-app.
File: dataset_evaluation.zip
Description: This archive contains the author-curated evaluation inputs and ground-truth structured specifications used to evaluate the text-to-JSON standardization component of THESEUS. It contains 90 TypeScript (.ts) files derived from 30 published target trial emulation studies: 15 OHDSI studies and 15 non-OHDSI studies.
Note: The dataset contains only study specifications and metadata extracted from the study papers; no individual-level or identifiable human participant data are included.
The directory structure is as follows:
ohdsi/primary/: 15 files containing the primary-analysis descriptions from OHDSI studies.ohdsi/multiple/: 15 files containing full-analysis descriptions from the same OHDSI studies, including primary and sensitivity analyses. These cases may contain multiple study periods, time-at-risk windows, propensity score adjustment strategies, or outcome models.ohdsi/primary_augmented/: 30 files containing two controlled, value-augmented versions of each OHDSI primary-analysis description. These versions preserve the general structure and wording of the original descriptions while modifying combinations of study-design settings for robustness evaluation.non-ohdsi/primary/: 15 files containing the primary-analysis descriptions from non-OHDSI studies.non-ohdsi/multiple/: 15 files containing full-analysis descriptions, including primary and sensitivity analyses, from the same non-OHDSI studies.
Each TypeScript file represents one evaluation case and exports two variables:
TEXT{StudyName}: The natural-language study-design description supplied to the large language model. Depending on the evaluation condition, it describes the study period, time-at-risk, propensity score adjustment, and outcome model for either the primary analysis or the full set of analyses.JSON{StudyName}: The corresponding author-created ground-truth specification. It encodes the expected study settings in a structured format, includingstudyPeriods, study-population and time-at-risk settings increateStudyPopArgs, propensity score settings inpsSettingsandcreatePsArgs, and outcome-model settings infitOutcomeModelArgs.
For augmented cases, the exported variable names include an Aug1 or Aug2 suffix corresponding to the filename. For example, IUDEHRE_aug1.ts exports TEXTIUDEHREAug1 and JSONIUDEHREAug1.
The files are designed for use with the evaluation code available at dr-you-group/theseus-evaluation, where input–ground-truth pairs are loaded to run model inference and compare the generated structured specifications with the reference answers.
Access information
Other publicly accessible locations of the data: