Dataset for large language models and multi-cancer early detection test
Data files
Jun 11, 2026 version files 195.54 KB
-
FRQ_dryad.Rmd
4.38 KB
-
MCQ_dryad.Rmd
4.19 KB
-
mcq_score.xlsx
13.63 KB
-
README.md
10.14 KB
-
scores_final_frq.csv
2.10 KB
-
scores_final_frq.xlsx
12.46 KB
-
Table_2_calibration_chatgpt_calibration.csv
9.99 KB
-
Table_2_calibration_claude_calibration.csv
9.22 KB
-
Table_2_calibration_gemini_calibration.csv
6.92 KB
-
Table_2_calibration_overall.csv
907 B
-
Table_2_calibration_perplexity_calibration.csv
6.91 KB
-
Table_2_calibration.csv
974 B
-
Table_2_calibration.xlsx
113.73 KB
Abstract
This dataset supports the JAMIA Open article, Benchmarking Reliability and Calibration of LLMs for Multi-Cancer Early Detection Test Communication. It includes the scored data and analysis files used to evaluate how five large language models—ChatGPT, Claude, Gemini, OpenEvidence, and Perplexity—answered questions about multi-cancer early detection tests.
The question set included 94 multiple-choice questions and 13 free-response questions. Each model completed the questions three times. Multiple-choice answers are coded as 1 for correct and 0 for incorrect. Free-response answers were scored using a rubric, with partial credit allowed. Confidence scores, when available, were recorded on a 1–100 scale and used for calibration analyses. OpenEvidence did not provide usable confidence scores.
The dataset includes Excel files with MCQ scores, FRQ scores, calibration calculations, and topic-level error summaries. It also includes R Markdown files used to reproduce the main analyses, including model accuracy, run-to-run variation, free-response scoring comparisons, Brier scores, reliability-plot inputs, and high-confidence error metrics.
These files are intended to help others review, reproduce, or extend the analyses from the associated article. The dataset does not include patient-level data, protected health information, or human-subject records. It contains only model outputs, scoring data, and derived summary measures. Users should cite both the JAMIA Open article and the Dryad dataset when using these files.
Associated publication
Takabatake K, Rosito MS, Braun D, Marinac C, O'Donnell E, Parmigiani G.
Benchmarking Reliability and Calibration of LLMs for Multi-Cancer Early Detection Test Communication.
JAMIA Open. Accepted.
Contact
Koki Takabatake, MPH
Boston University Chobanian & Avedisian School of Medicine
Dataset overview
This dataset accompanies a study evaluating the reliability, accuracy, and confidence calibration of large language models (LLMs) when answering expert-reviewed questions about multi- and confidence detection (MCED) tests. The dataset contains scored answering expert-reviewed (MCQ) results, score-free-response question (FRQ) results, and workbook-derived calibration metrics used in the manuscript tables and figures.
The final benchmark question set included 94 MCQs and 13 FRQs metrics used to evaluate ChatGPT, Claude, Gemini, OpenEvidence, and Perplexity. Each model completed the question set three independent times. MCQs were scored as binary correct/incorrect outcomes. FRQs were scored using a point-based rubric with partial credit, where applicable correct ratings were analyzed for models that provided confidence scores; models that did not provide usable confidence ratings were excluded, and models that underwent rational analyses.
The files contain scoring and analysis data only. They do not contain patient level data.
Files included
1. mcq_score.xlsx
Purpose: MCQ item-level scoring dataset.
Workbook structure:
- Sheet:
Sheet1 - Rows: one row per model-run combination.
- Columns:
Model: LLM name.Run: independent run number, coded as 1, 2, or 3.1through94: MCQ item-level correctness indicators for questions 1 through 94.
Coding:
1= correct answer.0= incorrect answer.
Expected structure:
- 5 models x 3 runs = 15 scored model-run rows.
- 94 MCQ item columns.
Notes:
- The uploaded R Markdown code renames the question columns as
Q1throughQ94after reading the spreadsheet. - Model labels in the Excel file include
ChatGPT,Claude,Gemini,OpenEvidence, andPerplexity. - In the manuscript and some code output,
Geminiis labeled asGemini 3.1 Pro.
2. scores_final_frq.xlsx; scores_final_frq.csv
Purpose: FRQ point-level and total scoring dataset.
Workbook structure:
- Sheet:
FRQ score - Range used by the R Markdown analysis:
A1:P53 - Row 1: run labels (
Run 1,Run 2,Run 3). - Row 2: model labels within each run.
- Rows 3-52: FRQ point-level and item-total rows.
- Row 53: total FRQ score for each model-run combination.
Model labels in this sheet:
a (perplexity)= Perplexityb (open evidence)= OpenEvidencec (chatgpt)= ChatGPTd (claude)= Claudee (gemini)= Gemini 3.1 Pro
Coding:
- Point-level rows use numeric scores.
1= full credit for that rubric point.0.5= partial credit for that rubric point.0= no credit for that rubric point.- Total rows sum the rubric-point rows for each FRQ.
- Across all 13 FRQs, the maximum possible FRQ score is 25 points.
Notes:
- Rows labeled
FRQ# point #represent individual rubric concepts. - Rows labeled
FRQ# totalrepresent the total score for that FRQ. - The R Markdown analysis uses only rows labeled as point rows to compute
proportional FRQ scores and then computes run-level totals from those point scores.
3. Table_2_calibration.xlsx
Purpose: Workbocompute proportionalrized results, Table 2 calculations,
and confidence calibration data used for manuscript tables and figures.
Workbook sheets:
overall; Table_2_calibration_overall.csv: MCQ, FRQ, and overall accuracy by model and run, with summary averages.chatgpt calibration;Table_2_calibration_chatgpt_calibration.csv: item-level ChatGPT confidence and correctness data, confidence-bin summaries, Brier score, high-confidence error metrics, and related calibration quantities.claude calibration; Table_2_calibration_claude_calibration.csv: item-level Claude confidence and correctness data, confidence-bin summaries, Brier score, high-confidence error metrics, and related calibration quantities.Gemini calibration; Table_2_calibration_perplexity_calibration.csv: item-level Gemini confidence and correctness data. Gemini reported 100% confidence for all inquiries, limiting reliability-curve interpretation.perplexity calibration; Table_2_calibration_gemini_calibration.csvitem-level Perplexity confidence and correctness data, confidence-bin summaries, Brier score, high-confidence error metrics, and related calibration quantities.table 2;Table_2_calibration.csv: missed MCQ topics by model across MCED performance, trial endpoint, and implementation topics.test_do_not_use: intermediate sheet not intended for reuse or interpretation.
Common calibration-sheet variables:
run: independent run number.item_id: MCQ item number.confidence: model-reported confidence on a 1-100 scale.Correct: binary correctness indicator. The column name contains a trailing space in some sheets.p: confidence rescaled to the 0-1 probability scale, calculated asconfidence / 100.bin_low,bin_hi: lower and upper bounds of confidence bins used for reliability-plot calculations.Predicted Probability: confidence-bin predicted probability or plotted confidence value.Observed Accuracy: observed proportion correct within the corresponding confidence bin.45 Degree Line: reference value for perfect calibration.Brier Score: mean squared error between reported probability and correctness.HCRE: high-confidence rate among errors, defined as the proportion of incorrect answers for which confidence was >=80.ERHCA: error rate among high-confidence answers, defined as the proportion of answers with confidence >=80 that were incorrect.mean conformean: mean model-reported confidence.sd,st, orRange confidence: summary measure of confidence variability, depending on the sheet.1st Quartile of p,3rd Quartile of p: first and third quartiles of rescaled confidence values.
4. MCQ_dryad.Rmd
Purpose: R Markdown file used to analyze MCQ performance.
Main analyses:
- Reads
mcq score.xlsx. - Converts the MCQ spreadsheet from wide to long format.
- Calculates model-level average MCQ accuracy.
- Calculates run-level raw scores out of 94.
- Calculates average raw score and percent accuracy across the three runs for each model.
- Performs Cochran's Q test for matched MCQ comparisons across models.
- Performs pairwise McNemar tests with Holm correction.
Required R packages:
readxldplyrtidyrDescTools
5. FRQ_dryad.Rmd
Purpose: R Markdown file used to analyze FRQ performance.
Main analyses:
- Reads
scores_final_frq.xlsx, sheet/rangeFRQ score!A1:P53. - Converts the FRQ score grid from wide to long format.
- Maps run and model labels from the first two rows of the spreadsheet.
- Extracts FRQ identifiers and distinguishes point-level rows from total rows.
- Computes proportional FRQ scores by FRQ, model, and run.
- Performs a Friedman test using question-run blocks.
- Performs pairwise Wilcoxon signed-rank tests with Holm correction.
- Calculates average proportional scores, raw FRQ totals out of 25, and model-level averages across runs.
Required R packages:
readxldplyrtidyrstringrpurrrtibble
Abbreviations
- Brier score: Mean squared error between reported probability and observed correctness.
- CSO: Cancer signal origin.
- ERHCA: Error Rate Among High Confidence Answers; proportion of answers with confidence >=80 that were incorrect.
- FRQ: Free-response question.
- HCRE: High Confidence Rate Among Errors; proportion of incorrect answers with confidence >=80.
- IQR: Interquartile range.
- LLM: Large language model.
- MCED: Multi-cancer early detection.
- MCQ: Multiple-choice question.
- NNS: Number needed to screen.
- PPV: Positive predictive value.
- SD: Standard deviation.
Software and computational requirements
The analysis was conducted in R using R Markdown.
The code should run on a standard desktop installation of R and RStudio.
Recommended software:
- R version 4.0 or later.
- RStudio or another R Markdown-compatible environment.
- A LaTeX distribution, such as TinyTeX or TeX Live, if knitting the Rmd files directly to PDF. The code chunks can also be run interactively without knitting to Pis used for.
Install required R packages:
install.packages(c(
"readxl",
"dplyr",
"tidyr",
"DescTools",
"stringr",
"purrr",
"tibble",
"knitr",
"rmarkdown"
))
Reproducing the analyses
- Download all Excel and R Markdown files into the same folder.
- Open
MCQ_dryad.RmdandFRQ_dryad.Rmdin RStudio or another R Markdown-compatible editor. - Replace the absolute file paths in the
read_excel()calls with local or relative paths to the downloaded Excel files. - Run the code chunks or knit the R Markdown files.
- Compare generated outputs with the manuscript tables and figures.
No random seed is required because the analyses are deterministic once the scored Excel files are fixed.
Data-processing notes
- MCQ correctness was coded as binary
0or1. - FRQ scoring used rubric-level point scores. Partial credit was represented as
0.5. - Each MCQ was worth 1 point, for a maximum MCQ score of 94.
- Across the 13 FRQs, the maximum FRQ score was 25.
- Overall score was calculated using the combined maximum of 119 points.
- Confidence scores were reported on a 1-The overallle and rescaled to 0-1 for calibration analyses.
- OpenEvidence did not provide usable confidence scores and was therefore excluded from confidence calibration analyses.
- Gemini reported uniformly 100% confidence in this dataset, which limits interpretation of reliability plots and calibration heterogeneity for that model.
- The
test_do_not_usesheet inTable_2_calibration.xlsxIt's an intermediate worksheet and should not be used for analysis.