Zhaoqi Li and Emma Brunskill

Access this dataset on Dryad

This repository contains the code and datasets used in our paper A Statistical
Test for the Benefits of Personalizing Interventions, together with the
instructions needed to reproduce all figures and tables.

---

Description of the data and file structure

The submission is delivered as five compressed archives. Each archive expands to one top-level directory, so the unpacked dataset has the structure shown below.

domain_data.tar.gz expands to /domain_data : Raw source data and preprocessed snapshots for the three public domains (Job Corps, education, joke).

personalization_revision_real_domain_results_round2.tar.gz expands to /real_domains : Saved result files (.pkl, .csv) used to plot the real-domain figures.

personalization_revision_simulation_dataset_round2.tar.gz expands to /simulation_dataset : Generated simulation datasets used as inputs to the simulation analyses.

personalization_revision_simulation_results_round2.tar.gz expands to /simulation : Saved result files used to plot the simulation figures and tables.

personalization_code_nodata.tar.gz (Zenodo) expands to /personalization_code : All analysis code (notebooks, Python, R, C++) to reproduce results, stored in Zenodo link.

---

Full file and folder manifest

Below is the directory structure produced when each archive is expanded.

domain_data.tar.gz → /domain_data

domain_data/
├── README_domain_data.txt                                  (1.8 KB; placement instructions + abstract)
├── jobcorps/
│   ├── jobcorps_cleaned_nonan_full_with_offset.csv         (676 KB; semi-synthetic)
│   ├── impact.sas7bdat                                     (89 MB; raw Job Corps source, Schochet et al. 2008)
│   └── key_vars.sas7bdat                                   (4.6 MB; raw Job Corps source, Schochet et al. 2008)
├── education/
│   ├── education.csv                                       (5.9 MB; 8-column )
│   └── wave12dataVerse.csv                                 (22 MB; raw source, Kizilcec et al. 2020)
└── joke/
    ├── RCT_joke_data.pkl                                   (39 MB; preprocessed (X, A, Y))
    ├── jester-data-1.xls                                   (16 MB; raw Jester source)
    └── jester-data-2.xls                                   (15 MB; raw Jester source)

Both the analysis-ready files and their raw source files are included in this archive. Variable definitions for the three analysis-ready files are in Section 5.

personalization_revision_real_domain_results_round2.tar.gz → /real_domains

Saved per-run result files used to plot the real-domain figures.

real_domains/
├── jobcorps/
│   ├── jobcorps_experiment_results_100runs_allpolicytypes.pkl
│   ├── jobcorps_pk_results_100runs_pitype_dr_econml_2_rmodel_random_forest.pkl
│   ├── jobcorps_pk_results_200runs_pitype_dr_econml_2_rmodel_random_forest.pkl
│   ├── jobcorps_srp_tstats_100runs.csv                     (SRP baseline test statistics)
│   └── jobcorps_srp_VD_100runs.csv                         (SRP baseline policy-value differences)
├── education/
│   ├── education_experiment_results_100runs_allpolicytypes.pkl
│   ├── education_pk_100runs_pitype_dr_econml_rmodel_random_forest.pkl
│   └── education_traineval_results_100runs_allpolicytypes.pkl
└── joke/
    ├── joke_experiment_results_100runs_allpolicytypes.pkl
    ├── joke_pk_100runs_pitype_joker_lasso_flat_rmodel_random_forest.pkl
    ├── joke_pk_100runs_pitype_joker_lasso_flat_rmodel_random_forest_subsets.pkl
    └── joke_pk_200runs_pitype_joker_lasso_flat_rmodel_random_forest.pkl

personalization_revision_simulation_dataset_round2.tar.gz → /simulation_dataset

Generated simulation datasets used as inputs to the simulation analyses, plus the generation scripts. This archive contains 2,342 CSV files organized into five subfolders. Because the CSVs are systematically generated, they are described by folder, naming pattern, and count rather than listed individually.

simulation_dataset/
├── gen_sim_data.ipynb                  (notebook: generates the main simulation CSVs)
├── gen_sim_data.py                     (script form of the generator)
├── gen_hte_sim_data.ipynb             (notebook: generates the HTE-test simulation CSVs, for Table S4)
├── hte_test/                           (20 CSVs)
├── no_hte_test/                        (20 CSVs)
├── sim_ate_null_personalization/       (202 CSVs)
├── threshold_large/                    (1,000 CSVs)
└── threshold_small/                    (1,100 CSVs)

File-naming convention. Each data file is named data_gap_<GAP>_seed_<SEED>.csv, where <GAP> is the personalization-gap parameter (formatted to 4 decimals) and <SEED> is the RNG seed. Per folder:

Each CSV is one simulated dataset for the given (gap, seed); column definitions are produced by the generator and shared across files in a folder.

personalization_revision_simulation_results_round2.tar.gz → /simulation

(ZIP archive — open with unzip.) Saved simulation result files used to plot the simulation figures and tables. Sizes are uncompressed.

simulation/
├── sim_threshold_large_pk_results_100runs.pkl               (10.08 GB; KPT results, large-threshold sim)
├── sim_threshold_small_pk_results_100runs.pkl               (5.28 GB; KPT results, small-threshold sim)
├── null_sim_ate_null_personalization_pk_results_100runs.pkl (960 MB; KPT, 0-ATE / positive-personalization sim)
├── no_hte_test_pk_results_20runs.pkl                         (32 MB; KPT results, no-HTE simulation)
├── hte_test_pk_results_20runs.pkl                            (16 MB; KPT results, HTE-present simulation)
├── sim_threshold_large_PAPD_results_100runs.pkl            (114 KB; PAPD baseline, large-threshold)
├── sim_threshold_small_PAPD_results_100runs.pkl            (60 KB; PAPD baseline, small-threshold)
├── sim_thres_large_SRP_tstats.csv                          (18 KB; SRP baseline test statistics, large)
├── sim_thres_large_SRP_VD.csv                              (20 KB; SRP baseline policy-value differences, large)
├── sim_thres_small_SRP_tstats.csv                          (20 KB; SRP baseline test statistics, small)
├── sim_thres_small_SRP_VD.csv                              (22 KB; SRP baseline policy-value differences, small)
├── hte_test/                                                (HTE-test benchmark outputs, HTE present)
│   ├── sim_HETCV_pval_HTE.csv                               (HETCV p-values)
│   ├── sim_HETCV_stat_HTE.csv                               (HETCV test statistics)
│   └── sim_RATE_pooled_HTE.csv                              (pooled RATE results)
└── no_hte_test/                                             (HTE-test benchmark outputs, HTE absent and present)
    ├── sim_HETCV_pval_HTE.csv
    ├── sim_HETCV_pval_no_HTE.csv
    ├── sim_HETCV_stat_HTE.csv
    ├── sim_HETCV_stat_no_HTE.csv
    ├── sim_RATE_pooled_HTE.csv
    └── sim_RATE_pooled_no_HTE.csv

personalization_code.tar.gz → /personalization_code

All analysis code.

personalization_code/
├── README.md
├── environment_kfold.yml                          (conda environment specification)
├── domains/
│   ├── README.md
│   ├── helper_func.py
│   ├── jobcorps/
│   │   ├── DataDoc/
│   │   │   ├── DataDoc_JC_Volume_I_MS20060076.pdf (from Schochet et al. 2008)
│   │   │   ├── DataDoc_JC_Volume_II_MS20060076.pdf (from Schochet et al. 2008)
│   │   │   ├── DataDoc_JC_Volume_III_MS20060076.pdf (from Schochet et al. 2008)
│   │   │   ├── DataDoc_JC_Volume_IV_MS20060076.pdf (from Schochet et al. 2008)
│   │   │   └── ReadMe.pdf (from Schochet et al. 2008)
│   │   ├── preprocess_jobcorps_data_from_sas.ipynb         (builds the semi-synthetic .csv from SAS source)
│   │   ├── analyze_jobcorps_Pk.ipynb                       (Pk / TrainEval / SRP effects)
│   │   ├── plot_figureS9S14.ipynb
│   │   └── plot_figureChoiceS_Jobcorps.ipynb              (Figure S15)
│   ├── education/
│   │   ├── preprocess_education_data.R                     (builds education.csv from raw source)
│   │   ├── process_education.ipynb                         (Pk / TrainEval / SRP effects)
│   │   └── plot_figureS11S13.ipynb
│   ├── joke/
│   │   ├── process_joke_data.ipynb                         (builds RCT_joke_data.pkl from raw .xls)
│   │   ├── run_joke_Pk.ipynb                               (Pk / TrainEval / SRP effects)
│   │   ├── plot_figureS12.ipynb
│   │   ├── plot_joke_varyS.ipynb                           (Figure S16)
│   │   └── plot_joke_varyN.ipynb                           (Figure S17)
│   └── shi_nefazodone/
│       ├── preprocess_nefazodone_data.r                    (restricted data; see Section 4)
│       ├── process_nefazodone_pk.ipynb                     (Pk / TrainEval / SRP effects)
│       └── plot_figureS10.ipynb
├── kfold/
│   ├── README.md
│   ├── __init__.py
│   ├── config/
│   │   ├── __init__.py
│   │   ├── config.py
│   │   └── params.json
│   ├── methods/
│   │   ├── __init__.py
│   │   ├── Pk_jan13.py                                     (K-fold Personalization test implementation)
│   │   ├── PAPD_new.py
│   │   ├── baselines.py
│   │   ├── influence_function.py
│   │   ├── learners.py
│   │   ├── online_learners.py
│   │   ├── training.py
│   │   └── utils.py
│   └── simulation/
│       ├── instance.py
│       ├── utils.py
│       ├── helper_plot_func.py
│       ├── plot_ci_styled.py
│       ├── plot_figure3.ipynb                              (Figure 3)
│       ├── plot_figure3_withPAPD.ipynb
│       ├── plot_figureS3S4_table_0ATE_pos_pers.ipynb       (Figures S3–S4, Table S5)
│       ├── plot_tableS2.ipynb                              (Table S2)
│       └── policy_acc_analysis/
│           ├── policy_acc.py
│           ├── run_policy_acc.ipynb                        (generates simulation results)
│           ├── run_hte_test.ipynb                          (KPT results for HTE test, Table S4)
│           ├── plot_figureS5.ipynb
│           ├── plot_figureS6S7.ipynb                       (Figures S6–S7, Table S1)
│           └── plot_figureS8.ipynb
├── srp_code/
│   ├── readme_shi_code.md                                  (build/setup instructions for the SRP baseline)
│   ├── RPtest.cpp                                          (compiled via R CMD SHLIB)
│   ├── Makevars
│   ├── Makevars_windows
│   ├── SRP_simulation.r
│   ├── SRP_jobcorps.r
│   └── Nefazodone_srp_100splits.r
├── hte_test_code/
│   ├── RATE_Nefazodone.R                                   (Table S3)
│   ├── RATE_simulation.R                                   (HTE benchmark results, Table S4)
│   ├── plot_tableS4.ipynb                                  (Table S4)
│   └── table_s3.tex
└── plots/
    ├── helper_func.py

---

Where to find each data file (quick reference)

Files and variables

jobcorps_cleaned_nonan_full_with_offset.csv — Job Corps semi-synthetic data. Definitions are from Schochet et al. 2008. https://www.openicpsr.org/openicpsr/project/113269/version/V1/view (DataDoc_JC_Volume_IV_MS20060076.pdf)

Original data from Schochet et al. 2008 (https://www.openicpsr.org/openicpsr/project/113269/version/V1/view).
To this we added a Semi-synthetic signal: for ages 18–19 (AGEGROUP == 2.0), a −5 offset is added
to EARNY4 for the treatment group and a +30 offset for the control group;
negative earnings are then clipped to 0. This injects a pattern in which
treatment is not beneficial for ages 18–19 while other groups are unchanged.

education.csv — Educational intervention dataset (preprocessed; 199,517 rows × 8 columns)

The columns below are the actual fields in education.csv. Definitions are
inferred from the source study (Kizilcec et al., 2020).

The raw source wave12dataVerse.csv (224,414 rows × 20 columns) is from (Kizilcec et al., 2020, https://osf.io/9bacu/).
That data additionally contains likely_complete_1, weekone–weekthree, grade,
weekOneEvents–weekThreeEvents, focalSample, days.from.start, strata,
and courseID; education.csv is the 8-column analysis subset derived from it.

RCT_joke_data.pkl — Jester joke recommender dataset (preprocessed)

A Python pickle storing a single dict with three aligned arrays over 48,445
records (open with pickle.load; requires numpy):

The raw source files jester-data-1.xls and jester-data-2.xls come from https://goldberg.berkeley.edu/jester-data/
and hold the original user × joke rating matrix (ratings on a −10 to +10 scale, with 99
denoting "not rated"); RCT_joke_data.pkl is the preprocessed (X, A, Y)
form used in the analysis.

Nefazodone–CBASP clinical trial data (restricted — not included)

Raw files (bmsdata_nomissing.txt and notes / readme.txt) are access-restricted
and are not part of this Dryad submission (see Access information), nor are the processed experimental output.

Result files (.pkl, .csv) in /real_domains and /simulation

These hold processed experimental output for plotting rather than raw measurements.
Each pickle stores per-run estimates of the personalization test statistic and
associated policy-value quantities (one record per repeated run).

---

Software and how to open the files

All file types in this submission can be opened with free, open-source software.

The source Jester data are distributed by the original provider as .xls
(Excel) files; these open in LibreOffice Calc or Gnumeric. The preprocessed,
openly readable version used in our analysis is provided as RCT_joke_data.pkl.

---

Reproducing Results

We provide scripts and notebooks to generate all paper figures. This code is available in the provided Zenodo link.
We discuss the pre-processing steps involved below, after the reproduction steps.

Step 0: Environment setup

• environment_kfold.yml is a yml file to set up the packages used in the code
• Reproducing the SRP results (used in jobcorps, nefazodone and simulation) requires first compiling a .cpp file from the SRP codebase
  See srp_code/readme_shi_code.md for details on the steps to do this

Step 1: Reproduce Pk, TrainEval and SRP (as relevant) personalization effects

The domains folder contains datasets used in the experiments, along with preprocessing scripts.

jobcorps/ – Job Corps Semi-Synthetic Dataset

Open and run domains/jobcorps/analyze_jobcorps_Pk.ipynb. This will put results for Pk, TrainEval and SRP inside paper_results/jobcorps folder.

---

shi_nefazodone/ – Nefazodone-CBASP Clinical Trial Data (Restricted)

• Access restricted — available to editors/reviewers via OSF during review.

Reproduction steps
A. Obtain data from OSF (link provided to editor).
B. Place the dataset files in domains/shi_nefazodone/.
C. Run domains/nefazodone/process_nefaozodone_pk.ipynb . This will put results for Pk, TrainEval and SRP inside paper_results/nefazodone folder.

---

education/ – Educational Intervention Dataset

Run domains/education/process_education.ipynb to get results inside paper_results/education folder.

---

joke/ – Jester Joke Recommender Dataset

Open and run domains/joke/run_joke_Pk.ipynb and get results inside paper_results/joke folder.

---

simulation_dataset/ - Simulated Environment

Reproduction steps

1. Unzip simulation_dataset.tar and place the resulting simulation_dataset directory inside the domains/ directory.
2. Run kfold/simulation/policy_acc_analysis/run_policy_acc.ipynb. This will store simulation results inside paper_results/simulation folder and simulation datasets inside domains/simulation_dataset/ folder.
3. Generate RPtest.so using this code in command line: rm -f RPtest.o RPtest.so RPtest.dylib and R CMD SHLIB RPtest.cpp. This enables executing the SRP baseline. The full instruction for installing environments is in in srp_code/readme_shi_code.md.
4. Run srp_code/SRP_simulation.r to get SRP results. Note: steps (1) and (2) must be done before this.

Step 2: Create figures

Figure 1

File to generate: n/a
Notes: Conceptual illustration of when HTE ≠ personalization benefit; not code-generated.

---

Figure 2

File to generate: n/a
Notes: Static schematic of KPT pipeline (not code-generated).

---

Figure 3

File to generate: kfold/simulation/plot_figure3.ipynb
Notes: Requires data generated in paper_results folder.

---

Figure S3 & S4 (Simulation Results) and Table S5 (0 ATE, Positive Personalization Effect Simulation)

File to generate: kfold/simulation/plot_figureS3S4_table_0ATE_pos_pers.ipynb
Notes: Expects simulation pickles in paper_results/simulation/*.

---

Figure S5

File to generate: kfold/simulation/policy_acc_analysis/plot_figureS5.ipynb
Notes: Expects simulation pickles in paper_results/simulation/sim_thres_*.

---

Figure S6 & S7

File to generate: kfold/simulation/policy_acc_analysis/plot_figureS6S7.ipynb
Notes: Assumes all data for all domains is generated and uses paper_results/simulation/*shi_tstats*.csv and *shi_VD*.csv, along with simulation pickles.

---

Figure S8

File to generate: kfold/simulation/policy_acc_analysis/plot_figureS8.ipynb
Notes: Expects simulation pickles in paper_results/simulation/sim_thres_*.

---

Figure S9 & S14

File to generate: domains/jobcorps/plot_figureS9S14.ipynb
Notes: Expects jobcorps results in paper_results/jobcorps/jobcorps_pk_100runs_allpolicytypes.pkl, obtained from analyze_jobcorps_Pk.ipynb.

---

Figure S10 (s10_combined_nefazodone.png)

File to generate: domains/shi_nefazodone/plot_figureS10.ipynb
Notes: Expects nefazodone results in paper_results/jobcorps/nefazodone_experiment_results_100runs_allpolicytypes.pkl, obtained from process_nefazodone_Pk.ipynb.

Figure S11 & S13 (s10_combined_educ.png, s11_traineval_cdf_educ.png)

File to generate: domains/education/plot_figureS11S13.ipynb
Notes: Expects education results in paper_results/education/education_experiment_results_100runs_allpolicytypes.pkl, obtained from process_education.ipynb.

Figure S12 (s10_combined_joke.png)

File to generate: domains/joke/plot_figureS12.ipynb
Notes: Expects joke results in paper_results/joke/joke_experiment_results_100runs_allpolicytypes.pkl, obtained from run_joke_Pk.ipynb.

Figure S15 (semi-synthetic JobCorps for varying number of splits)

File to generate: domains/jobcorps/plot_figureChoiceS_Jobcorps.ipynb
Notes: Expects jobcorps results in paper_results/jobcorps/jobcorps_pk_results_200runs_pitype_dr_econml_2_rmodel_random_forest.pkl', obtained from analyze_jobcorps_Pk.ipynb`.

Figure S16 (Joke for varying number of splits)

File to generate: domains/joke/plot_joke_varyS.ipynb
Notes: Expects joke results in paper_results/joke/joke_pk_200runs_pitype_joker_lasso_flat_rmodel_random_forest.pkl', obtained from run_joke_Pk.ipynb`

Figure S17 (Joke for varying subsets of the dataset)

File to generate: domains/joke/plot_joke_varyN.ipynb
Notes: Expects joke results in paper_results/joke/joke_pk_100runs_pitype_joker_lasso_flat_rmodel_random_forest_subsets.pkl', obtained from run_joke_Pk.ipynb`

Table S1

File to generate: kfold/simulation/policy_acc_analysis/plot_figureS6S7.ipynb
Notes: Requires data generated in paper_results folder.

Table S2

File to generate: kfold/simulation/plot_tableS2.ipynb
Notes: Requires data generated in paper_results folder.

Table S3

Notes: Table data generated from running hte_test_code/RATE_Nefazodone.R. We used R version 4.3.3, grf package version 2.4.0 and evalITR package version 1.0.0.

Table S4

File to generate: hte_test_code/plot_tableS4.ipynb
Notes: Requires running domains/simulation_dataset/gen_hte_sim_data.ipynb first to generate simulation data for HTE test, then running hte_test_code/RATE_simulation.R to generate HTE test results for benchmark methods, and running kfold/simulation/policy_acc_analysis/run_hte_test.ipynb to generate KPT results for the same simulation data. For running R, we used R version 4.3.3, grf package version 2.4.0 and evalITR package version 1.0.0.

---

Dataset preprocessing

For all public datasets, we have included a snapshot of the preprocessed data used
for generating personalization results as part of the repo. Here we provide the code
and information for the location of the source data, and preprocessing.

1. Preprocessing: jobcorps/ – Job Corps Semi-Synthetic Dataset

Source

• We use impact.sas7bdat and key_vars.sas7bdat from the data repo of Schochet et al., 2008.

Key files

• jobcorps_cleaned_nonan_full_with_offset.csv: Semi-synthetic outcome file produced by the preprocess script.
• process_jobcorps_Pk.ipynb: Notebook to perform the transformation.

Semi-synthetic signal we inject

• For participants of Ages 18–19 (AGEGROUP == 2.0):
  • Treatment group: add a negative offset -5 to EARNY4.
  • Control group: add a positive offset +30 to EARNY4.
• After applying offsets, we clip negative earnings to 0 to keep the outcome domain valid.

This produces a known heterogeneity pattern: treatment is harmful for ages 18–19 relative to control, while other age groups are left unchanged.

Open and run preprocess_jobcorps_data_from_sas.ipynb to generate the processed semi-synthetic data jobcorps_cleaned_nonan_full_with_offset.csv. This latter file is already included in the repository.

2. shi_nefazodone/ – Nefazodone-CBASP Clinical Trial Data (Restricted)

Goal. Analyze policy-learning methods in a real-world clinical trial with multiple treatment arms.

Key files

• bmsdata_nomissing.txt: Text-based format without missing entries. (available given permission)
• notes and readme.txt: Documentation from the original data contributor.

Source

• Nefazodone-CBASP Clinical Trial ([Keller et al., 2000])
• Access restricted — available to editors/reviewers via Dryad during review.

Variables/features

• Patient demographics: age, gender, comorbidities.
• Baseline depression measures (HRS-D scores).
• Treatment group: Nefazodone only, CBASP therapy only, or both.
• Outcome measures: Post-treatment HRS-D scores.

Preprocessing steps

• No additional initial preprocessing needed, done as part of the process_nefazodone_pk.ipynb notebook.

3. Preprocessing: education/ – Educational Intervention Dataset

Goal. Evaluate personalization in an educational setting with multiple treatment arms and student-level covariates.

Key files

• education.csv: Main dataset for experiments.
• process_education.ipynb: Processing script for the cleaned dataset.
• Data Analysis Scripts/: R scripts for reproducing original study analyses (tables, figures, balance checks).

Source

• We use wave12dataVerse.csv from a publicly shareable dataset in the repo of Kizilcec et al., 2020.

To reproduce the initial preprocessed data, open and run domains/education/process_data.R to generate education.csv.

4. joke/ – Jester Joke Recommender Dataset

Goal. Provide a public, real-world recommendation dataset for evaluating personalization in settings with numeric ratings.

Source

• We use jester-data-1.xls (jester-data-1.zip) and jester-data-2.xls (jester-data-2.zip) from publicly available dataset on Jester Online Joke Recommender System.

To reproduce the initial preprocessed data, open and run domains/joke/process_joke_data.ipynb to obtain processed data RCT_joke_data.pkl.

---

Citation

When citing this code, please use:

DOI: 10.5061/dryad.bg79cnpp7