This dataset accompanies a paper that uses a Gaussian Mixture Model (GMM) to analyse how a large language model's (LLM's) output changes when the order of the answer options is permuted. The method supports two scenarios and two tasks:

• Scenarios
  • Classification: every question in a dataset shares the same fixed set of options (e.g., {positive, negative}); only the order in which the options are presented is permuted.
  • MCQ (multiple-choice question): each question has its own specific set of options.
• Tasks
  • Prediction: predict the answer of a classification / MCQ question from the LLM outputs collected under several option orders.
  • Abstention (MCQ only): decide, for each MCQ question, whether the model should abstain (i.e. decline to answer) because it is likely to be wrong — also known as selection classification.

Both tasks rely on posterior probabilities produced by the GMM.

---

Files in this submission

All files are placed at the top level of this submission (there are no subfolders for the code):

.
├── LLM-MCQ.py                  # generates LLM logit-score outputs for MCQ datasets (needs raw text data)
├── LLM-classification.py       # generates LLM logit-score outputs for classification datasets (needs raw text data)
├── gmm_mcq.py                  # GMM analysis for the MCQ scenario (runs directly on the uploaded data)
├── gmm_classification.py       # GMM analysis for the classification scenario (runs directly on the uploaded data)
├── abstention_mcq.py           # abstention evaluation for the MCQ scenario (runs directly on the uploaded data)
└── Data_for_github_(Desensitized).zip   # the data (LLM logit scores + pre-computed GMM results)

The two LLM-*.py scripts cannot be run directly from this submission, because they require the original raw text datasets as input, and those raw datasets are not distributed here for licence reasons (see "Data description" section below). The three GMM/abstention scripts can be run directly on the data inside the zip and reproduce the paper's results.

---

Data description

All data is stored in Data_for_github_(Desensitized).zip. Unzipping it produces a single top-level folder, Data for github/, which contains 13 datasets. Each dataset has the same layout:

Data for github/
└── <dataset name>/
    ├── LLM results/                        # the LLM's logit-score outputs, one JSONL file per option order
    │   ├── GLM4/
    │   ├── llama2/
    │   └── llama3/
    ├── calibrated results/                 # pre-computed GMM results (single-dataset: estimated and applied on the same dataset)
    │   ├── GLM4/
    │   ├── llama2/
    │   └── llama3/
    └── calibrated results(param transfer)/ # ONLY for the MCQ datasets ARC, MedMCQA, MedQA:
                                            # pre-computed GMM results where the GMM is estimated on MMLU
                                            # and transferred to this dataset
        ├── GLM4/
        ├── llama2/
        └── llama3/

The 13 datasets, grouped by scenario, are:

Three LLMs are used: GLM4, llama2 and llama3 (one subfolder per model under each results folder).

The LLM-results files (JSONL)

These are the primary data. For each (dataset, subset, model, option order) combination there is one JSONL file. Each line is one question/record. The filename encodes the subset and the option order:

<LLM results>/<model>/<subset>_results_<order>.jsonl

• <subset> is the data split. Most datasets use train and test; MedQA uses train and devtest.
• <order> is the option-order string (the suffix). 0123 is the default order; any other suffix permutes the options. Convention: if the suffix is denoted $\tau$, then the option that is the $i$-th in the default order is placed at the $\tau(i)$-th position, and suffix = τ(0)τ(1)…τ(C-1). For example, for a 4-option question, 0123 is the default order and 1023 swaps the first two options. The number of orders differs by dataset: the 2-, 3- and 4-option classification datasets use all permutations of the options, while the 5-option datasets (BBC News, CSQA) and the MCQ datasets use the $C$ cyclic-shift orders (e.g. 0123, 1230, 2301, 3012 for 4 options).

Each JSONL record contains:

• scores — a list of numbers, the LLM's logit score for each option, listed in presentation order (the order in which the options were shown to the LLM in that file). A logit score is the raw, unnormalised output of the LLM for an option; larger means the LLM considers that option more likely. The GMM scripts convert these logits into probabilities internally.
• label (classification datasets) or answer (MCQ datasets) — an integer, the index of the correct option. The GMM scripts read the ground truth from the default-order file (suffix 0123, 01, 012, …), where the presentation order equals the default order, so this index is the position of the correct option in the default order. (In the non-default-order files the same field records the correct option's position in that file's presentation order; the GMM scripts only use it from the default-order file.)
• label_text (classification datasets only) — the human-readable name of the correct label (e.g. negative, positive).
• A few MCQ datasets also keep auxiliary fields such as id (MedQA, ARC), or subject / category (MMLU); these are not used by the GMM scripts.

How these files are created. Each scores list is produced by presenting the question to an LLM (with the options arranged in the order indicated by the suffix) and reading the LLM's logit score for each option token (A, B, C, …). This is exactly what LLM-MCQ.py and LLM-classification.py do. Because this step needs the original raw text of each dataset (the questions and the option texts), and those raw datasets are not distributed here, the two LLM-*.py scripts are not directly runnable from this submission alone. The raw text datasets are all publicly available (e.g. on GitHub or Hugging Face); the original raw datasets and the original per-model LLM-generation scripts can also be found in the companion GitHub repository https://github.com/ito-integration/GMM-option-orders.

The calibrated-results files (JSON)

The calibrated results/ and calibrated results(param transfer)/ folders contain pre-computed reference results produced by the GMM pipeline, so that users can verify their own runs without re-running everything. There is one JSON per (dataset, model):

• classification datasets: calibrated results/<model>/results.json
• MCQ datasets: calibrated results/<model>/single_results.json (single-dataset), and, for ARC / MedMCQA / MedQA only, calibrated results(param transfer)/<model>/single_results.json (parameter transfer from MMLU).

Each such JSON records: the accuracy of the GMM method and of the baseline methods; the per-option-order accuracies; the per-sample posterior majority probabilities (GMM_scores, usable as prediction confidence); the per-sample inconsistency scores (used for abstention); the abstention threshold(s); and the estimated GMM parameters. (These files were produced by the same pipeline; depending on the version, some baseline key names may differ slightly from those printed by the current scripts, but the GMM accuracy and scores are directly comparable.)

The paper's results can be reproduced directly from the data in each dataset's LLM results/ folder using gmm_mcq.py / gmm_classification.py; the calibrated results/ folder is provided only as a reference.

---

Python scripts

The five scripts form a pipeline. In plain language:

Raw text datasets → LLM-MCQ.py / LLM-classification.py → LLM logit-score JSONL files (the uploaded LLM results/) → gmm_mcq.py / gmm_classification.py → GMM result JSON → abstention_mcq.py → abstention result JSON.

Only the middle and last stages (the GMM and abstention scripts) can be run directly on this submission's data; the first stage needs the raw text datasets.

LLM-classification.py

• Purpose: use a locally-deployed LLM (here GLM-4) to answer each question of a classification dataset under each option order, and record the per-option logit scores.
• Inputs: a raw classification dataset (read from a local raw data/ subfolder) and a local LLM model checkpoint. Not included in this submission.
• Outputs: one JSONL file per option order, <output-dir>/GLM4-<dataset>-<subset>_results_<order>.json, where each record carries the original fields plus a scores list (logits in presentation order). (These are the files stored under LLM results/<model>/ in the zip, with the leading GLM4-<dataset>-<subset>_ prefix removed.)
• Note: not directly runnable from this submission (needs raw text data).

LLM-MCQ.py

• Purpose: same as above, but for MCQ datasets (MedQA is the worked example in the script).
• Inputs: a raw MCQ dataset (local raw data/ subfolder) and a local LLM model checkpoint. Not included in this submission.
• Outputs: one JSONL file per option order, <output-dir>/prompt<prompt>_<subset>_results_<order>.jsonl, each record carrying the original fields plus a scores list (logits in presentation order).
• Note: not directly runnable from this submission (needs raw text data).

gmm_classification.py

• Purpose: estimate a separate GMM per label on the LLM logit scores of a classification dataset, then compute the posterior probability that a majority of option orders select each label, and predict the answer. Also reports the Avg, MV and PriDe baselines for comparison.
• Inputs: the LLM-results JSONL files for a chosen (dataset, model) — i.e. the uploaded LLM results/<model>/<subset>_results_<order>.jsonl files — given as path prefixes. One prefix for estimation (train), one for application (test); option orders; and optional censoring.
• Outputs: a result JSON (accuracy of GMM/Avg/MV/PriDe, per-order accuracies, GMM confusion matrix, GMM_scores, inconsistent_scores, thresholds, estimated coefs, and config) plus a console summary. Optionally also a saved-parameters JSON.

gmm_mcq.py

• Purpose: estimate a single shared GMM (on the probability of label 0 at each position) and apply it to every option of every question — a parameter-transfer setting (estimate on one MCQ dataset, e.g. MMLU, apply to another, e.g. MedQA). Reports the same baselines.
• Inputs: the uploaded LLM results/<model>/<subset>_results_<order>.jsonl files (path prefixes), as estimation data and application data; the $C$ cyclic-shift orders (--num-options C); optional censoring.
• Outputs: a result JSON (same fields as the classification script, with thresholds as a scalar) plus a console summary. Optionally also a saved-parameters JSON.

abstention_mcq.py

• Purpose: evaluate several abstention methods in the MCQ scenario, reporting R-Acc (accuracy among retained/answered samples) and A-F1 (F1 of the abstain decision) for every (method, option-order) pair.
• Inputs: the result JSON produced by gmm_mcq.py (for the GMM abstention method's threshold and inconsistency scores); plus the uploaded LLM-results JSONL files for calibration (train) and evaluation (test). The evaluation data must be the same data gmm_mcq.py was applied to, so the rows stay aligned.
• Outputs: a result JSON (methods, orders, R_Acc / A_F1 per method per order, means over orders, raw_accs, config) plus a console table.

---

Dependencies

numpy, pandas, scipy, scikit-learn, mpmath

(LLM-MCQ.py and LLM-classification.py additionally require torch and transformers, since they run an LLM.)

---

Reproducing the results

First unzip the data archive. Then, from the folder that contains the scripts, the GMM/abstention scripts can be run directly. The examples below use the GLM4 model and reference the unzipped data paths.

Classification scenario (MedNLI, GLM4, 3 options):

python gmm_classification.py 
    --estimation-data-path "Data for github/MedNLI/LLM results/GLM4/train_results_" 
    --application-data-path "Data for github/MedNLI/LLM results/GLM4/test_results_" 
    --all-orders "012;021;102;120;201;210" --use-censor --censor-ratio 0.01 
    --result-path MedNLI_GLM4_results.json

MCQ scenario — parameter transfer (estimate the shared GMM on MMLU train, apply to MedQA devtest, both GLM4, 4 options):

python gmm_mcq.py 
    --estimation-data-path "Data for github/MMLU/LLM results/GLM4/train_results_" 
    --application-data-path "Data for github/MedQA/LLM results/GLM4/devtest_results_" 
    --num-options 4 --use-censor --censor-ratio 0.01 
    --save-params-path MMLU_GLM4_coefs.json 
    --result-path MedQA_GLM4_results.json

Abstention scenario (MCQ, MedQA, GLM4; reuses the gmm_mcq.py result above):

python abstention_mcq.py 
    --gmm-result-path MedQA_GLM4_results.json 
    --estimation-data-path "Data for github/MedQA/LLM results/GLM4/train_results_" 
    --application-data-path "Data for github/MedQA/LLM results/GLM4/devtest_results_" 
    --num-options 4 
    --result-path MedQA_GLM4_abstention.json

For an MCQ dataset estimated and applied on itself (no transfer), pass the same dataset's train_results_ and test_results_ (or devtest_results_ for MedQA) as the estimation and application paths. To re-apply previously estimated parameters, replace --estimation-data-path with --saved-params-path.

---

Detailed reference

Data format expected by the GMM scripts

The GMM scripts read the LLM logit scores saved under each option order, one JSONL file per order, named {base_path}{suffix}.jsonl, where suffix is the option-permutation string (see above). Each JSONL record must contain:

• scores: a list of the per-option logit scores, in presentation order;
• label: an integer, the index of the correct option in the default order (the field name answer is also accepted).

--estimation-data-path / --application-data-path are both path prefixes; the script automatically appends {suffix}.jsonl.

Specifying option orders (all_orders)

Specify via any of the following (in order of precedence, from highest to lowest):

• --all-orders-csv PATH: a CSV file with one permutation per line (comma-separated such as 0,1,2,3, or a digit string such as 0123); the k-th line corresponds to the k-th suffix read.
• --all-orders "0123;3021;2301;1230": inline, separated by ;, where each segment may be a digit string or comma-separated.
• --num-options N: conveniently generates $N$ cyclic-shift orders (consistent with the paper's MCQ setting); this is the default usage in the MCQ scenario.

Note: all_orders[0] must be the default (identity) order, because the PriDe method uses the probability under the 0-th order for prediction.

Command-line arguments — gmm_classification.py / gmm_mcq.py (identical)

Three run modes are supported:

1. Estimation + Application: provide both --estimation-data-path and --application-data-path.
2. Application only (parameter transfer): provide --saved-params-path and --application-data-path; estimation is skipped.
3. Estimation only: provide only --estimation-data-path (and --save-params-path); the posterior is not computed.

At least one of --estimation-data-path or --saved-params-path must be provided.

Censoring and saved parameters

• When --use-censor is enabled: in the parameter estimation stage, the uncensored EM is first used for initialization, and then the censored EM is used for estimation; the posterior probability stage likewise uses the censored GMM (i.e., out-of-bounds values in the test z-scores are clamped to the censoring bounds and passed into the posterior computation).
• The estimated parameter JSON saves left_censor / right_censor (z-score censoring bounds, per dimension).
• When --saved-params-path is used together with --use-censor, the script reads left_censor / right_censor from the parameter file; if they are missing from the file, an error is raised (i.e., "the left and right censoring bounds must be provided").
• When censoring is disabled: only the uncensored EM is used, and no censoring bounds are passed into the posterior computation.

Output of the GMM scripts

After the run finishes, the accuracies of the four methods (GMM / Avg / MV / PriDe), the per-order individual accuracies, and the GMM confusion matrix are printed, and the results are saved to the JSON specified by --result-path, which contains: GMM_acc / Avg_acc / MV_acc / PriDe_acc / raw_accs, GMM_scores (the posterior majority probability, usable for prediction), inconsistent_scores and thresholds (for abstention), coefs (the estimated parameters), and config (the run configuration).

Abstention — abstention_mcq.py

abstention_mcq.py evaluates the performance of various abstention methods under each option order in the MCQ scenario, outputting R-Acc and A-F1 for each (method, option order) pair (point estimates; no 95% CI is computed). It takes the result JSON from gmm_mcq.py as one of its inputs (providing the thresholds and inconsistent_scores required by the GMM abstention method).

Command-line arguments

The seven abstention methods (fixed order)

Evaluation metrics

For each option order $k$, define golden = (prediction ≠ true label) using the raw argmax prediction under that order (1 = incorrect, should abstain):

• R-Acc (Retained Accuracy) = the proportion of correct predictions among the retained (non-abstained) samples;
• A-F1 (Abstention F1) = the F1 of the abstention decision (1 = abstain) against golden (1 = should abstain).

The features and thresholds of Conf/Ent vary with $k$; the features of KL/Var are order-independent, but their thresholds are re-calibrated per order based on "the correctness of order $k$" (consistent with the original code); the abstention decisions of GMM/Avg Conf/Cons Pre are independent of $k$, but their R-Acc/A-F1 still vary with $k$ (because golden varies with $k$).

Output

Prints the R-Acc, A-F1, and the mean across orders for each method. The result JSON contains methods, orders/orders_str, R_Acc/A_F1 ($[n_methods, K]$), R_Acc_mean_over_orders/A_F1_mean_over_orders, raw_accs (the raw accuracy of each order), and config.