This directory contains all datasets for training and testing dpvt models as described in the manuscript Unifying phylogenetic traversal and deep learning to guide tree exploration.
We provide alignments used to generate training and testing data in simulated_alignments/ and empirical_alignments/ and the pickled data with the correct input format for dpvt models in dpvt_data/.

None of the empirical_alignments/ source alignments are included in this upload — they must be downloaded before use with the provided scripts. See Obtaining the empirical alignments below.

Directory structure

data_for_dryad/
├── empirical_alignments.tar.gz/
│   ├── influenzaC_fluC_M/           # populated by download_endemic_viruses.py, see below
│   ├── influenzaC_fluC_NS/          # populated by download_endemic_viruses.py, see below
│   ├── influenzaC_fluC_PB2/         # populated by download_endemic_viruses.py, see below
│   ├── rotavirusA_H_H2/             # populated by download_endemic_viruses.py, see below
│   ├── download_endemic_viruses.py
│   ├── orthomam/                    # populated by download_orthomam.py, see below
│   ├── orthomam_train_ids.txt
│   ├── orthomam_test_ids.txt
│   ├── download_orthomam.py
│   ├── pandit/                      # populated by download_pandit.py, see below
│   ├── pandit_train_ids.txt
│   ├── pandit_test_ids.txt
│   └── download_pandit.py
├── simulated_alignments.tar.gz/
│   ├── simulated_25_seq_100_sites_200_algnmnts_filtered_0.8_spr/
│   ├── simulated_25_seq_100_sites_500_algnmnts_filtered_0.8_spr/
│   ├── simulated_50_seq_100_sites_200_algnmnts_filtered_0.8_spr/
│   └── simulated_50_seq_100_sites_500_algnmnts_filtered_0.8_spr/
└── dpvt_data.tar.gz/
    └── *.p (pickle files)

Obtaining the empirical alignments

empirical_alignments/ ships three download scripts instead of the alignment files themselves. Each fetches data from its original source and lays it out to match the structure shown above.

Requirements: Python 3 with the requests package installed (pip install requests).

cd empirical_alignments
python download_endemic_viruses.py   # influenza C (M, NS, PB2) and rotavirus A (H_H2)
python download_orthomam.py          # reads orthomam_{train,test}_ids.txt
python download_pandit.py            # reads pandit_{train,test}_ids.txt

• download_endemic_viruses.py fetches the aligned FASTA files for influenza C segments M, NS, and PB2, and rotavirus A segment 11 (genotype H2), from the endemic human virus data GitHub repository (Kistler and Bedford, 2023), then cleans them to match the alignments actually used in this project (deduplicating identical sequences, stripping "." from accession-derived names, normalizing ambiguity codes, and, for the influenza segments, adding a consensus "root" sequence), and writes them to <name>/<name>.fasta. Pass --raw to skip the cleaning step and keep the untouched files as fetched from GitHub.
• download_orthomam.py fetches each gene's alignment individually from OrthoMaM v12's bulk file server, using the ID lists in orthomam_{train,test}_ids.txt (which record which alignments belong to which split).
• download_pandit.py fetches the single ~58MB PANDIT flatfile archive once (cached under empirical_alignments/.cache/), then extracts the DNA alignment for each family listed in pandit_{train,test}_ids.txt. PANDIT has been frozen since 2008 and has no per-family download endpoint, so this is unavoidable, but the file is small.

All scripts skip alignments that are already present, so they can be safely re-run or interrupted and resumed. Pass --force to re-download (for the download scripts), and --split train/--split test to fetch only one split (for the OrthoMaM/PANDIT scripts). Run any of them with --help for the full set of options.

File formats

• Alignments (empirical_alignments/, simulated_alignments/): FASTA format (.fasta)
• dpvt data (dpvt_data/): Python pickle files (.p), compatible with Python 3. These can be loaded using pickle.load() and contain the input format required by dpvt models.

All datasets in dpvt_data have been generated by the Snakemake pipeline provided in /dpvtex/larch/ of the dpvt-experiments-1 repo.

Citation

If you use this dataset, please cite the accompanying manuscript: Unifying phylogenetic traversal and deep learning to guide tree exploration.

Data sources

Empirical data

Empirical alignments are from three sources (see Obtaining the empirical alignments for how to download each):

• endemic human virus data (Kistler and Bedford, 2023): influenza C segments M, NS, and PB2, and rotavirus segment 11
• orthologous mammalian markers from the OrthoMaM database (Allio et al., 2023)
• coding DNA sequence of a variety of families of homologous protein domains from the PANDIT database (Whelan et al., 2006)

We perform the following preprocessing steps before generating the dpvt_data/ pickle files from the OrthoMaM and PANDIT alignments:

1. Remove all sequences where more than 20 % of characters are gaps or ambiguous characters.
2. Remove all sites containing gaps or ambiguous characters.
3. Remove uninformative sites (sites where all sequences have the same character, or where only one sequence differs).
4. Discard alignments that lost too many sites: at least 80 % of original sites must remain for PANDIT, or 50 % for OrthoMaM.
5. Split into 80 % training and 20 % testing sets.

For OrthoMaM, we randomly draw 1,000 alignments for training and 200 for testing.
For the endemic human viruses, we generate testing data from a single alignment per virus.

As there can be many maximum parsimony trees for an alignment, we include up to 200 trees per alignment in the dpvt_data/ sets.

Simulated data

Simulated alignments have been generated using this script.
For generating dpvt_data for simulated alignments with 50 sequences, both SPR and random subtree replacements have been used as perturbation methods.