This repository accompanies the research paper "Generalizable IoT Traffic Representations for Cross-Network Device Identification" and contains the datasets, pretrained encoder models, and processing pipeline required to reproduce the traffic representation learning methodology presented in the accompanying study.

This release builds upon our previous DATA2016 Dryad repository:

https://doi.org/10.5061/dryad.6q573n6c1

which introduced the Custom Flow representation and provides detailed documentation of the flow format, feature definitions, and dataset organization. The present repository contributes:

• DATA2025v1 and DATA2025v2 datasets
• Five pretrained encoder models
• Preprocessing and embedding-generation scripts
• Configuration files and supporting documentation

Together, the two Dryad repositories provide the complete collection of datasets, pretrained models, and software artifacts used for our studies on IoT traffic representation learning.

---

Repository Contents

This release consists of the following downloadable archives.

Datasets

This release contains two large-scale IoT traffic datasets represented using the Custom Flow format.

Note: The DATA2016 is not included in this release. It is available separately from our previous Dryad repository (linked above).

DATA2025v1

• Approximately 7.7 million Custom Flow records
• 18 IoT device types
• Collected from a university laboratory deployment
• Used for representation learning and downstream evaluation

DATA2025v2

• Approximately 8.8 million Custom Flow records
• 10 overlapping IoT device types
• Collected from a separate deployment environment
• Used exclusively for cross-environment evaluation

Both datasets consist of traffic generated by physical IoT devices operating in live network environments. They do not contain synthetically generated traffic or replay traces.

---

Parquet File Format

The DATA2025v1 and DATA2025v2 datasets are distributed as collections of Apache Parquet files. Each Parquet file contains Custom Flow records for a single day of network traffic, where each row represents one Custom Flow. All Parquet files within both datasets share the same schema.

The columns contained in each Parquet file are summarized below.

Further details of the Custom Flow representation, including its design rationale and payload encoding methodology, are available in our previous DATA2016 Dryad repository (linked above).

---

Pretrained Encoder Models

Five pretrained TensorFlow encoder models are included.

Each archive expands into the directory structure expected by the embedding-generation scripts.

For example,

ae-numeric.tar.gz
└── ae-numeric/
    └── blocks4_latent40_epochs100/
        └── final_model/

Similarly,

vae-numeric.tar.gz
└── vae-numeric/
    └── blocks4_latent40/
        └── checkpoint_epoch_100/

The default paths in config.py correspond to these extracted directory structures.

---

Processing Pipeline

The archive scripts.tar.gz contains the complete processing pipeline.

After extraction, its structure is

scripts/
├── README.md
├── config.py
├── configs/
│   ├── data2016-config.yaml
│   ├── data2025v1-config.yaml
│   └── data2025v2-config.yaml
├── modules/
│   ├── cutils.py
│   └── env_config.py
├── 1a-scale_numeric.py
├── 1b-scale_categorical.py
├── 2-consolidate_splits.py
├── 3a-generate_ae_numeric_embeddings.py
├── 3b-generate_ae_categorical_embeddings.py
├── 3c-generate_vae_numeric_embeddings.py
├── 3d-generate_ae_numeric_limited_embeddings.py
└── 3e-generate_vae_numeric_limited_embeddings.py

---

Pipeline Overview

The released pipeline consists of three stages.

Stage 1 — Custom Flows → Scaled Custom Flows

Convert unscaled Custom Flow datasets into normalized numeric or categorical representations suitable for neural-network inference.

Scripts:

• 1a-scale_numeric.py
• 1b-scale_categorical.py

---

Stage 2 — Construct Dataset Splits

Merge daily files into consolidated

• training
• validation
• testing

datasets.

Script:

• 2-consolidate_splits.py

---

Stage 3 — Generate Traffic Embeddings

Generate fixed-dimensional traffic embeddings using one of the pretrained encoder models.

Scripts:

• 3a-generate_ae_numeric_embeddings.py
• 3b-generate_ae_categorical_embeddings.py
• 3c-generate_vae_numeric_embeddings.py
• 3d-generate_ae_numeric_limited_embeddings.py
• 3e-generate_vae_numeric_limited_embeddings.py

The generated embeddings correspond to those used throughout the accompanying paper for downstream IoT device identification.

---

Quick Start

1. Download and extract

• DATA2025v1.tar.gz
• DATA2025v2.tar.gz
• the desired encoder model(s)
• scripts.tar.gz

1. Edit

scripts/config.py

to specify

• dataset locations (UNSCALED_CUSTOMFLOWS_DIR)
• environment configuration (ENVIRONMENT_CONFIG_YAML)
• encoder checkpoint paths (AE_NUMERIC, AE_CATEGORICAL, VAE_NUMERIC, AE_NUMERIC_LIMITED, VAE_NUMERIC_LIMITED)

1. Generate scaled Custom Flows

python 1a-scale_numeric.py
python 1b-scale_categorical.py

1. Construct train/validation/test splits

python 2-consolidate_splits.py

1. Generate traffic embeddings

python 3a-generate_ae_numeric_embeddings.py

or execute any of the remaining embedding-generation scripts corresponding to the desired encoder.

Detailed documentation for each processing stage is provided in

scripts/README.md

---

Requirements

• Python 3.9 or later
• TensorFlow ≥ 2.10
• pandas
• numpy
• pyarrow
• tqdm
• pyyaml
• joblib

Install the required packages using

pip install "tensorflow>=2.10" pandas numpy pyarrow tqdm pyyaml joblib

GPU acceleration is optional but recommended when generating embeddings for large datasets.

---

Reproducibility

This release is intended to support the reproducibility of the accompanying study.

The provided datasets, pretrained encoder models, configuration files, and processing scripts are sufficient to reproduce the complete traffic representation learning pipeline and regenerate the traffic embeddings used throughout the experimental evaluation.

---

Citation

We encourage users of these datasets and software artifacts to cite the accompanying paper:

A. Sivanathan, D. Warren, D. Mishra, S. Ruj, N. Fernandes, Q. Z. Sheng, M. Tran, B. Luo, D. Coscia, G. Batista, and H. Habibi Gharakheili, Generalizable IoT Traffic Representations for Cross-Network Device Identification, IEEE Transactions on Network and Service Management (TNSM), 2026 (to appear).

We also encourage users of the Custom Flow representation or the DATA2016 dataset to cite our previous Dryad release:

https://doi.org/10.5061/dryad.6q573n6c1