Summary

This repository contains the dataset and code used for the manuscript "A data-driven supervised machine learning approach to estimating global ambient air pollution concentrations with associated prediction intervals." The study developed a supervised machine learning framework to predict hourly air pollution concentrations for NO2, O3, PM10, PM2.5, and SO2 at a 0.25° spatial resolution globally. The dataset includes the raw pollutant concentrations target vector, feature vectors (temporal, meteorological, remote sensing, and emissions), and the resulting predictions with associated prediction intervals.

File Structure

The file structure for this repository is as follows:

Output_Data_for_A_Data-Driven_Supervised_Machine_Learning_Approach_to_Estimating_Global_Ambient_Air_Pollution_Concentrations_With_Associated_Prediction_Intervals.zip: This folder contains all of the output data produced by the model. The name structure followed is "month-day-year hour.feather" for all of the hours during 2022. For example, the filename for the air pollution concentrations at 8 AM on 4th January 2022 would be "01-04-2022 08000.feather".

Output Directory

The Output/ directory contains the predicted hourly air pollution concentration datasets. Each file corresponds to a specific hour of a given date and includes modelled values for a range of pollutants and prediction quantiles.

File Naming Convention

Files are named using the following format:

DD-MM-YYYY HHMMSS.feather

• DD-MM-YYYY: The date of the prediction (e.g., 01-01-2022)

• HHMMSS: The time of day (in 24-hour format), representing the hour for which the predictions apply

• Example: 000000 = 00:00, 010000 = 01:00, etc.

Example Files

File Contents

Each .feather file contains structured predictions across the global 0.25° resolution grid for the given hour. Variables typically include:

• Pollutant Predictions:

• NO₂, O₃, PM₁₀, PM₂.₅, SO₂

• For each pollutant:

- Mean estimate
- 5th percentile estimate
- 50th percentile (median) estimate
- 95th percentile estimate

• Geographic Identifiers:

• Latitude

• Longitude

• Global Model Grid ID (Unique cell identifier)

These files allow users to conduct hourly air pollution analyses at global scale, including spatially explicit exposure estimates, temporal trends, and quantile-based risk assessments.

Models_for_A_Data-Driven_Supervised_Machine_Learning_Approach_to_Estimating_Global_Ambient_Air_Pollution_Concentrations_With_Associated_Prediction_Intervals.zip:

global_models: Models produced during the study were grouped into the experiment type (spatial, spatial_kfold, temporal), with the subdirectory name relating to the particular data that was excluded for each model.
global_auxiliary_models_quantile_regression: The supporting models were created to provide quantile regression estimates for all predictions.

Models Directory

The Models/ directory contains the trained machine learning models used to estimate air pollution concentrations globally. Models are organised by quantile (e.g., 0.05, 0.5, 0.95) or mean, pollutant type, spatial/temporal focus, and training data scope.

Each model is saved in two parts: a .pkl file for the model itself and a .csv with the results associated with that model. Within the Model directory, .csv files (LOOV_station) are also provided, documenting the monitoring stations that were excluded from the complete dataset for the experiment described in that directory. These stations constitute the Leave-One-Out Validation set.

Example Filepath

global_auxiliary_models_quantile_regression/0.5/temporal/models_global/no2/2014_2022_temporal/

This directory contains the median (0.5 quantile) prediction model for Nitrogen Dioxide (NO₂), trained on data spanning 2014–2022, using global station data for temporal generalisation.

Feature_and_Target_Vector_Input_Data_for_A_Data-Driven_Supervised_Machine_Learning_Approach_to_Estimating_Global_Ambient_Air_Pollution_Concentrations_With_Associated_Prediction_Intervals.zip: The directory contains the .feather files for the training data for all of the different air pollutants covered in the study.

Core Dataset Files

The following .feather files contain the processed datasets used for model training and evaluation. Each file corresponds to a specific air pollutant and contains the final structured dataset with feature vectors and target values at monitoring station locations.

Each file is saved in Apache Feather format, enabling efficient loading and processing within Python using Pandas or PyArrow. These datasets are typically used to train and evaluate machine learning models for air pollution prediction.

Note on File and Folder Names:
Some of the files and folders in this dataset contain long names that may exceed path length limits on certain operating systems, particularly Windows. If you encounter issues accessing or extracting these files due to path length constraints, you may rename the files and folders locally. Renaming does not affect the reproducibility or analysis of the data.

Overview

Addressing the global challenge of ambient air pollution requires scalable and high-resolution data. However, many regions lack a comprehensive monitoring infrastructure. This research introduces a machine learning framework that extends air quality estimation globally, using remote sensing, meteorological reanalysis, and emissions datasets to produce hourly pollutant concentrations at a 0.25-degree spatial resolution across the globe.

This work builds on the LightGBM-based framework introduced in A framework for scalable ambient air pollution concentration estimation, adapting it for a global context. LightGBM, a fast and accurate gradient-boosted decision tree algorithm, enables this large-scale application with high predictive performance and interpretability.

Dataset Purpose

This dataset empowers diverse stakeholders, including researchers, policymakers, urban planners, and public health authorities, providing a robust basis for conducting air quality assessments and interventions at unprecedented resolution. The improved granularity facilitates more precise studies into air pollution impacts on human health, urban resilience, and environmental justice, surpassing the capabilities of conventional lower-resolution approaches. The framework’s computational efficiency and scalability further demonstrate its potential applicability for similar pollution estimation challenges globally, especially in regions with limited observational infrastructure.

The dataset is readily accessible and stored online, ensuring rapid retrieval and ease of use for various analytical and operational purposes. Users can confidently perform high-resolution analyses supported by validated machine-learning-driven estimates, thereby enhancing informed decision-making and targeted interventions aimed at reducing air pollution exposure and promoting sustainable urban environments.

For questions regarding this dataset, please reach out to Liam J Berrisford.

Models Datasets

All of the models used in the research are included within this dataset. The models included have been trained to predict the mean air pollution concentration and the 5th / 50th (median) / 95th percentile of the air pollution concentration. The model is saved as the LightGBM booster within a .txt file, and the model parameters are kept within a .json file. The code that is used to recreate a LightGBM object within Python that can be used to make predictions is available via the Environment Insights Python package. The directory structure provides details about whether all of the monitoring stations were used for the models, or whether all of the feature vectors were used for the respective model.

Air Pollution (Target Vector) Dataset Overview

The dataset includes hourly estimates for key ambient air pollutants: Nitrogen Dioxide (NO2), Ozone (O3), Particulate Matter with a diameter of 10 μm or less (PM10), Particulate Matter with a diameter of 2.5 μm or less (PM2.5), and Sulphur Dioxide (SO2). Rigorous validation was performed to assess the accuracy of model predictions for forecasting air pollution concentrations, estimating values at previously unmeasured locations, and capturing extreme pollution episodes.

Estimates were produced at the mean, as well as at the 0.05, 0.5, and 0.95 quantiles, to capture both the typical and extreme variations in air pollution concentrations. The mean provides a central tendency estimate, useful for general assessments and long-term policy planning. Quantile-based predictions, specifically at the 0.05, 0.5, and 0.95 quantiles, offer deeper insights into the variability and uncertainty inherent in pollution concentration estimates. These quantile predictions enable stakeholders to understand the range and extremes of pollutant levels, supporting risk assessments and targeted interventions, such as public health advisories or emergency response planning during pollution peaks.

Feature Vector Dataset Overview

The dataset developed in this study includes a comprehensive set of feature vectors used to estimate ambient air pollution concentrations across the globe. Feature vectors represent environmental conditions and phenomena known to influence air pollutant concentrations, including meteorological variables (e.g., wind speed and temperature), emissions from various human activities (e.g., traffic intensity, industrial processes) ,and remotely sensed air pollution measurements. Each feature was selected based on established scientific evidence linking these variables to the formation, dispersion, and accumulation of air pollutants. Incorporating such a diverse and detailed set of features enables the machine learning model to robustly capture complex spatial and temporal variations in air quality, ultimately improving the accuracy and applicability of pollution estimates.

Data Description

Coordinate Variables

Each NetCDF file is indexed by (Timestamp, Latitude, Longitude).

Data Variables

Output Variables

Input Variables

Training Data Variables

Alongside the additional variables included in NetCDF files across this subdirectory, each NetCDF has additional attributes detailing the given station's official Site Name, Site Code, and Site Type. For convenience, the Global Model Grid ID is also provided as an attribute.

Code

During the course of the study, 21 .ipynb were used, which are also included in this repository:

• global_air_pollution_concentrations_quantile_regression.ipynb: This notebook includes code for applying quantile regression to estimate global air pollution concentrations.
• global_air_pollution_data_anlaysis.ipynb: This notebook is used for analyzing global air pollution data.
• global_air_pollution_maps_predictions.ipynb: This notebook generates prediction maps of global air pollution concentrations.
• global_all_maps_visulisation.ipynb: This notebook provides visualization scripts for displaying various global air pollution maps.
• global_data_download_airPollution-openAQ.ipynb: This notebook downloads air pollution data from the OpenAQ platform.
• global_data_GEE_monitoringStationFeatureVector.ipynb: This notebook creates feature vectors for monitoring stations using Google Earth Engine data.
• global_data_monitoringStationSiteFeatureVector.ipynb: This notebook constructs site-specific feature vectors for air pollution monitoring stations.
• global_data_preprocessing_trainingData_featureVector.ipynb: This notebook preprocesses and prepares the training data feature vectors for the machine learning model.
• global_data_processing_grids_enviromentalFeatureVector.ipynb: This notebook processes environmental feature vectors for grid-based data analysis.
• global_model_uncertainity_monitoring_station_placement.ipynb: This notebook analyzes model uncertainty to optimize the placement of air pollution monitoring stations.
• global_data_processing_grids_remoteSensingFeatureVector.ipynb: This notebook processes remote sensing data to create feature vectors for grid-based environmental analysis.
• global_model_absoluteLegislationCompliance.ipynb: This notebook evaluates air pollution data against absolute legislation compliance standards and the overall AQI index sum maps used within the manuscript.
• global_model_figures.ipynb: This notebook generates figures and visualizations for the global air pollution model results.
• global_model_individual_station_predicitions_spatial.ipynb: This notebook analyzes the spatial predictions for individual air pollution monitoring stations.
• global_model_individual_station_predicitions_temporal.ipynb: This notebook analyzes the temporal predictions for individual air pollution monitoring stations.
• global_model_individual_stations_correlation_bias.ipynb: This notebook examines the correlation and bias in predictions for individual monitoring stations.
• global_model_output_air_pollution_map_analysis.ipynb: This notebook analyzes the output maps of air pollution concentrations generated by the global model.
• global_model_training_emissions_area.ipynb: This notebook trains the model using emissions data for specific areas.
• global_model_training_emissions_area_quantile_regression.ipynb: This notebook applies quantile regression to train the model using emissions data for specific areas.
• globalEmissions.ipynb: This notebook processes and analyzes global emissions data for use in the air pollution model.