Data and code from: How to use score-based diffusion in earth system science: A satellite nowcasting example
Data files
Jul 02, 2026 version files 38.78 GB
-
all_models_Chase2025.tar
5.76 GB
-
README.md
5.15 KB
-
simple_code.md
4.04 KB
-
test_set_Chase2025.tar
2.47 GB
-
train_set_Chase2025.tar
28.07 GB
-
validation_set_Chase2025.tar
2.48 GB
Abstract
Here are the GOES satellite data used to train, validate, and test the score-based diffusion model in the paper titled: "How to use score-based diffusion in earth system science: A satellite nowcasting example". The data are Zarr files, designed to be lazily loaded into memory for efficient training of the machine learning models. For convenience, the Zarr files are tarred up and produced here, named: Train: train_set_Chase2025.tar (25 GB), Validation: validation_set_Chase2025.tar (2.4 GB), and Test: test_set_Chase2025.tar (2.4 GB). Specifically, these are randomly generated 256 pixel by 256 pixel patches from GOES-16 that are channel 13 (IR brightness temperatures). The satellite data could be used to forecast clouds and precipitation.
Dataset DOI: 10.5061/dryad.xpnvx0kwc
Description of the data and file structure
Reproduced here from the paper:
Data:
The primary dataset of this paper is Channel 13 (10.3 𝜇𝑚 wavelength) from the Advanced Baseline Imager (ABI; Schmit et al. 2017) on board the GOES-16 satellite, which was part of the Geostationary Operational Environmental Satellite (GOES) satellite series. Channel 13 is a longwave infrared channel and is chosen because of its wide use in general meteorology, particularly for detecting clouds at all times, including both day and night. GOES-16 was operational from December 2017 until April 2025 at a central longitude of -75 W. Since the motivating goal of this paper is a global cloud forecast, we use full-disk (10 min resolution) data from April 2023 through June 2024. A full-disk image for Channel 13 has approximately two-kilometer resolution near nadir (i.e., over the Amazon) and has a raw image shape of 5,424 by 5,424 pixels. Unfortunately, for the Graphical Processing Units (GPUs) available for this project (a single NVIDIA GH-200 system, hereafter GH-200) and the fixed underlying neural network architecture, the 5,424 by 5,424 image size is prohibitively large and involves gradient computations that would not fit into memory. Thus, we take an approach where we create our datasets by randomly choosing smaller patches of the larger image.
Training Data:
The training set is selected from 2023 and includes several quality control steps. From all files in 2023, we select a random file with replacement. We check whether the previous and next time steps (minus and plus 10 mins) exist. If they exist, we open the files. From the 5,424 by 5,424 grid, we randomly sample 250 grid indices with a viewing zenith angle less than 65 degrees. This step avoids selecting imagery at the edge of the full disk where data are invalid. From these 250 grid incidences, we slice out a patch of 256 by 256 (about 500 km by 500 km, indicated as colored patches in Fig. 8). Out of these 250 patches, we only keep those that have at least 10% cloud fraction. From the cloudy patches, we then include only patches that occur during the day (85 deg solar zenith). Daytime-only data were chosen so auxiliary information like visible imagery and more accurate retrievals of cloud top height and base height are available for evaluation. From this final set of patches, we randomly choose 16 of them to save to avoid including too many samples from the same time step. We repeat this process until there are more than 30,000 patches for training.
Validation Dataset:
The same procedure is used to generate a validation set, but the files from January to August 2024 are used, and only 1,000 total patches are used for evaluation. The primary use of this validation set is to determine the hyperparameters for image generation.
Test Dataset:
The same procedure is used to generate a test set, but the files from August 2024 to February 2025 are used and only 1,000 total patches are used for evaluation.
Files and variables
The dataset files contain normalized 256 by 256 images from GOES. These images are channel 13 and were randomly selected. Once loaded, the dataloader provides 2 tensors: the input and output images. Example loader class can be found in the simple_code.md
File: validation_set_Chase2025.tar
Description: Archive of the validation set. Contains 2 Zarr files, edm_GOES_ch13_validation_dataset.zarr and edm_GOES_ch13_validation_dataset_latent.zarr.
File: test_set_Chase2025.tar
Description: Archive of the test set. Contains 2 Zarr files, edm_GOES_ch13_test_dataset.zarr and edm_GOES_ch13_test_dataset_latent.zarr.
File: train_set_Chase2025.tar
Description: Archive of the training set. Contains 3 files, edm_GOES_ch13_train_dataset.zarr, edm_GOES_ch13_train_dataset_CorrDiff.zarr and edm_GOES_ch13_train_dataset_latent.zarr
There are also all the trained models for the paper located in the following tar file
File: all_models_Chase2025.tar
Description: Archive of all the trained PyTorch models. unet_vanilla.tar, edm_plain_diffusion.tar, edm_ldm_radames.tar, edm_corrdiff.tar
Inside, there is a tar of each of the model checkpoints.
File: simple_code.md
Description: Some simple code examples of the classes used in the GitHub repo.
Code/software
An example of how to run the forecasts can be seen in the GitHub repo: https://github.com/dopplerchase/cira-diff/blob/main/scripts/Chase_2025/Run_Forecasts_Chase2025.ipynb, or the same named file in the included zip of the repo on the associated Zenodo release. You can also see a few simple class examples in simple_code.md
Access information
Data was derived from the following sources:
Reproduced here from the paper:
Data:
The primary dataset of this paper is Channel 13 (10.3 𝜇𝑚 wavelength) from the Advanced Baseline Imager (ABI; Schmit et al. 2017) on board the GOES-16 satellite, which was part of the Geostationary Operational Environmental Satellite (GOES) satellite series. Channel 13 is a longwave infrared channel and is chosen because of its wide use in general meteorology, particularly for detecting clouds at all times, including both day and night. GOES-16 was operational from December 2017 until April 2025 at a central longitude of -75 W. Since the motivating goal of this paper is a global cloud forecast, we use full-disk (10 min resolution) data from April 2023 through June 2024. A full-disk image for Channel 13 has approximately two-kilometer resolution near nadir (i.e., over the Amazon) and has a raw image shape of 5,424 by 5,424 pixels. Unfortunately, for the Graphical Processing Units (GPUs) available for this project (a single NVIDIA GH-200 system, hereafter GH-200) and the fixed underlying neural network architecture, the 5,424 by 5,424 image size is prohibitively large and involves gradient computations that would not fit into memory. Thus, we take an approach where we create our datasets by randomly choosing smaller patches of the larger image.
Training Data:
The training set is selected from 2023 and includes several quality control steps. From all files in 2023, we select a random file with replacement. We check whether the previous and next time steps (minus and plus 10 mins) exist. If they exist, we open the files. From the 5,424 by 5,424 grid, we randomly sample 250 grid indices with a viewing zenith angle less than 65 degrees. This step avoids selecting imagery at the edge of the full disk where data are invalid. From these 250 grid incidences, we slice out a patch of 256 by 256 (about 500 km by 500 km, indicated as colored patches in Fig. 8). Out of these 250 patches, we only keep those that have at least 10% cloud fraction. From the cloudy patches, we then include only patches that occur during the day (85 deg solar zenith). Daytime-only data were chosen so auxiliary information like visible imagery and more accurate retrievals of cloud top height and base height are available for evaluation. From this final set of patches, we randomly choose 16 of them to save to avoid including too many samples from the same time step. We repeat this process until there are more than 30,000 patches for training.
Validation Dataset:
The same procedure is used to generate a validation set, but the files from January to August 2024 are used, and only 1,000 total patches are used for evaluation. The primary use of this validation set is to determine the hyperparameters for image generation.
Test Dataset:
The same procedure is used to generate a test set, but the files from August 2024 to February 2025 are used, and only 1,000 total patches are used for evaluation.