Supplementary data for: Nowcasting 3D cloud fields using forward warping optical flow
Data files
Sep 19, 2025 version files 1.01 GB
-
clavrx_files.zip
1.01 GB
-
README.md
7.29 KB
Abstract
Large- to global-domain short-term prediction of clouds (0-3 hours), or cloud nowcasting, remains relevant to civilian and military applications ranging from solar energy production to intelligence gathering. Despite the capabilities of contemporary numerical weather prediction models, nowcasting methods based on near real-time observations (i.e. satellite imagery) hold operational value due to their relative computational efficiency and accuracy for short-term applications. A commonly used nowcasting approach involves using two or more images to retrieve the apparent motions of features, or optical flow, which can be used to extrapolate the future location of those features. However, such approaches generally assume that the optical flow field remains unchanged with respect to time which is challenging to apply to piecewise cloud fields from satellite imagery. Here, we propose a method to nowcast clouds that adapts a computer vision technique for image interpolation, commonly referred to as warping, to account for temporal changes to optical flow fields derived from infrared satellite imagery. We evaluate the proposed method for 991 randomly selected regional cases from 2024 and perform a detailed analysis on three specific cases. Applying a dense (every image pixel) optical flow retrieval technique to full-disk GOES infrared imagery, we demonstrate that forward warping of the optical flow field when coupled with simple occlusion reasoning, improves skill in cloud nowcasting.
DOI: https://doi.org/10.5061/dryad.jm63xsjmd
Dryad data contains only the observed CLAVR-x (Clouds from AVHRR Extended) cloud property retrievals, namely cloud top height (CTH) and cloud base height (CBH) derived from GOES-16 ABI (Advanced Baseline Imager) imagery. These data are utilized in the example script (contained in Zenodo:https://doi.org/10.5281/zenodo.16740427) that produces full-disk nowcasts (short-term forecasts 0 - 3 hours) using the I-NOW (Improved Nowcasting via Optical flow Warping) technique presented in "Nowcasting 3D Cloud Fields Using Forward Warping Optical Flow" by Matthew King, Jason Apke, Steven Miller, Katherine Haynes, Yoo-Jeong Noh, and John Haynes, 2025: J. Atmos Ocea. Tech.
Description of the data and file structure
There are 19 .hdf files that all have the same naming convention. The files start with "clavrx_goes16_" which references that they are CLAVR-x retrieval files for the GOES-16 Geostationary Satellite. The next set of of numbers is the 4 digit year (e.g. 2023). The next set of numbers is the 3 digit Julian day of the year (e.g. 016). The next set of numbers represents the time of the satellite imagery scan (e.g. 160020), where the first two digits represents the hour, the next two digits represents the minute, and the last two digits represent milliseconds. The file then ends with ".level2.hdf" which indicates that is level 2 data and are in .hdf file format.
The following files are contained in clavrx_files.zip:
clavrx_goes16_2023_016_160020.level2.hdf
clavrx_goes16_2023_016_161020.level2.hdf
clavrx_goes16_2023_016_162020.level2.hdf
clavrx_goes16_2023_016_163020.level2.hdf
clavrx_goes16_2023_016_164020.level2.hdf
clavrx_goes16_2023_016_165020.level2.hdf
clavrx_goes16_2023_016_170020.level2.hdf
clavrx_goes16_2023_016_171020.level2.hdf
clavrx_goes16_2023_016_172020.level2.hdf
clavrx_goes16_2023_016_173020.level2.hdf
clavrx_goes16_2023_016_174020.level2.hdf
clavrx_goes16_2023_016_175020.level2.hdf
clavrx_goes16_2023_016_180020.level2.hdf
clavrx_goes16_2023_016_181020.level2.hdf
clavrx_goes16_2023_016_182020.level2.hdf
clavrx_goes16_2023_016_183020.level2.hdf
clavrx_goes16_2023_016_184020.level2.hdf
clavrx_goes16_2023_016_185020.level2.hdf
clavrx_goes16_2023_016_190020.level2.hdf
To access the contents of a file, xarray may be used in Python to open the file in the following manner:
import xarray as xr
file_dir = 'clavrx_goes16_2023_016_160020.level2.hdf'
Clavrx_xarray = xr.open_dataset(file_dir, engine='netcdf4)
CBH_values = CLavrx_xarray['cld_height_base'].data
CTH_values = Clavrx_xarray['cld_height_acha'].data
Sharing/Access information
The CLAVR-x software, which was utilized to create the retrieval data contained in the Dryad repository, is available for download from: https://cimss.ssec.wisc.edu/cspp/clavrx_v3.0.shtml
GOES imagery files, which are required for CLAVR-x to retrieve cloud properties, are available from NOAA's Registry of Open Data on AWS: https://registry.opendata.aws/noaa-goes/
Code/Software
The script (and relvant python files) which produces the I-NOW nowcasts and utilizes the CLAVR-x data are available both on Zenodo (https://doi.org/10.5281/zenodo.16740427) and GitHub (https://github.com/AFwxking/I-NOW). CLAVR-x files contained in Dryad are also available on Zenodo as well as GitHub.
The following is information for the aforementioned script:
Requires:
- Python 3.12.2 (version used for development, other versions may work)
- Data files which include CLAVRx_data and GOES Full Disk Band 13 data (code to download GOES data from AWS is included)
- Python scripts contained in OCTOPY folder (contains scripts to run python version of OCTANE optical flow code and the I-NOW nowcast method which requires temporal interpolation scripts)
- Code requires GPU with support for CUDA (developed on NVIDIA RTX A6000)
- Data files within the DOI in data availability statement of manuscript (CLAVRx CTH/CBH data saved in clavrx_data folder on GitHub Repo)
Dependencies: netcdf4, numba, cudatookit, s3fs, matplotlib, cartopy, xarray, metpy
- netcdf4, numba, and cudatoolkit required to run nowcasts
- s3fs required to download GOES data from AWS
- xarray, matplotlib, cartopy, and metpy required in code to plot results
Recommend using a virtual environment to run this code, such as conda
environment.yml file provided in directory for easy installation of all dependencies
$ conda env create -f environment.yml
Should the environment.yml file fail to install workable environment, environment can be installed using the commands below in the
order they are provided.
create an anaconda environment:
$ conda create -n INOW_env python=3.12.2
$ conda activate INOW_env
Install Netcdf4 first, this one does not like to play nice with the others:
$ conda install -c conda-forge netcdf4
Install numba
$ conda install -c numba numba
Install the cudatoolkit, use the anaconda distribution, the conda-forge one has a driver issue
$ conda install -c anaconda cudatoolkit
Now matplotlib, use the pip-install for this
$ python -m pip install -U matplotlib
Now cartopy
$ conda install -c conda-forge cartopy
Now xarray
$ conda install -c conda-forge xarray
Now metpy
$ conda install -c conda-forge metpy
Now s3fs
$ conda install -c conda-forge s3fs
Example run commnand (unix):
$ python Nowcast_Example.py
Code Author: Matthew P. King, Lt Col, USAF
Date: 27 June 2025
Version: 1.0
Explanation of code:
Nowcast_Example.py provides an example of how to run the I-NOW nowcasting method on GOES-16 Band 13 data. Code downloads GOES-16 Band 13 data from AWS, runs a python version of OCTANE optical flow code to calculate the optical flow using the values contained in the JTEC article. Rather than linear extrapolation, the code uses a warping method formed from computer vision temporal interpolation methods to account for occlusions and time related changes to the optical flow field in the forward time direction rather than a time that is intermediate of two images. The code provides nowcasts of GOES-16 10.3 micron brightness temperature,CLAVR-x derived cloud top heights (CTH), and CLAVR-x derived cloud base heights (CBH). Observed brightness temperature data and CLAVR-x data are plotted alongside the nowcast results in 10-minute intervals. The code also creates animated gifs of the nowcaststo easier visualize the results through time. The code is designed to be run on a GPU with CUDA support, and the user can define the save directory for the results to be saved and data that is downloaded. CLAVR-x data is not downloadable from the internet,so the GitHub repository contains a folder with CLAVR-x data for the example date and time used in the code that has been reduced to contain only the required CTH and CBH data instead of full CLAVR-x retrieval data.
Explanation of code: Nowcast_Example.py provides an example of how to run the I-NOW nowcasting method on GOES-16 Band 13 data. Code downloads GOES-16 Band 13 data from AWS, runs a python version of OCTANE optical flow code to calculate the optical flow using the values contained in the JTEC article. Rather than linear extrapolation, the code uses a warping method formed from computer vision temporal interpolation methods to account for occlusions and time related changes to the optical flow field in the forward time direction rather than a time that is intermediate of two images. The code provides nowcasts of GOES-16 10.3 micron brightness temperature,CLAVR-x derived cloud top heights (CTH), and CLAVR-x derived cloud base heights (CBH). Observed brightness temperature data and CLAVR-x data are plotted alongside the nowcast results in 10-minute intervals. The code also creates animated gifs of the nowcasts to easier visualize the results through time. The code is designed to be run on a GPU with CUDA support, and the user can define the save directory for the results to be saved and data that is downloaded. CLAVR-x data is not downloadable from the internet,so the repository contains a folder with CLAVR-x data for the example date and time used in the code. The CLAVR-x files have been reduced to contain only the required CTH and CBH data instead of full CLAVR-x retrieval data to save on storage space in the repository.
