An object-based analysis of lightning characteristics in pre-tropical cyclogenesis environments
Data files
Jun 30, 2026 version files 716.02 MB
-
GLM-Genesis-Project-Code.zip
716.01 MB
-
README.md
14.81 KB
Abstract
The Geostationary Lightning Mapper (GLM) provides continuous, high-resolution lightning observations that enable a novel investigation of lightning attributes in pre-tropical cyclogenesis environments. Compositing and tracking methods from the Tracking and Object-Based Analysis of Clouds (tobac) Python package enable the consideration of lightning lifetime alongside GLM-measured area and optical energy. We first apply the object-based framework to lightning observed prior to the formation of Tropical Storm Claudette (2021), a genesis case within range of the NEXRAD network. Colocated ground-based, dual-polarization radar observations suggest that small-area and low-energy lightning is indicative of stronger convection and updrafts, based on composite vertical radar profiles of tobac lightning features. We then apply the physical interpretations of these lightning attributes to lightning 72 hours prior to genesis and within 200 km of the best-track invest center for five North Atlantic disturbances. Developing disturbances include Claudette (2021), Ida (2021), Earl (2022), and Beryl (2024), which we compare against non-developing AL96 (2024). Small-area and low-energy lightning, previously associated with stronger updrafts and convection, is the dominant lightning mode for the four developing cases with large-area and high-energy lightning also identified at various times. Lightning activity appears to be modulated by deep-layer vertical wind shear, and electrified convection coincides with improvements in convective organization in multiple instances. While more work is needed to evaluate lightning attributes across a larger composite of disturbances, this study offers a novel characterization of lightning in pre-tropical cyclogenesis environments for the North Atlantic to aid in the understanding and forecasting of tropical cyclone genesis.
https://doi.org/10.5061/dryad.x95x69pzq
Description of the data and file structure
The code associated with this dataset bins GLM groups at 0.05-degree and 5-minute resolution. The tobac tracking algorithm is then applied to this gridded field to obtain lightning features and their associated attributes for lightning with 200 km of a storm center and 3 days prior to tropical storm formation. We use this approach to investigate lightning in five tropical disturbances in the North Atlantic. For one of these storms, we also consider coincident ground-based, dual-polarization radar data to assess how lightning attributes are related to convective strength and microphysical composition.
Code/software
The Tracking and Object-Based Analysis of Clouds Python package can be found at https://github.com/tobac-project/tobac. This code uses Version 1.6.
Workflow
Initial Set-up
- Set file paths for the following files in Misc_files. File path locations in parentheses.
- G16_east_grid.bin (ABI_grid.py Line 13)
- G16_east_SATZEN.bin (ABI_grid.py Line 49)
- dist2coast_1deg.nc (tobac_merge_pregenesis.py Line 114)
Creating Your Daily NetCDF Output File
- Use namelist_latlon_bounds.ipynb in the Notebooks folder to determine the lat/lon bounds of the daily NetCDF output file.
- Ensure degree radius buffer and degree resolution are acceptable
- Input storm id # for desired storm and edit path to best-track file in the third cell if necessary
- Ensure the red box captures the storm track and 200 km radii under section "Verify bounds capture 200 km radius from storm center"
- Edit path to Namelist file in section "Write the bounds and resolution to the Namelist file" if necessary
- ENSURE THE NAMELIST FILE HAS BEEN CHANGED TO REFLECT DESIRED CHANGES
- Getting your daily netCDF output file
- Adjust directory paths in nc_ouput.py
- Line 36: Where you want netCDF files saved
- Adjust directory paths in mesa_functions.py
- Line 161 tc_info()
- Line 315 interp_tc_info()
- Input parameters in nc_output_multiprocess.py
- Storm ID (e.g., al022024)
- Storm name
- Year
- Range of days of year of interest (NOT END INCLUSIVE)
- Time interval
- Space interval
- To submit this as a job to a job scheduler, run nc_output.sh in Shell_scripts
- Specify name of job and email for notifications
- IMPORTANT: Match # of processors in Line 12 with number of days in DOYs array in nc_output_multiprocess.py Line 9 to maximize efficiency
- Lines 15 + 16: Specify location and name of output and error files
- Line 21: Specify where .py file is located
- Line 24: Specify environment you want to use
- Adjust directory paths in nc_ouput.py
Running tobac
- Running tobac for each day of interest
- Change filepaths in tobac_single_day.py
- Line 31: Location of daily NetCDF output files
- Line 52: Location to save tobac single day outputs
- Input parameters in tobac_single_day_multiprocess.py
- Storm name
- Year
- Range of days of year of interest (NOT END INCLUSIVE)
- Time interval
- Space interval
- To submit this as a job to a job scheduler, run tobac_single_day.sh in Shell_scripts
- Specify name of job and email for notifications
- IMPORTANT: Match # of processors in Line 12 with number of days in DOYs array in tobac_single_day_multiprocess.py Line 8 to maximize efficiency
- Lines 15 + 16: Specify location and name of output and error files
- Line 21: Specify where .py file is located
- Line 24: Specify environment you want to use
- Change filepaths in tobac_single_day.py
- Merging tobac outputs from multiple days
- Change filepaths in tobac_merge_pregenesis.py
- Line 40: Location to save tobac merged pre-genesis outputs
- Line 52: Location of daily NetCDF output files
- Line 53: Location of daily mask segmentation files
- Line 71: Location of daily feature files
- Line 114: Location of land mask file
- Input parameters for tobac_merge_pregenesis.py on Lines 130-140
- Storm name
- Storm ID
- Year
- Start day of year
- End day of year
- Space resolution
- Time resolution
- Estimated grid spacing in meters for tobac tracking
- Run 'main' function in file on Line 165
- To submit this as a job to a job scheduler, run tobac_merge_pregenesis.sh in Shell_scripts
- Specify name of job and email for notifications
- Lines 15 + 16: Specify location and name of output and error files
- Line 21: Specify where .py file is located
- Line 24: Specify environment you want to use
- Change filepaths in tobac_merge_pregenesis.py
Subsetting Lightning Features
Use tracks_200km_72hr.py to subset lightning tracks of relevance to this work.
- Line 10: Update location where best-track files reside
- Line 37: Update location where daily netCDF files reside
- Line 42: Update location where pregenesis tobac lightning tracks reside
- Line 49: Update location for saving the 200 km and 72 hr subset of pregenesis tobac lightning tracks
- To submit this as a job to a job scheduler, run tracks_200km_72hr.sh in Shell_scripts
- Specify name of job and email for notifications
- Lines 15 + 16: Specify location and name of output and error files
- Line 21: Specify where .py file is located
- Line 24: Specify environment you want to use
Claudette Radar Analysis
The notebook claudette_radar_analysis.ipynb uses colocated lightning and dual-polarization radar data from a case of genesis near land. This code serves to assess how lightning attributes vary with convective strength and microphysical parameters in a pregenesis environment.
- In the second cell under "Call in Files", change the file paths for the ds and lng_features_w_stats_noland variables to reflect where the daily netCDF and pregenesis tobac lightning track files are located.
- In the second cell under "Distance of Features from Radar and Storm Center", change the file path for lightning_radar_features.to_hdf(). This file will later be used in the composite analysis and method plots notebooks, as discussed below.
- In the cell under "tobac Segmentation Masks on Binned Group Centroids", set the file path of lightning_mask to reflect where the tobac segmentation masks for Claudette.
Composite Analysis
The notebook composite_analysis.ipynb considers lightning features within 200 km and 72 hours prior to genesis for the four developing disturbances considered. This notebook is used to obtain percentiles of lightning cell lifetimes, assess sensitivity to grid resolution, and compare distributions.
- In the third cell under "Call in Track Dataframes for 4 Storm Composite", change the best_track variable to reflect the directory where the best-track files have been stored. In the same cell, also change the file paths for the ds and pregenesis tracks variables to reflect where the daily netCDF and pregenesis tobac lightning tracks are located.
- In the second cell under "Radar Subset versus 4 Storm Composite", change the radar_features variable to reflect the directory with the location of the Claudette radar subset.
- In the first cell under "Non-Developer Comparison", change the file path for nondev_tracks to reflect where the pregenesis tobac lightning tracks are located.
- In the first cell under "tobac Resolution", change the track variable to reflect the directory with the pregenesis tobac lightning tracks.
Analysis Example for Beryl (2024)
- Run the NetCDF output files and tobac code for June 26-29, 2024 (DOYs: 178, 179, 180, 181)
- A sample best track file for Beryl is located in Misc_files.
- Open tobac_merged_pregenesis_analysis.ipynb notebook
- Edit filepaths
- Under "Case Info", Line 5: Location of best-track file created by Alan Brammer. You can find a sample of this file in Misc_files
- In third cell under "Case Info", Line 9: Location of daily NetCDF output files
- In fourth cell under "Case Info", Line 2: Location of merged pre-genesis tobac track file
- Comment out vorticity steps, or download corresponding files from ERA5 Climate Data Store.
Methods Plots
The notebook methods_plots.ipynb provides various visualizations that were used in the manuscript to depict our methodology.
- In the first cell under "GLM-TOBAC METHODOLOGY", change the file paths to reflect the locations and masks of the tobac lightning features on 18 June, 2021.
- Under the "Radar" section, include the path to the Claudette best-track file (first cell) and Claudette radar features subset that was saved in claudette_radar_analysis.ipynb (second cell).
- In the second cell in the "Tracks" section, edit the best_track file path to reflect where your best-track files are located.
Files and variables: GLM-Genesis-Project-Code.zip
Files: /Misc_files/BEST_bal...nc
Description: Best-track files for Claudette (032021), Ida (092021), Earl (062022), Beryl (022024), and AL96.
Relevant Variables:
- track_lat (degrees north) - Storm latitude
- track_lon (degrees east) - Storm longitude
- track_type (N/A) - Storm type/strength (e.g., tropical storm (TS), hurricane (HU), etc.)
- dshr_mag (m s-1)- area: mean wind shear magnitude within radius range 0 to 300 km, vertical: difference between 850 and 200 hPa
- dshr_hdg (degrees) - area: mean wind shear heading within radius range 0 to 300 km, vertical: difference between 850 and 200 hPa
File: /Misc_files/dist2coast_1deg.nc
Description: Distance to land file at 0.01-degree resolution.
Relevant Variables:
- dist (km) - distance to nearest coastline
File: /Misc_files/G16_east_grid.bin
Description: Provides the Advanced Baseline Imager grid by outputting the gridded latitude and longitude of each grid cell. Can be opened using Python, as shown in ABI_grid.py Line 15.
Relevant Variables:
- lat (degrees north) - latitude of ABI grid cells
- lon (degrees east) - latitude of ABI grid cells
File: /Misc_files/G16_east_SATZEN.bin
Description: Provides the Advanced Baseline Imager satellite zenith angle. Outputs the gridded zenith angle and azimuth angle. Can be opened using Python, as shown in ABI_grid.py Line 51.
Relevant Variables:
- satzen (degrees) - satellite zenith angle
- satazm (degrees) - satellite azimuth angle
File: /Misc_files/Namelist
Description: Output file of namelist_latlon_bounds.ipynb. Determines boundaries and resolution of latitude and longitude to reduce size of file created form nc_output.py.
Relevant Variables:
- LAT_MIN (degrees north) - Minimum latitude boundary
- LAT_MAX (degrees north) - Maximum latitude boundary
- LAT_RES (degrees) - Latitude resolution
- LON_MIN (degrees east) - Minimum longitude boundary
- LON_MAX (degrees east) - Maximum longitude boundary
- LON_RES (degrees) - Longitude resolution
Files: /Notebooks/claudette_radar_analysis.ipynb
Description: Jupyter notebook that investigates colocated GLM and ground-based, dual-polarization radar observations.
Files: /Notebooks/composite_analysis.ipynb
Description: Jupyter notebook that uses lightning data from all four cases considered in this work to obtain various composite statistics and plots.
Files: /Notebooks/methods_plots.ipynb
Description: Jupyter notebook that illustrates project data and methodology.
Files: /Notebooks/namelist_latlon_bounds.ipynb
Description: Jupyter notebook that rewrites the latitude and longitude boundaries for a storm to the Namelist file.
Files: /Notebooks/tobac_merged_pregenesis_analysis.ipynb
Description: Jupyter notebook that investigates pre-genesis trends for an individual storm.
Files: /Scripts/ABI_grid.py
Description: Functions for obtaining ABI grid.
Files: /Scripts/Functions.py
Description: Functions with multi-use purposes within the scope of the project.
Files: /Scripts/Get_ABI.py
Description: Functions for reading, windowing, and applying quality control to ABI data.
Files: /Scripts/Get_GLM.py
Description: Functions for reading, windowing, and applying quality control to GLM data.
Files: /Scripts/mesa_functions.py
Description: Functions with multi-use purposes within the scope of the project.
Files: /Scripts/nc_output_multiprocess.py
Description: Python script that runs nc_output.py for multiple days associated with a single storm using parallel processing.
Files: /Scripts/nc_output.py
Description: Python script that creates a daily netCDF file for a storm that included ABI, GLM, and best-track information.
Files: /Scripts/TC_Functions.py
Description: Functions with multi-use purposes within the scope of the project.
Files: /Scripts/tobac_merge_pregenesis.py
Description: Python script that merges tobac output files (i.e., features, masks, tracks) across multiple days into a single file for each output type.
Files: /Scripts/tobac_single_day_multiprocess.py
Description: Python script that runs tobac_single_day.py for multiple days associated with a single storm using parallel processing.
Files: /Scripts/tobac_single_day.py
Description: Python script that runs tobac tracking algorithm on lightning data from daily netCDF output files to output lightning features, masks, and tracks.
Files: /Scripts/tracks_200km_72hr.py
Description: Python script that subsets tracks within 200 km of storm center and 72 hours before genesis from tobac_merge_pregenesis.py track output.
Files: /Shell_scripts/
Description: Shell scripts used to run .py scripts in Sun Grid Engine (SGE) or SLURM
Access information
Other publicly accessible locations of the data:
- Data from the Advanced Baseline Imager and Geostationary Lightning Mapper were accessed from the Amazon Web Services S3 Explorer server at https://registry.opendata.aws/noaa-goes.
- NEXRAD data was accessed from the Amazon Web Services S3 Explorer server at https://registry.opendata.aws/noaa-nexrad/.
- ERA5 vorticity downloaded from the ECMWF Climate Data Store at https://cds.climate.copernicus.eu/.
- KLIX sounding was obtained from https://weather.uwyo.edu/upperair/sounding.shtml.
Data was derived from the following sources:
- ATCF best-track data can found at https://ftp.nhc.noaa.gov/atcf/archive/.
This dataset includes the code required to process publicly available data and reproduce results presented in the manuscript. The dataset also includes track files for the four cases considered in this work.
Cases and Tracks
To ensure a full 3-days of evolution prior to genesis was analyzed, we advect the first record from the Automated Tropical Cyclone Forecasting System (ATCF) b-decks backwards in time to 72h before the genesis time following the methodology of the Trajectory and Beta model (DeMaria et al. 2022), using environmental winds between 850-400hPa. For the cases presented here, this was used to generate the locations for the first 12 and 30 hours of analysis for Ida and AL96, respectively. We obtain additional storm-relative variables, such as deep-layer vertical wind shear and vorticity, from the European Centre for Medium-Range Weather Forecasts (ECMWF) 5th Generation reanalysis (ERA5) fields for all times in the merged track (Hersbach et al. 2020). The merged track and its corresponding storm-relative variables are then linearly interpolated to five-minute intervals to match the timestep used to process GLM data. These track files are listed in the database under Misc_files and are titled starting with "BEST_bal...nc".
Advanced Baseline Imager
We access ABI data remotely through the "noaa-goes-16" bucket in the AWS S3 Explorer. For this work, we utilize the 10.3𝜇m ``Clean Infrared" band (Channel 13).
Geostationary Lightning Mapper
We access GLM data remotely through the "noaa-goes-16" bucket in the AWS S3 Explorer and utilize GLM groups for this work. GLM group centroids are binned at a 0.05-degree and 5 minute resolution and exclude any groups that have been flagged for data quality purposes.
Tracking and Object-Based Analysis of Clouds
We use tobac v1.6 to create a database of lightning features and obtain additional spatiotemporal attributes (Heikenfeld et al. 2019; Sokolowsky et al. 2024). The tobac tracking algorithm is applied to the gridded GLM group centroid field. For tobac segmentation, we introduce a "ghost pixel" and enable diagonal connectivity in the segmentation.py tobac script to ensure all pixels associated with a lightning feature are considered. Our application of tobac uses a maximum search radius of 15 km and the "random" linking method. A land mask is also used to exclude lightning cells within 25 km of land.
Ground-based Radar
We access NEXRAD data through the "unidata-nexrad-level2" bucket in the AWS S3 Explorer. We use the LROSE-Colette software (DeHart et al. 2025) to grid and QC volumetric scans from the three radars, while calculating specific differential phase (KDP) and particle identification (PID) fields using a KLIX sounding from 0000 UTC 19 June. PID is output from the LROSE RadxPID software, which uses a fuzzy logic algorithm outlined in Vivekanandan et al. (1999) to estimate the National Center for Atmospheric Research (NCAR) PID. QC involves excluding gates with correlation coefficient values greater than 1. We also apply a differential reflectivity correction following the method detailed in Cunningham et al. (2013) prior to calculating KDP and PID. LROSE then grids the dual-polarization radar data on a 1 km horizontal grid with 0.5 km vertical spacing.