Data from: Extreme heat adaptation planning: A review of evaluation, monitoring, and reporting
Data files
Jan 28, 2025 version files 54.06 KB
-
Heat_plan_eval_and_indicators_Holtan_2025.xlsx
50.42 KB
-
README.md
3.64 KB
Abstract
Extreme heat events are increasing in intensity and duration. Although heat adaptation planning is increasing across the US, the effectiveness of adaptation strategies across contexts remains unknown. Evaluation helps heat adaptation planners understand the impact of investments and increase accountability. To understand how evaluation is or is not happening in extreme heat planning, we purposively sampled and analyzed 65 plans that would likely include extreme heat adaptation strategies. We found that although 55% (n=36) of plans included heat evaluation or monitoring plans in some form, fewer than 30% (n=19) were associated with subsequent reports. Of these, only six were implemented as planned, and none were implemented at the regional or neighborhood level. We also found that monitoring indicators did not match the heat impacts, vulnerabilities, and needs identified in the plan. We provide evaluation recommendations to guide and support evaluation and monitoring efforts in the heat planning process.
This dataset is an Excel spreadsheet that includes the full plan list and coding, the coding instructions and the full 399 indicators list as well as a duplicate of the summary table included in the article.
README: Data from: Extreme heat adaptation planning: A review of evaluation, monitoring, and reporting
https://doi.org/10.5061/dryad.vhhmgqp1t
Description of the data and file structure
To understand how evaluation is or is not happening in extreme heat planning, we purposively sampled and analyzed 65 plans that would likely include extreme heat adaptation strategies. We used purposive sampling to identify plans that were most likely to include heat adaptation strategies. In total, the 65 plans covered a variety of jurisdictions: 26 federal, one state, ten regional, five metro-region, 21 municipal, and two neighborhood-level plans. All but one were from the United States.
For each plan document, we answered the following five questions: 1) Is extreme heat identified as a hazard in the plan? 2) Is there a strategy to respond to extreme heat? 3) How was evaluation used during the planning process? 4) How are evaluation and monitoring planned for? And 5) What indicators are included in the evaluation and monitoring plans? We used qualitative analysis software and spreadsheets to code the plans for heat adaptation actions, where evaluation occurred in the planning cycle and to what extent, and indicators or metrics related to heat evaluation. For plans that indicated an intent to evaluate or monitor, we reviewed websites, dashboards, reports, and evaluation activities to determine how governments implemented that monitoring and evaluation. Using Google search engine, we searched the plan lead authors’ website and the web search terms “plan name” AND “report* OR eval* OR monitor*.” We created a code book that described the coding process (See Appendix 2). Two coders worked together to code each plan twice, and both coders also followed the methodology for uncovering whether the evaluation was happening, resulting in an 81% agreement in results. We reviewed the evidence for each discrepancy to finalize the pattern coding.
Files and variables
File: Heat_plan_eval_and_indicators_Holtan_2025.xlsx
Description: Excel document with table of contents, list of plans, coding instructions and indicators.
Variables
- All variables are described and spelled out in the Excel in the coding notes tab and as summarized below:
- Column A: Plan ID
- Column B: Name of document
- Column C: Document author/publisher
- Column D: Year adopted
- Column E: Planning scale
- Columns F through M: Plan analysis variables:
- Y=yes
- N=no
- R=related, but not specific enough to heat to be included in the analysis
- n/a = "not applicable" as the cells associated with that row were not applicable to the planning document as analyzed. For example, plans that did not contain a strategy related to heat, were not analyze to see if there were heat related monitoring indicators or reporting.
- Column N Typology - the type of reporting:
- 1. Yes, Plan for process, report on something else;
- Yes, Plan for impact evaluation, report on process;
- Yes, Selective reporting of positive items only;
- Yes, Data only alone;
- Yes, Constellation of plans website with circular references
- Yes, Evaluation through plan update/re-write;
- No, but there is constellation of plans
- No, but there is a new evaluation plan
- Column O: Report/evaluation date (if applicable)
- Column P: Report name (if applicable)
- Column Q: Report link (if applicable)
- Column R: Document source search category
Code/software
No special software is needed.
Methods
We conducted a qualitative content analysis of 65 plans from federal, state, regional, and municipal sources to understand how planners approached evaluation and monitoring in heat planning. Plan analysis is especially appropriate for heat adaptation research, as planning documents may be the only public-facing documentation of interagency and inter-jurisdiction coordination (Keith et al., 2021). Most plans are adopted by governing bodies and indicate a measure of government accountability, which is essential for climate adaptation (Mees & Driessen, 2019).
We used purposive sampling to identify plans that were most likely to include heat adaptation strategies. Using professional networks and the Google search engine, we found nine standalone extreme heat adaptation plans: one state-level plan, two regional plans, four municipal plans, and two neighborhood-level plans. All but one were from the U.S. We then looked for plans from multiple levels of governance to identify heat adaptation strategies and evaluation included in broader planning efforts. We analyzed all 26 federal climate adaptation plans produced by Executive Order 14008 (Biden, 2021). The plans provided us with an overview of where heat planning and evaluation may or may not be happening at the highest level of government in the U.S. (Biden, 2021). We then pulled plans from an existing database of 175 plans in the 50 most populated municipalities in the U.S. that was created through previous research about how local governments plan for extreme heat (French & Hondula, 2021; Turner et al., 2022). From this database, we identified 20 urban plan documents that included a heat statement related to studying or monitoring in the Northeast and Midwest U.S. because we assumed these would have the highest likelihood of including an evaluation strategy or plan. If the plan had been updated, we analyzed the most recent plan. Lastly, we analyzed ten regional climate resilience plans in New York State to look for approaches to evaluation and monitoring outside of municipal geographies. We selected New York, the Northeast, and Midwest geographies because this research was part of a larger research project for the New York State Department of Environmental Conservation. See Table 1 for exclusion and inclusion criteria.
In total, the 65 plans covered a variety of jurisdictions: 26 federal, one state, ten regional, five metro-region, 21 municipal, and two neighborhood-level plans (See Appendix 1). The geographic bias towards large cities and a variety of scales reflects our purposive sampling designed to identify places most likely to have heat strategies incorporated in plans, and potential monitoring and evaluation as follow up (Gabbe et al., 2021). While the number of plans analyzed is smaller than more automated plan analyses, our interest in the varied ways monitoring and evaluation shows up is conducive to qualitative, “smaller n” analyses (Lieberson, 1991).
For each plan document, we answered the following five questions: 1) Is extreme heat identified as a hazard in the plan? 2) Is there a strategy to respond to extreme heat? 3) How was evaluation used during the planning process? 4) How are evaluation and monitoring planned for? And 5) What indicators are included in the evaluation and monitoring plans? We used qualitative analysis software and spreadsheets to code the plans for heat adaptation actions, where evaluation occurred in the planning cycle and to what extent, and indicators or metrics related to heat evaluation.
For plans that indicated an intent to evaluate or monitor, we reviewed websites, dashboards, reports, and evaluation activities to determine how governments implemented that monitoring and evaluation. Using Google search engine, we searched the plan lead authors’ website and the web search terms “plan name” AND “report* OR eval* OR monitor*.” We created a code book that described the coding process (See Appendix 2). Two coders worked together to code each plan twice, and both coders also followed the methodology for uncovering whether the evaluation was happening, resulting in an 81% agreement in results. We reviewed the evidence for each discrepancy to finalize the pattern coding.
Biden, Joseph R. 2021. “Executive Order on Tackling the Climate Crisis at Home and Abroad.” The White House. January 27, 2021. https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/27/executive-order-on-tackling-the-climate-crisis-at-home-and-abroad/.
French, M, and Emma Hondula. 2021. “How Are U.S. Cities Planning for Heat?” DesignSafe-CI. https://www.designsafe-ci.org/data/browser/public/designsafe.storage.published//PRJ-3180.
Gabbe, C. J., Gregory Pierce, Emily Petermann, and Ally Marecek. 2021. “Why and How Do Cities Plan for Extreme Heat?” Journal of Planning Education and Research, October, 0739456X2110536. https://doi.org/10.1177/0739456X211053654.
Keith, Ladd, Sara Meerow, David M. Hondula, V. Kelly Turner, and James C. Arnott. 2021. “Deploy Heat Officers, Policies and Metrics.” Nature 598 (7879): 29–31. https://doi.org/10.1038/d41586-021-02677-2.
Lieberson, Stanley. 1991. “Small N’s and Big Conclusions: An Examination of the Reasoning in Comparative Studies Based on a Small Number of Cases.” Social Forces 70 (2): 307. https://doi.org/10.2307/2580241.
Mees, Heleen, and Peter Driessen. 2019. “A Framework for Assessing the Accountability of Local Governance Arrangements for Adaptation to Climate Change.” Journal of Environmental Planning and Management 62 (4): 671–91. https://doi.org/10.1080/09640568.2018.1428184.
Turner, V Kelly, Emma M French, John Dialesandro, Ariane Middel, David M. Hondula, George Ban Weiss, and Hana Abdellati. 2022. “How Are Cities Planning for Heat? Analysis of United States Municipal Plans.” Environmental Research Letters 17. https://iopscience-iop-org.gate.lib.buffalo.edu/article/10.1088/1748-9326/ac73a9.