https://doi.org/10.5061/dryad.h18931zx2

Description of the data and file structure

A meta-synthesis of food, energy, and water systems manuscripts was conducted to determine the extent of stakeholder engagement, how stakeholders were engaged, outcomes of engagement, and to identify best practices in stakeholder engagement.

Files and variables

Files: crc-engage.zip includes a PDF description of R analysis contained in Supplemental Information (Supplemental_Information_2025.pdf), an Excel workbook, and a .csv for R data analysis

Description: Description of R analysis of the data

File: csv_for_statistical_analysisAug2025.csv

Description: csv for input into R for analysis

Variables

• were stakeholders involved (Y, N)

• level of stakeholder engagement (Ghodsvali Scale) - Whether stakeholders were involved in the research and details about their involvement if they were included. Stakeholder engagement is defined and operationalized as the inclusion of non-academic people or organizations in any part of the research process. Table 3 describes the SE scale used (Ghodsvali et al., 2019).

• solution proposed and/or implemented - Whether a solution was proposed and details about the solution if it was proposed. A solution is defined and operationalized as a proposition to address or solve the identified problem. The article must explicitly articulate the solution, including a mechanism or plan of action for its implementation. (e.g., process, workflow, or framework with defined stages or phases) with details on the actions and resources needed to apply the solution. Providing the tools and methods for implementation of the solution to stakeholders is considered a solution, even if the stakeholders do not implement it. Solutions were required to be studied as a variable or directly related to the variables studied in the model, experiment, or methods.

  The solution must have intended outcomes to improve the previous state; must relate to the research conducted in the article; must inform decisions, decision-making, or behavior in practice; must be implementable and achievable; cannot be recommendations for future research; and cannot be hypothetical or theoretical. Whether the solution was implemented and details about the implementation. Implementation is defined and operationalized as the application in practice of methods specified in the article by making a tool, process, or other solution accessible to decision-makers, by creating a system to employ the methods in practice, or by putting the methods into practice within the community (Albrecht et al., 2018). If a computer or statistical model was accessible to policymakers through an interface allowing them to interact with the model without training and a url to use the model was provided we counted that as an implemented solution.

• solution type - Technological

  Use of engineered or designed technology (civil, chemical, computer, electrical, and mechanical). Examples include sensors to monitor and adjust use (e.g., energy or water), anaerobic digesters, alternative or green energy generators, computational or statistical models and software, cyberinfrastructure, gamification, infrastructure additions, improvements, or changes to infrastructure (Carroll, 2017).

  Policy or Policy Change

  Use of bylaws, codes, regulations, statutes, strategies, or other formal policy mechanisms (Colebatch, 2009).

  Institutional

  Use of mechanisms that are directed at change at a governance level (city, county, state, nation, academic). This includes interventions by the institutions themselves, such as changes in mandate, mission, organization, roles, structure, members, functions, relationships, procedures, operations, or the creation of a new institution (Gräbner and Ghorbani, 2019).

  Social

  Any solution that requires awareness of, consideration of, or modification of attitudes, beliefs, preferences, behaviors, biases, mores, norms, or values. Includes outreach. Potential solutions could include capacity building (e.g., increases in the ability of groups to overcome a problem, providing money, social capacity).

  Economic, Financial, or Market

  Use of economic instruments including taxes and subsidies. Includes both economic incentives (tax credits, investments, subsidies, etc.) and economic disincentives (imposing a tax, fines, penalties, tariffs, etc.) Also includes economic analysis or economic models if proposed as part of another solution. Economic models do not count as economic solutions in and of themselves; the authors must provide an economic recommendation and a user interface and access (Backhouse and Medema, 2009).

  Ecological

  Use of biota, conservation practices related to ecosystems, ecological corridors, greenspaces, organisms, or pathogens to effect changes or maintenance of atmospheric, hydrological, oceanic, or terrestrial systems. Ecological outcomes or models do not count as ecological solutions on their own. Instead, the authors must provide an ecological recommendation (Wallace, 2007).

  Educational

  Use of instructional, formal or informal learning approaches to inform or transfer knowledge to citizens or stakeholders and improve their situational awareness and critical thinking. This includes games to educate and inform. This also includes service/outreach (e.g., service-learning, extension services, etc.). Education solutions transfer knowledge through tools; provide frameworks to address issues and self-sustaining methods to continue improved performance within communities; provide outreach to or from schools, and other educational institutions. The main goal of an educational solution is to improve awareness, cognition, and critical thinking (Rizvi and Lingard, 2009).

• researcher field of expertise - Disciplines of researchers were based on the affiliation or department(s) of the authors listed in the manuscript. These were categorized as engineering, mathematical/statistical, computer science, physical science (e.g., chemistry, biology, ecology, hydrology, etc.), interdisciplinary (environmental science, sustainability or an institute that included both physical and social science components), social science, economics and agriculture. If the discipline was unclear from the name of the department, a Google search was conducted and a determination was made based upon the description on the departmental website. For those departments we were unable to find, we did not include the discipline in the analysis.

• stakeholder type - Based on working group experience and discussions we included farmers; governmental; tribal/indigenous; combined coalitions, NGOS and citizen groups; business/industry; general public; youth; academics; experts; and migrants. "Experts" was added as a separate category because this was the word used in many manuscripts to describe their stakeholders.

• Geographic scale of study - Local included households to metropolises. Regional included county to sub-national levels. National included nations. Multi-national included more than one nation and Global included worldwide studies. NA was coded if the scale of the study was not described (Leitner, 1997).

• Location of study/regional scale - Country where the research or a case study took place. This could be the location of: data used in a model, experiment, community engagement, or other research. The countries were combined into the following geographic areas for analysis: China, US, Africa, Asia, Central America, Europe, Global, Middle East, North America, Oceania, and South America. If no region of study was provided, it was coded NA. China and the US were analyzed separately from their region because these two countries accounted for an outsize proportion of the articles (34%).

• Was a computer or statistical model used - If a computer-generated or mathematical model was used, we input Y in the cell and entered a short description of the model in Excel.

File: complete_ENGAGE_analysisclean.xlsx

Description: Original data entered from manuscripts into Excel in narrative or numerical form. Literature search and selection. Absent data are listed as "N/A" or are blank cells.

To synthesize the literature on development and implementation of solutions to FEWS challenges, we conducted a systematic review, following PRISMA guidelines for systematic reviews and meta-analyses (Page et al., 2021). Systematic reviews can be designed to answer narrowly defined questions and critically appraise evidence (Atkinson and Cipriani, 2018)– we do this for the three research questions listed above.

Eligibility criteria/Search strategy

Our search string included the terms: [(“food energy water” OR “food water energy” OR “water energy food” OR “water food energy” OR “energy water food” OR “energy food water”) AND (stakeholder OR engagement OR community OR nexus)]. In ScienceDirect, we searched for this Boolean expression in “Title, Abstract, and Author specified keywords” and in WorldCat we searched in “Keywords.” We limited our search to the English language and to journal articles and book chapters. The initial searches returned 717 articles from which we removed 3 duplicate records.

We reviewed the abstracts of the remaining 714 articles to verify whether the study included both SE and FEWS. If the abstract was unclear, we conducted a text search to examine the context around key terms within the body of the articles. If there was no indication that the paper included at least one term from both groups of terms (FEWS and SE) and that the paper related to at least two FEWS sectors, it was excluded (n=123). Any paper was also excluded if it was a review article or a purely conceptual paper, as we wanted to focus on solutions based on empirical investigation (n=106). We then manually screened the papers and removed papers that did not meet the criteria above upon reading the manuscript, yielding 483 papers for inclusion in the review (Figure 1).

Information sources

We searched the ScienceDirect and WorldCat online databases to identify relevant peer-reviewed research articles published between 2015 and April 2023.

Selection process/Data collection process/Bias assessment

Following the principles of systematic literature review (PRISMA) we analyzed each article to identify variables describing geographic scale, region, a stated proposed solution, a stated implemented solution, category of stakeholder, discipline of researcher(s), types of FEWS solutions, and types of stakeholder engagement levels (Tables 1 – 3). To start, a sample of 50 articles were assessed by two team members to ensure agreement on the concepts of interest, particularly regarding the level of engagement and the types of solutions proposed. Potential biases in study selection, types of solutions, and engagement levels were further minimized by assigning screening tasks to more than one person independently with regular reviews of the sample to address reviewer bias. Once sufficient agreement was reached, one team member individually analyzed the remainder of the papers with regular sampling and review by the coauthor group. Any questions were taken to monthly working group meetings (consisting of the authors and representatives of the research coordination network) for discussion and resolution.

Table 1: Description of the variables that were examined in the selected research articles.

Research Variable

Description

Proposed solution

(Dependent variable)

Whether a solution was proposed and details about the solution if it was proposed. A solution is defined and operationalized as a proposition to address or solve the identified problem. The article must explicitly articulate the solution, including a mechanism or plan of action for its implementation. (e.g., process, workflow, or framework with defined stages or phases) with details on the actions and resources needed to apply the solution. Providing the tools and methods for implementation of the solution to stakeholders is considered a solution, even if the stakeholders do not implement it. Solutions were required to be studied as a variable or directly related to the variables studied in the model, experiment, or methods.

The solution must have intended outcomes to improve the previous state; must relate to the research conducted in the article; must inform decisions, decision-making, or behavior in practice; must be implementable and achievable; cannot be recommendations for future research; and cannot be hypothetical or theoretical.

Implemented solution

(Dependent variable)

Whether the solution was implemented and details about the implementation. Implementation is defined and operationalized as the application in practice of methods specified in the article by making a tool, process, or other solution accessible to decision-makers, by creating a system to employ the methods in practice, or by putting the methods into practice within the community (Albrecht et al., 2018). If a computer or statistical model was accessible to policymakers through an interface allowing them to interact with the model without training and a url to use the model was provided we counted that as an implemented solution.

Stakeholder type

(Independent variable)

Based on working group experience and discussions we included farmers; governmental; tribal/indigenous; combined coalitions, NGOS and citizen groups; business/industry; general public; youth; academics; experts; and migrants. "Experts" was added as a separate category because this was the word used in many manuscripts to describe their stakeholders.

Discipline of researchers

(Independent variable)

Disciplines of researchers were based on the affiliation or department(s) of the authors listed in the manuscript. These were categorized as engineering, mathematical/statistical, computer science, physical science (e.g., chemistry, biology, ecology, hydrology, etc.), interdisciplinary (environmental science, sustainability or an institute that included both physical and social science components), social science, economics and agriculture. If the discipline was unclear from the name of the department, a Google search was conducted and a determination was made based upon the description on the departmental website. For those departments we were unable to find, we did not include the discipline in the analysis.

Geographic Scale

(Independent variable)

Local included households to metropolises. Regional included county to sub-national levels. National included nations. Multi-national included more than one nation and Global included worldwide studies. NA was coded if the scale of the study was not described (Leitner, 1997).

Regional Scale

(Independent variable)

Country where the research or a case study took place. This could be the location of: data used in a model, experiment, community engagement, or other research. The countries were combined into the following geographic areas for analysis: China, US, Africa, Asia, Central America, Europe, Global, Middle East, North America, Oceania, and South America. If no region of study was provided, it was coded NA. China and the US were analyzed separately from their region because these two countries accounted for an outsize proportion of the articles (34%).

Model

(Independent variable)

If a computer-generated or mathematical model was used, we input Y in the cell and entered a short description of the model in Excel.

Stakeholder involvement

(Independent variable)

Whether stakeholders were involved in the research and details about their involvement if they were included. Stakeholder engagement is defined and operationalized as the inclusion of non-academic people or organizations in any part of the research process. Table 3 describes the SE scale used (Ghodsvali et al., 2019).

Time of stakeholder involvement

(Independent variable)

If adequately described in the article, we characterized when stakeholders were involved as throughout the project, early in the project, midway in the project or late in the project.

 

Table 2: Description of the seven types of solutions that were identified in the review.

Types of Solutions

Description

Technological

Use of engineered or designed technology (civil, chemical, computer, electrical, and mechanical). Examples include sensors to monitor and adjust use (e.g., energy or water), anaerobic digesters, alternative or green energy generators, computational or statistical models and software, cyberinfrastructure, gamification, infrastructure additions, improvements, or changes to infrastructure (Carroll, 2017).

Policy or Policy Change

Use of bylaws, codes, regulations, statutes, strategies, or other formal policy mechanisms (Colebatch, 2009).

Institutional

Use of mechanisms that are directed at change at a governance level (city, county, state, nation, academic). This includes interventions by the institutions themselves, such as changes in mandate, mission, organization, roles, structure, members, functions, relationships, procedures, operations, or the creation of a new institution (Gräbner and Ghorbani, 2019).

Social

Any solution that requires awareness of, consideration of, or modification of attitudes, beliefs, preferences, behaviors, biases, mores, norms, or values. Includes outreach. Potential solutions could include capacity building (e.g., increases in the ability of groups to overcome a problem, providing money, social capacity).

Economic, Financial, or Market

Use of economic instruments including taxes and subsidies. Includes both economic incentives (tax credits, investments, subsidies, etc.) and economic disincentives (imposing a tax, fines, penalties, tariffs, etc.) Also includes economic analysis or economic models if proposed as part of another solution. Economic models do not count as economic solutions in and of themselves; the authors must provide an economic recommendation and a user interface and access (Backhouse and Medema, 2009).

Ecological

Use of biota, conservation practices related to ecosystems, ecological corridors, greenspaces, organisms, or pathogens to effect changes or maintenance of atmospheric, hydrological, oceanic, or terrestrial systems. Ecological outcomes or models do not count as ecological solutions on their own. Instead, the authors must provide an ecological recommendation (Wallace, 2007).

Educational

Use of instructional, formal or informal learning approaches to inform or transfer knowledge to citizens or stakeholders and improve their situational awareness and critical thinking. This includes games to educate and inform. This also includes service/outreach (e.g., service-learning, extension services, etc.). Education solutions transfer knowledge through tools; provide frameworks to address issues and self-sustaining methods to continue improved performance within communities; provide outreach to or from schools, and other educational institutions. The main goal of an educational solution is to improve awareness, cognition, and critical thinking (Rizvi and Lingard, 2009).

 

Table 3: Description of stakeholder engagement levels from the framework used in the analysis (Ghodsvali et al., 2019).

Categorization of FEWS Engagement

Engagement Level

Description

Nominal

Limited stakeholder involvement that does not lead to change in the outcomes (e.g., data gathering from stakeholders)

Instrumental

Use of skills and knowledge of stakeholders in the research process

Representative

Give stakeholders a voice in decision-making and in the implementation of solutions that may affect them

Transformative

Focus on empowering local stakeholders

 

Data items/Synthesis methods

We identified solutions and determined if they were implemented using the guidelines set out in Table 1. We then categorized the identified solutions into seven types: technological, policy, institutional, social, economic, ecological, and educational (Table 2). The typology of solutions was an outcome of an expert panel assessment from the more than 100 members of the EngageINFEWS research coordination network (Kliskey et al., 2021). For technological solutions, only computational models made available in a manner usable by policymakers and/or the general public were included as solutions, which required providing the URL and a user interface. Models with the sole purpose of the advancement of computational or statistical science, or for the purpose of research, were identified but not counted as solutions. For ecological solutions, we differentiated between ecological actions (e.g., preserving wetlands) which were included as solutions, versus ecological outcomes (e.g., improved water quality) or other activities that did not have explicit ecological influence (e.g., water recycling) which were not included. Finally, we included only solutions that were directly related to the research and were specifically described. For example, a study that said ‘policy solutions should be created for water management’ would not count as a solution, but a study that recommended ‘policies that provide economic subsidies to farmers to improve the efficiency of their irrigation practices’ with a plan or mechanism described for providing those subsidies would count. Solution types were coded in a CSV file for subsequent analysis in R using binary values for each of the seven separate types of solutions. Each solution type was entered into a separate column, so technological solutions, for example, in one column, was coded a ‘0’ if that type of solution was not present and coded as ‘1’ if a technological solution was present. This binary coding was undertaken for each solution type across seven separate columns. If there was missing data it was simply excluded. Table 2: Description of the seven types of solutions that were identified in the review. We also identified whether stakeholders were engaged and if so, categorized the type of engagement and when they were engaged (Table 3). We define a ‘stakeholder’ as actors or organizations who have an interest in, have power or influence over, or are affected by the research or its implications. This may include individuals or groups, state or non-state actors, community members, or others. Although researchers and scientists can be stakeholders (Kliskey et al., 2021), for this analysis they were not counted if they were the only stakeholder but were included if there were other stakeholders. We defined ‘stakeholder engagement’ (SE) as the inclusion of people and/or organizations in any part of the project or research (Kliskey et al. 2021). To categorize levels of SE, we used a framework adapted from a categorization of transdisciplinary research for the FEWS nexus that categorizes SE at four levels: Nominal, instrumental, representative, and transformative (Ghodsvali et al., 2019).

The levels denote increasingly greater roles, influence, and/or authority given to stakeholders during the engagement process. Table 3 provides a description of the levels of engagement in the framework. Stakeholder engagement levels were coded in a CSV file for subsequent analysis in R using binary values for each of the four separate types of engagement level. Each engagement level was entered into a separate column, so nominal engagement, for example, in one column, was coded a ‘0’ if that type of engagement was not present and coded as ‘1’ if a nominal engagement was present. This binary coding was undertaken for each engagement level across four separate columns. If there was missing data it was simply excluded. Table 3: Description of stakeholder engagement levels from the framework used in the analysis (Ghodsvali et al., 2019). If described in the article, the following variables were entered into Excel in narrative form or entered as a count in a category as appropriate: discipline of researchers, stakeholder type, study location by region and spatial scale, time of stakeholder engagement, type of stakeholder and a narrative description of proposed and/or implemented solutions along with a count for each solution type. The following variables were coded into a csv file for analysis in R: solution proposed or implemented (Y, N), computational or statistical model (Y, N), solution type, researcher discipline (count for each discipline represented in the paper), stakeholder type (count for each type listed), geographic region, and spatial scale (1 if included, 0 if not). If there was missing data it was simply excluded.

Statistical analysis

All analyses were performed using R, version 4.1.2. Given our interest in determining which factors influence whether solutions are proposed and/or implemented, our approach was threefold (Figure 2):

1. Descriptive statistics. Overall descriptive statistics and exploratory data analysis was performed to provide a broad assessment of manuscript characteristics. Frequencies and summary plots were used to examine trends over time and distributions across stakeholder types and engagement levels. Plotting and graphs were produced using the ggplot2 package.

2. Generalized and logistic regression models. To assess multi-categorical and binary outcomes for both stakeholder engagement and whether a solution was proposed or not, several regression-based models were run across the entire 483 sample dataset. The primary dependent variable was whether a solution was proposed / developed while all other variables were used as independent variables (Table 1). Logistic regression models were used to estimate the odds ratios of proposing or implementing a solution based on specific stakeholder engagement categories. For categorical predictors, such as engagement level or stakeholder type, odds ratios (OR) compare each category to a reference group. Confidence intervals (95%) accompany each OR to indicate estimated precision, with significance evaluated using an alpha of .05. Regression models were produced using the caret package.

3. Random forest classification. Random forest classifiers (ensembled decision trees) were employed to identify the most important predictors of solutions, as well as to visualize the conditional structure of these relationships. Again, the primary dependent variable was whether a solution was proposed / developed. Decision trees are a method of constructing a set of decision rules on a predictor variable (Brieman et al., 1984; Verbyla, 1987; Clark and Pregibon, 1992) that is categorical, using variance partitioning. These rules are constructed by recursively partitioning the data into successively smaller groups with binary splits based on a single predictor variable, with the goal of describing the relationship using the smallest number of splits (tree ‘branches’) (Prasad et al., 2006). Random forests, or ensemble decision trees, are a combination of many decision trees, where each tree depends on the values of a random vector, sampled independently and with the same distribution for all trees in the forest (Breiman, 2001). Random forest modeling reduces the potential for overfitting using bootstrap aggregation (averaging across many trees) and provides a level of feature importance for assessing predictor power. Each random forest model (1,000 trees) was evaluated with stratified 10-fold cross-validation (repeated 5x) in the caret package, with area under the curve (AUC) computed from the out-of-fold predictions. We also report the apparent AUC from the model refit to the full dataset with the selected hyperparameters. For single tree comparisons we used the recursive partitioning and regression trees package rpart (CART), while random forests were implemented with ranger.(Breiman et al, 1984; Wright and Ziegler, 2017; Atkinson and Therneau, 2000;).

 

Variables

• time of engagement of stakeholders - If adequately described in the article, we characterized when stakeholders were involved as throughout the project, early in the project, midway in the project or late in the project.
• activities in which stakeholders were engaged - if described in the article, we copied and pasted that description into the Excel worksheet. These included, for example, questionnaires, focus groups, scenario development, brainstorming, visioning, identifying the research issue or problem, workshops, etc. We counted the number of articles that described how they engaged their stakeholders.
• how were stakeholders identified - if described in the article we described how researchers identified their stakeholders. These included, for example, interviews, internet searches, snowball sampling, recommendations of key players, etc. We counted the number of articles that described how they identified stakeholders.

Code/software

Excel is needed to view the Excel file, Excel or a similar software is needed to view the csv. No software is needed to view the pdf. R 4.4.2 was used to run the csv.

Workflow: Literature identification -> literature assessment -> coding of data from literature -> independence testing -> generation of stakeholder diversity index -> generalized linear model