Appendix C Summary of Evaluations: Did Workshop Presentations Enhance Participants’ Understanding?
Workshop attendees and project team members were invited to complete an evaluation after each workshop to assess what information they considered useful, what other information might be useful, and whether the presentations increased their understanding of the topics discussed. The evaluations were part of a formative assessment of the project to determine whether it gathered the appropriate information to achieve its long-term goals.
Questionnaires were distributed via paper at all three workshops. Evaluation was tailored to the specific sessions, although some questions (e.g., whether understanding was increased) were similar across all three workshops. Because separate evaluations were given at the end of each workshop day the number of respondents differs across the various workshop sessions.
The responses were analyzed by the project evaluator and are summarized here by topic as presented in the report chapters.
Interactions – Defining the Problems
Of the 26 evaluations returned after the first workshop, a majority of respondents agreed or strongly agreed that the panel presentations on Interactions afforded them a better understanding of how scientific, engineering, or social interventions influence the interactions of climate, engineered systems, and society (Table A.1). In addition, most of the attendees rated the speakers (McCarthy, Golden, and Bostrom) and respondents (Herkert and Delborne) as effective, although two thought the presentations were too general.
Of the 23 capstone workshop attendees who returned the evaluations, most agreed both that they gained a better understanding of climate change, adaptation, and infrastructure vulnerability (Table A.1) and that they would use the information learned. Nine attendees listed other topics that could have been usefully included:
- More information on ecosystem services; more illumination of human connection to natural world and identification of areas/connections that, if damaged, are the biggest threat.
- Climate change affects the economy/society not just through disasters but also altering the ecology, thus affecting ecological services. We need a discussion on increasing climate change discourse in this area.
- How do international political tensions affect climate change?
- Perhaps a presentation by a government/federal policymaker to provide insight into issues related to how the government sets policy and collects info to do so.
- Rural concerns – thought Patricia Mariella did address this to some extent.
- Engineering solutions that have been implemented and have been successful for adaptation.
- More thorough explanation of exactly what the smart grid is.
- What do the educational and outreach materials for passing this information along to the public look like?
- How to connect/start up a stakeholder network on climate adaptation knowledge access?
TABLE 1 Percentage of respondents indicating they “Agree” or “Strongly Agree” that they gained a better understanding of the problem.
This session gave me a better understanding of: | % |
How scientific interventions influence the interactions of climate, engineered systems, and society. | 65 |
How these influences might affect the success of programs and recommendations. | 69 |
How engineering interventions influence the interactions of climate, engineered systems, and society. | 65 |
How these influences might affect the success of programs and recommendations. | 77 |
How social interventions influence the interactions of climate, engineered systems, and society. | 62 |
How these influences might affect the success of programs and recommendations. | 62 |
Climate change, adaptation, and infrastructure vulnerability. | 87 |
Ways in which climate change, adaptation, and infrastructure vulnerability are interconnected. | 78 |
Interventions – Examining the Range of Sociotechnical Responses
After the first workshop most of the 23 respondents indicated that they gained a better understanding of the social dimensions and potential consequences of adaptation, mitigation, and geoengineering in response to climate change (Table A.2). A majority also rated the speakers (Rubin, Kepke, Daniel, and Robock) and respondents (Johnson and Slutzky) as effective. Although one attendee commented that the talks were too general, others commented positively on the session:
- Really appreciated this session—Ed [Rubin] & Jackie [Kepke] are “in the trenches,” so to speak, very conversant on climate/engineering/policy issues.… This session really crystallized things for me.… Alan Robock’s presentation was also great. Maybe geoengineering shouldn’t be in the grant…but the technical solution is so sexy for engineers? It does seem shortsighted of us not to address it in some way, especially if it happens without us.
- Jackie was a very effective, powerful speaker. Hopefully she will become an exemplar for how CC-relevant engineers ought to be.
- Alan’s geoengineering myths need to become more visible and relevant, especially to engineers that invoke geoengineering as a technical fix to climate change.
After the talks by Munevar and Lapp at the capstone workshop, a majority of attendees agreed that they gained a better understanding of engineering perspectives on climate change and infrastructure problems as well as the ways in which engineering, climate change, and infrastructure resilience are interconnected (Table A.2), although some thought the discussions should have presented examples of successful engineering solutions.
Based on presentations about water resources, including two (Kiker and Lempert) on sea level rise and storm surge and a panel on the Colorado River (Jerla, Brothers, Mahmoud, and Neal), a majority of the 21 respondents reported a better understanding of sea level rise, storm surge, and resources available in and around the Colorado River (Table A.2) and indicated that they would use the knowledge acquired in the sessions in the future. One respondent specifically mentioned “reverse planning” and another the “cluster analysis and relationship/contrast between vulnerability analysis and systems analysis.” Attendees also thought it would have been useful to include human justice and land use issues related to the Colorado River, presentations from different parts of the United States, and a “wider range of infrastructure (e.g., the focus was energy and water/flooding).”
TABLE 2 Percentage of respondents indicating they “Agree” or “Strongly Agree” that they gained a better understanding of the problem.
This session gave me a better understanding of: | % |
Social dimensions of adaptation in response to climate change | 65 |
Potential consequences of adaptation | 74 |
Social dimensions of mitigation in response to climate change | 70 |
Potential consequences of mitigation | 61 |
Social dimensions of geo-engineering in response to climate change | 78 |
Potential consequences of geo-engineering | 91 |
Sea level rise and storm surge | 81 |
The resources in the Colorado River | 61 |
Cross-Cutting Themes
Based on the Cross-Cutting Themes panel at the first workshop, a majority of attendees agreed that they gained a better understanding of justice, sustainability, governance, trust, and public engagement in relation to climate change education (Table 3). The speakers (DesJardins, Thompson, and Moser) were viewed as effective, although one person commented that the presentations needed to link more directly to “climate change or engineered systems or both” in order to apply more directly to the project.
The capstone workshop included two panels relating to cross-cutting themes, and a majority of respondents again indicated that they gained a better understanding of these issues in relation to climate change, adaptation, and infrastructure vulnerability (Table 3) and would use the information they had learned. Despite the positive ratings, some respondents listed information they felt was missing from the discussions:
- A session exploring social justice conceptualizations (in the context of ways of life). More on how clients and communities work with engineers to address problems.
- A little more of a world-view might be nice if there are any, like Australia’s aborigines having a share of water rights.
- More on environmental justice/equity issues.
- The governance panel/discussion could’ve used more examples of cases of the social and political dynamics of how these vulnerabilities were developed in the first place; and how we deal with the structural thinking patterns that may limit adaptation and rebuilding sociotechnical systems.
TABLE 3 Percentage of respondents indicating they “Agree” or “Strongly Agree” that they gained a better understanding of the problem.
This session gave me a better understanding of: | % |
Justice in relation to climate change education | 63 |
Sustainability in relation to climate change education | 54 |
Governance in relation to climate change education | 54 |
Trust in relation to climate change education | 67 |
Public engagement in relation to climate change education | 67 |
The policy and governance challenges and strategies for assessing the problem of climate change, adaptation, and vulnerability | 65 |
The engineering, justice, and human rights challenges inherent in climate change, adaptation, and vulnerability | 70 |
How engineering, climate change, adaptation, policy and governance, justice, and human rights interconnect | 87 |
Potentials in Formal Engineering and Science Education
Of the 23 attendees at the second workshop who returned the evaluations, the majority indicated that the session on effective interventions in undergraduate engineering education had enhanced their understanding of both diffusion of educational innovations and associated challenges, opportunities, and barriers, although they came away with a clearer idea of institutional barriers than local/state or national/federal barriers to integrating climate change and engineered systems in engineering curricula. Most also agreed that they had a better understanding of K–12 science standards and learning progressions, but were less clear about how engineering or climate change fits into K–12 curricula (Table A.4). A few respondents said they had trouble relating information presented in the K–12 engineering session to the project.
Following the capstone workshop, a majority of the 15 respondents indicated they had a better understanding of the challenges of educating engineers about climate change, adaptation, and infrastructure vulnerability (Table A.4) and that they would use the information learned in the session, although some were unsure how to translate the information into practice. Two respondents found Riley’s presentation and initiatives particularly helpful.
TABLE 4 Percentage of respondents indicating they “Agree” or “Strongly Agree” that they gained a better understanding of the problem.
This session gave me a better understanding of: | % |
Diffusion of educational innovation | 96 |
The potential challenges of integrating climate change and engineered systems into the engineering curricula | 96 |
The potential opportunities of integrating climate change and engineered systems into the engineering curricula | 83 |
The potential institutional barriers to integrating climate change and engineered systems into the engineering curricula | 87 |
The potential local and/or state barriers to integrating climate change and engineered systems into the engineering curricula | 35 |
The potential national/federal barriers to integrating climate change and engineered systems into the engineering curricula | 35 |
How engineering curricula are used in K–12 education | 48 |
The potential challenges for the topics of climate change and engineered systems in the changing science standards for K–12 education | 57 |
The potential opportunities for the topics of climate change and engineered systems in the changing science standards for K–12 education | 43 |
The idea of a naturalized philosophy of sciences | 17 |
How science and engineering practices can be integrated into science standards | 74 |
Learning performances and progressions in relation to science standards | 70 |
The challenges of educating engineers about climate change, adaptation, and infrastructure vulnerability | 87 |
Potentials in Informal Engineering and Science Education
Most respondents at the first workshop said they gained a better understanding of the role of science and technology centers in education efforts related to climate change (Table A.5) and found the session useful and engaging. For a majority of the 15 respondents, presentations by representatives of the Boston Museum of Science, the Marian Koshland Science Museum, the Science Museum of Minnesota, and Chabot Space and Science Center yielded a better understanding of informal science and engineering education as well as ways science museums and formal education institutions can work together. Respondents also indicated that they would use this knowledge, although some commented that they were unsure how to translate it into action (Table A.5). There were several suggestions of other information that could have been included:
- Other types of informal education including nonprofit environmental education organizations, environmental learning centers, government agencies like National Park Service, USDA Forest Service Interpretation.
- Historic preservation and architecture perspectives were missing from the conversation and at the whole conference.
- Inspired by Science Center discussion, but would like to consider who these audiences are, who can make it to a museum? Not the vulnerable populations. Cultural inclusion in today’s world is mandatory.
The panel presentation from artists (Roberts, Appelhans, Levy, and Mosher) led a majority of respondents to agree that they gained a better understanding of engaging the public in climate vulnerability and adaptation (Table A.5) and would use the information they learned. As with other sessions, some respondents commented that including ways to apply the information to their work would have been helpful, and one stated specifically that “best practices for stakeholder gathering/engagement [such as] how to recruit, structure and encourage collaboration… could have been included in “Engaging the Public” panel.” Another stated that “it would be better if there is a topic in the public’s sense of value towards the climate change from multi-perspectives including people from different class levels,” but one respondent “appreciated the session on arts/design – thought it was a helpful way to bridge the techno-specific focus of engineering and the human-dimensions focus of human rights/justice, etc.”
TABLE 5 Percentage of respondents indicating they “Agree” or “Strongly Agree” that they gained a better understanding of the problem.
This session gave me a better understanding of: | % |
The role that science and technology (S&T) centers play in the educational community | 91 |
The institutional strengths of S&T centers in communicating multifaceted information | 61 |
The role that S&T centers can play in local and regional outreach | 87 |
How S&T centers will engage the general public and school-aged audiences in the topic of climate change, engineered systems, and society | 96 |
Informal science education opportunities in climate change, adaptation, and infrastructure vulnerability | 100 |
How informal science education and engineering education can work together to increase knowledge of climate change, adaptation, and infrastructure vulnerability | 67 |
Engaging the public in climate vulnerability and adaptation | 76 |
Perspectives of Engineering Professional Societies, Business and Industry, Local Government, and Native Americans
The second workshop included perspectives from both engineering professional societies and industry representatives. Of the 16 attendees who returned the evaluations, a majority agreed that they had a better understanding of the educational priorities of the societies represented (Table A.6), although comments indicated that Lapp (Canadian Standards Association) was the most helpful and informative. A majority of those respondents also indicated that they had a better understanding of what employers expect from graduates in terms of underlying engineering principles and engineering skills, although they did not gain as much knowledge related to expectations of engineering experiences. A majority also gained understanding of whether employers recognize and how they are working to address issues concerning climate, engineered systems, and society (Table A.6). Comments indicated the session was helpful and suggested topics for further discussion:
- Tension between student technical and communication skills – desirable for employers vs. social/ethical training that can’t apply if it isn’t profitable?
- Educating corporations to view justice beyond corporate social responsibility or philanthropy.
- The challenge is always that employers desires are often contradictory or difficult to translate to educational experience (e.g., they must be technically excellent but must also be great communicators but don’t give them too many weird projects).
A majority of the 21 respondents to the capstone workshop panels also agreed that they gained a better understanding of local government solutions to and Native American perspectives on climate change and infrastructure vulnerability (Table 6) and would use the knowledge moving forward. Comments indicated that 7 respondents found the Local Government Solutions and another 2 found the Native American Perspectives panels particularly helpful. Comments also suggested topics such as “Native American traditions of resource management that seem to be part of their culture as a model for thinking about sustainability in the engineering world” for future consideration.
TABLE 6 Percentage of respondents indicating they “Agree” or “Strongly Agree” that they gained a better understanding of the problem.
This session gave me a better understanding of: | % |
The educational priorities of ABET in terms of climate, technology, and society | 69 |
The educational priorities of ASCE in terms of climate, technology, and society | 75 |
The educational priorities of the Canadian Standards Association in terms of climate, technology, and society | 94 |
What employers expect from engineering graduates in terms of underlying engineering principles | 75 |
What employers expect from engineering graduates in terms of engineering skills | 63 |
What employers expect from engineering graduates in terms of engineering experiences | 44 |
Whether employers recognize the issues concerning climate, engineered systems, and society | 88 |
How employers are working to address the issues concerning climate, engineered systems, and society | 69 |
Local government solutions to climate change and infrastructure vulnerability | 100 |
Native American perspectives on climate change and infrastructure vulnerability | 76 |