National Academies Press: OpenBook

Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop (2021)

Chapter: Participant Reflections on WorkshopPresentations and Discussions

« Previous: Challenges and Limitations thatArise When Evaluating SexualHarassment Prevention Efforts
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

5

Participant Reflections on Workshop Presentations and Discussions

Throughout the workshop, participants discussed ways to make progress in evaluating sexual harassment prevention efforts in higher education. They also discussed actions for moving the field forward, including ways to support and strengthen evaluation in sexual harassment prevention in higher education. During some of the workshop sessions, Layne Scherer, a visual practitioner and senior program officer at the National Academies, assimilated ongoing discussions and produced graphics for active ease of viewing. Figure 5-1 shows such a graphic from a session in which Eden King, Rice University, shared key themes from Day 1 of the workshop. This chapter covers participants’ reflections, which are described below.

TAKING A SYSTEMATIC AND OUTCOME-FOCUSED APPROACH

Workshop conversations centered on the notion that evaluation ought to be approached systematically. There is a need to examine sets of programs and relationships in context to look at the issue holistically: Examining the system as a whole allows for clearer comparisons and the ability to generalize, King noted. This has not yet happened; despite the knowledge that programs are embedded in systems, most evaluation methods have been focused on a specific program. Armando Estrada, Temple University, added that thinking more systematically about evaluation, including on

Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

how to intervene, is vital; these are complex problems that require complex solutions. As part of this systemic approach, King, Clemencia Cosentino (National Science Foundation), Elissa Perry (Columbia University), and others noted that considering context is critical in approaching the evaluation of sexual harassment prevention in higher education. The power dynamics are unique in higher education, and this has great implications for the type of evaluation work that is needed.

Further evidence of this need for a systemic approach was supported by the frameworks presented through the prevention and implementation science lens, as discussed by Cindy Crusto, Yale University School of Medicine; Lisa Hooper, University of Iowa; and Raechel Soicher and Kathryn Becker-Blease, Oregon State University. These approaches offered lessons on how to conduct evaluation; while the approaches varied, participants made the point that evaluation should be conducted systematically and strategically.

Citing institutional climate surveys as an example of the need for a more systemic approach to evaluation, Estrada noted that while these surveys may be able to help provide baseline data, they may not be able to capture the complexity of these issues. These surveys may be more useful for diagnosing than prescribing, as noted by Brian Martinson, HealthPartners Institute.

Additionally, workshop participants reflected on the need to approach evaluation of sexual harassment prevention efforts with a focus on the intended outcomes. At the start of the workshop, Perry had stated that outcome measures should be aligned with both program purpose and time frame (short term and long term). Perry also noted that an evaluation process often unfolds over time; in the initial stages, it is often an exploratory process. However, as time goes on, evaluation can help to assess the effect of programs. Establishing an evaluation before implementing a program is important as it encourages discussion about what the program is intended to accomplish. It also provides an opportunity to determine when, how, and what data are needed.

Perry suggested that including those conducting the evaluation in program planning conversations early also allows for planning an evaluation in collaboration with the program. King had also highlighted that defining goals of the program in context is critical. The need to clearly specify outcomes is particularly important given the range of prevention programs and their varied purposes, noted King. Crusto added that it is important to recognize that no one culture or context is static; things change all the time—the outcomes that one had hoped for at one time may be affected by

Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Image
FIGURE 5-1 Summary of key themes from Day 1 of the workshop.
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

these changing contextual factors and, as such, one may not get the same outcomes in an ongoing or sustainable way.

King noted that outcomes exist at many levels, which complicates evaluation of sexual harassment prevention; evaluations must capture not only behavior or knowledge, but also the secondary and tertiary implications of programs and policies. She added that there is a need to broaden the notion of outcomes to be inclusive and related to the ultimate goals of the program. Relatedly, metrics should match the goal, stated King. As one participant noted, if the goal is to provide knowledge about sexual harassment through prevention efforts, one should measure that knowledge. There is also a need to measure ideas and constructs from multiple perspectives, another participant commented. King noted that needs analysis and change readiness can be used to support these efforts. Vicki Magley, University of Connecticut, noted that the distinction between how to measure effectiveness and the choice of study design is a big one. In organizations, people often ask “how do I measure . . . ,” but they really are asking questions about inference, and valid inferences ensue from strong research designs.

CONSIDERING THOSE MOST VULNERABLE TO HARASSMENT

Participants noted the importance of building relationships during evaluation, centering around those marginalized groups most likely to be targets of harassment (e.g., historically excluded groups, international faculty and students, undocumented students, etc.). As one participant noted, “If you want to do evaluation, you need to build relationships first. We are embedded in a culture that does not allow diverse voices to collaborate and be present; putting a survey out is not intersectional.” King noted the importance of relationships, particularly in institutions with only a few people of color in faculty who may be included in an evaluation, and highlighted that expanding efforts beyond surveys or subgroup analyses will help cultivate such relationships.

As mentioned in Chapter 4 and reinforced by Melissa Kwon, University of California, Berkeley, it is important to make the connection between evaluating sexual harassment and diversity, equity, and inclusion (DEI) explicit. Gómez, Kelley Bonner (National Oceanic and Atmospheric Administration), and others added that there is a need to carefully consider the connection between sexual harassment and DEI in evaluation, par-

Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

ticularly as DEI efforts are needed to protect groups at highest risk of harassment. As Nelia Viveiros, University of Colorado, Anschutz Medical Campus, stated, there is a need to tie in issues of intersectionality into sexual harassment programs. Micro-interventions and proximal goals could be useful to this end and to better gauge progress. As Bonner notes, “we create this false division between sexual harassment and DEI efforts. We completely exclude the concept of intersectionality and its huge impact on prevention and response efforts for harassment.”

Complicating this challenge, said Theodosia Cook, University of Colorado, DEI leaders often find themselves as the only people with an equity lens at the leadership table; it is also very likely that they are the only woman, person of color, or immigrant. In this role, they are also too frequently discounted by people in academia. There is a need to create conditions for DEI leaders to lead, she added. As one participant noted, it is important to use an intersectional lens for both programs and measurement, and to build collaborations with others, including DEI staff and leaders.

More training on intersectionality is needed, one participant noted, because “many of us in the sexual violence field are not trained on DEI, and many in the DEI field are missing training on the layered impact of sexual violence.” In fact, another participant noted, “I’m more interested in evaluation of accountability and culture change at the top of the hierarchy than training programs. Misunderstandings at the top are part of the reason DEI and sexual harassment are handled separately.” Others noted that continuing to develop measures that capture power dynamics is important in this space, particularly given that these dynamics persist in higher education.

Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

One participant noted: “Some of our communities that experience marginalization have also said it is embarrassing to have to keep ‘justifying’ why we need to do this work. It belittles their humanity.”

LEADERSHIP AS A LINCHPIN: BUILDING SUPPORT AND COMMUNICATING VALUE

Workshop participants reflected on the need to build the interest and capacity of leadership to support evaluation of sexual harassment prevention efforts. Results from evaluation efforts can be used to encourage leaders to be more involved in sexual harassment prevention. Reinforcing this point, Bonner stressed the importance of communication with leaders; providing leaders with the information from evaluations, logic models, and other sources can help to develop the case for improving programs. Discussion with leaders about sexual harassment as posing a “risk” to the institution is also critical, noted Bonner. As one participant noted, “We have to figure out how to make leadership more afraid of not knowing than knowing.”

Estrada added that leaders may recognize that their institution has a problem but may not have a concrete understanding about how to address it. Through evaluation, they may have a framework they can use to unpack a complex situation and translate action into research. Additionally, continuous leadership and engagement was raised as particularly important by participants discussing Case Study D (in Appendix D), which describes an effort by Argonne National Laboratory to encourage employees to recognize colleagues for demonstrating the institution’s “core values.” In this case study, the program was originally set to run for a short period, but through leadership’s support it was extended because of its value as a positive tool for recognition and encouragement of positive behavior.

Bonner noted that it is the job of prevention staff and others managing sexual harassment prevention efforts to support leaders so that they feel

Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

confident and competent to address these challenges. Successful evaluation of sexual harassment prevention efforts requires the ability to appeal to different types of leaders through compelling arguments about the importance of these programs and efforts. As one participant noted, evaluative measures could be used to convince leadership about the value of prevention programs.

In discussions about how to create incentives to engage leadership, Karen Stubaus, Rutgers University, and others suggested developing or incorporating outcomes related to sexual harassment in U.S. News–type rankings or other rankings of best places to work. Janine Clayton, National Institutes of Health (NIH), noted that institutions should also consider opportunities to encourage adoption of proven strategies, such as NIH’s Challenge Prize for Enhancing Gender Diversity, which recognizes institutions that have achieved sustained improvement in gender diversity. Participants also cited the STEMM Equity Achievement (SEA) Change program of the American Association for the Advancement of Science (AAAS), which aims to advance institutional transformation in support of diversity, equity, and inclusion in higher education institutions. Another participant noted that evaluation research that supports the idea that people can stop the harm in the first place can be used to convince leadership of the program’s value. Evaluation is the basis for getting funding and demonstrating outcomes and can allow practitioners to appeal to leaders, stated Kiana Swearingen, University of Washington.

EXPANDING THE EVALUATION TOOLKIT

King and others noted the importance of considering a broader range of tools, both those that have been used historically and those that are new and innovative, in the evaluation of sexual harassment prevention efforts. This could include text mining for techniques or papers that have examined climate culture attitudes or identifying new forms of data analysis that may be less obtrusive. Pulse surveys, in which participants are asked one question per week, were also discussed by King and others. These surveys may

Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

allow for more immediate insight into measurement of changes given their frequency. King and others noted that bystander intervention research and other older techniques may be useful to apply to the measurement toolkit. Kurt Kraiger, University of Memphis, added that measures, such as a sense of psychological safety could be useful, as well as understanding the number of women (and others) who leave the institution. Another participant noted that as institutional climate is the strongest predictor of the presence of sexual harassment, it may be important to focus on metrics that focus on this predictor.

Participants discussed the importance of incorporating evaluation from the beginning. Michael Leiter, Acadia University and Deakin University, noted that talking to people within an organization about what problems they are facing may be a good place to start. “It is also critical to think outside the box about what evaluation can look like at the outset, including to the target audience,” stated Jane Stapleton, Soteria Solutions. The target audience is central to understanding what needs to be evaluated and should be involved in identifying new solutions and methodologies, as noted by one participant. As Kwon discussed, a program should be developed alongside stakeholders from the beginning and throughout.

Related to conversations about building trust, multiple presenters discussed approaches for improving participant engagement. Leiter stated that strong participant engagement is crucial to the program success and the ability to collect accurate data. This is particularly true in the health care industry where staff are often over-surveyed. The risk of over-surveying participants is particularly important to consider for marginalized populations; we need to use these surveys and data responsibly, stated Swearingen. There is also a need for larger discussion at the institutional level about what we are asking of those we are surveying, stated Stapleton, adding that there are other ways to collect data than surveys.

Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

BUILDING COMMUNITY, COLLABORATION, AND TRUST

Participants reflected on the importance of continuing to collaborate and share resources, working together to support evaluation efforts in higher education. Swearingen and others discussed the need to continue to share resources and work together to move toward broader evaluation of sexual harassment prevention efforts in higher education. This may mean moving away from the mindset that sharing information about available resources can be detrimental in situations where there is a scarcity of those resources. Swearingen offered an example of an unhelpful scarcity mindset: “If I do something successful, I’m not going to share it, because then I might not get funding.” The idea of sharing information requires a fundamental culture change for academia.

Participants reflected on the importance of transparency and trust in the population being evaluated. Kwon and others reiterated the importance of transparency and trust in developing and evaluating programs designed to prevent sexual harassment. Building trust among participants is critical and must be developed by engaging participants throughout the process, noted several participants. As Stapleton said, providing participants with data throughout the evaluation process, including after the work is completed, can help build trust and transparency, which are particularly important in issues related to sexual harassment.

Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

This page intentionally left blank.

Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 29
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 30
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 31
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 32
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 33
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 34
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 35
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 36
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 37
Suggested Citation:"Participant Reflections on WorkshopPresentations and Discussions." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 38
Next: Evaluation in Action:Examples and Resources »
Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop Get This Book
×
Buy Paperback | $50.00 Buy Ebook | $40.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Rising awareness of and increased attention to sexual harassment has resulted in momentum to implement sexual harassment prevention efforts in higher education institutions. Work on preventing sexual harassment is an area that has recently garnered a lot of attention, especially around education and programs that go beyond the standard anti-sexual harassment trainings often used to comply with legal requirements.

On April 20-21, 2021, the National Academies of Sciences, Engineering, and Medicine hosted the workshop Developing Evaluation Metrics for Sexual Harassment Prevention Efforts. The workshop explored approaches and strategies for evaluating and measuring the effectiveness of sexual harassment interventions being implemented at higher education institutions and research and training sites, in order to assist institutions in transforming promising ideas into evidence-based best practices. Workshop participants also addressed methods, metrics, and measures that could be used to evaluate sexual harassment prevention efforts that lead to change in the organizational climate and culture and/or a change in behavior among community members. This publication summarizes the presentations and discussion of the workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!