5

Assessing the Science Activation Portfolio: Broadening Participation and Networks

In this chapter, we continue our discussion of the Science Activation (SciAct) Phase 1 portfolio assessment, focusing on the two remaining themes: broadening participation and networks. As described in the previous chapter, the committee arrived at these two themes because they emerge repeatedly as important issues for SciAct stakeholders. Indeed, each of these themes is described in SciAct materials as both central components of and desired outcomes for the portfolio’s work. For this reason, the committee took care to enhance its collective expertise through invited presentations on best practices and considerations for prioritizing diversity, equity, inclusion, and accessibility in network models. Similar to our discussion in Chapter 4, for each theme we discuss principles to consider based on the current evidence base, current representation in the portfolio, and considerations for the next phase of the program.

BROADENING PARTICIPATION

SciAct’s mission is to deliver activities and experiences to learners of many backgrounds and to leverage scientist-educator partnerships that have demonstrated diverse, broad, and deep national education and communications impact. In her presentation to the committee, Director Kristen Erickson emphasized that SciAct’s efforts toward broadening participation are directly aligned with one of the priority areas described in the current national agenda for science, technology, engineering, and mathematics (STEM) education: “to increase diversity, equity and inclusion in STEM” (National Science and Technology Council, 2018, p. 5). Additionally, the

efforts of SciAct are linked to one of the aims of the National Aeronautics and Space Administration’s (NASA’s) education and communications program, which seeks to better serve groups historically underrepresented in STEM fields. Thus, broadening participation is clearly identifiable as one of SciAct’s stated commitments and values.

Principles for Broadening Participation

When education and outreach efforts are focused on broadening participation, it is important to consider four distinct aspects of the design and implementation of the program: diversity, equity, inclusion, and accessibility (DEIA). In her presentation to the committee, Julie Johnson, program director and Advancing Informal STEM Learning (AISL) Program lead at the National Science Foundation, provided definitions for each of these components. Diversity is defined as the wide range of differences among people and their perspectives; equity refers to the fair and just treatment of all members of a community; inclusion refers to seeking and embracing contributions from all sources, including underrepresented groups, regions, and institutions; and accessibility is giving equitable access to everyone along the continuum of human ability and experience. Organizations can indicate and demonstrate DEIA as a core value through expressed commitments, organizational policy, operational practices, grant-making or other program areas, and accountability mechanisms (Johnson, 2019).

The SciAct Program administrators and awardees place emphasis on reaching a diversity of learners. This is an important strategy, particularly because education systems are becoming more diverse. For example, the Department of Education predicts that by 2022, 54.7 percent of the U.S. student population will comprise students of color (Stevens, 2015). The importance of inclusion is highlighted by current research indicating that the benefits of diversity are not realized without substantive and deliberate efforts toward inclusion (Sherbin and Rashid, 2017). Simply increasing the diversity within an organization does not lead to a sense of belonging for all participants (Puritty et al., 2017). Other indicators of inclusivity within an educational organization may include the following:

• Individuals from the community that an initiative is intended to serve are involved, as partners, leaders, advisors, and staff.
• The voices and perspectives of those that an initiative is intended to serve are heard and inform decision making.
• The initiative is co-designed to address the needs and interests of the intended audience.
• The program or project staff develop and demonstrate skill in working with the full range of intended audiences.

A third element in DEIA, equity, is distinct from equality (Nasir et al, 2014). Equality is providing the same resources to all groups, which assumes a level playing field and posits that educational equality will be realized when the same quantity and quality of resources characterize the educational experiences of all racial groups in the United States (Parsons and Turner, 2014). However, this conceptualization leaves intact the political, economic, social, and educational barriers that some groups face. Equity, in contrast, recognizes that different resources or approaches are needed to remedy the uneven playing field in order to produce and sustain the highest-quality outcomes for everyone (Grogan, 1999; Parsons and Turner, 2014). Equity “requires commitment to strategic priorities, resources, respect and civility, as well as ongoing action and assessment of progress towards achieving specified goals” (Johnson, 2019). Furthermore, incorporating equity into educational programs may require greater use of culturally responsive or other equity-based pedagogical approaches that enhance achievements for diverse learners. It may also require specific focus on the economic diversity of the populations served, given that access to high-quality resources (e.g., curriculum, technology) is often predicated on income (Gay, 2002; O’Day and Smith, 2016).

Accessibility includes the design of products, devices, services, or environments for people with disabilities (Johnson, 2019). According to the American Alliance of Museums, accessibility also encompasses the broader meanings of compliance and refers to how organizations make space for the characteristics that each person brings. The principles of Universal Design for Learning (UDL) are important accommodations to make in order to provide educational experiences that meet the needs of all learners, thereby maximizing accessibility (Rose, 2000). Inaccessibility is the result of design choices, and sometimes these choices are linked to decisions about who is worth designing for (or with). Hence, lack of accommodations may reflect a failure to anticipate the ways in which whole groups of people (e.g., people who use wheelchairs) might approach an experience or an event, or a willful decision that people who require accessibility accommodations are not numerous enough (or important enough) to merit consideration. The concept of accessible design ensures that both “direct access” (i.e., unassisted) and “indirect access” (compatibility with a person’s assistive technology, e.g., computer screen readers) are afforded.

Representation of Broadening Participation Within the Portfolio

Promotional materials for the SciAct Program refer to reaching “learners of all ages” and “being active in all 50 states.” Additionally, all SciAct projects include metrics for broadening participation with regard to diversity within their evaluation plans. However, the particulars of who is served

and how are defined by the awardees. Being explicit about key audiences is essential to designing programs that are truly responsive and evidence-based for engaging those audiences. Examining the materials provided for the 24 projects, the committee found that 3 awards include some information about intended demographics of participants in the project titles, and 16 awards make some mention of diverse learners or underserved or underrepresented communities within the background information for the projects. Of the latter, six project descriptions mention some aspect of particular identities, including Indigenous Alaskans, girls, Spanish-language speakers, people with disabilities, people of all socioeconomic backgrounds, and Native Appalachians. Some projects include partner organizations that represent the voices of the underrepresented audiences to be served, and some projects provide training in culturally relevant pedagogical approaches. Finally, some projects seek to educate the rest of the network on their area of expertise (e.g., how to serve learners of all ages effectively using the principles of UDL).

Looking across the portfolio, the committee found that the current SciAct projects collectively reach large groups of learners across different regions, abilities, age groups, socioeconomic levels, and race and ethnicities, among other facets of diversity. However, it is unclear to the committee whether or not the portfolio consistently and intentionally engages individuals who are representative of the entire U.S. population. Furthermore, it is unclear whether or not appropriate attention has been given to the inclusion, equity, and accessibility dimensions of broadening participation within all of the projects.

Across the SciAct Phase 1 portfolio, the committee also observed that broadening participation is largely an accountability mechanism. SciAct project evaluation plans often include goals to serve a diverse group of learners, or in particular learners from groups that are underrepresented in STEM, and milestones toward those goals are described in the project annual reports. By employing a metric crosswalk of the portfolio, the committee found evidence of commitments to reaching diverse populations with target metrics for participation by a general population of underrepresented learners or by those from a specifically defined group. Most commonly, the metrics reported denote the numbers of people from particular groups who were reached. The committee agrees that measuring the compositional diversity of key audiences reached across projects is important, because this metric can be used to demonstrate a commitment to promoting equity with respect to increasing opportunity to learn; however, as mentioned earlier, diversity targets do not sufficiently capture whether or not learning environments are inclusive. The committee believes more steps should be taken to ensure that activities within SciAct intentionally reflect this commitment (see the next section for a discussion of considerations for broadening participation that go beyond numbers of participants).

Considerations for Future Planning

In the committee’s view, the second phase of the SciAct Program affords an opportunity to develop a shared understanding of DEIA across the portfolio, drawing upon the existing expertise and experience of the awardees. This phase could also be used to build capacity for attending successfully to broadening participation throughout the projects in the portfolio. The current project evaluation reports regularly describe numbers of participants from different groups, but there are other considerations for broadening participation that could be part of Phase 2 program evaluation plans:

• To what extent does the SciAct portfolio map onto the full range of DEIA dimensions?
• How are the voices of the key audience(s) for a given activity involved in the governance, planning, and implementation of the activity?
• What is the purpose of the activity and how does it further the interests and needs of the intended audience?
• To what extent do participants have access to role models and examples that increase their own confidence within the STEM domain?
• To what extent does the activity address community-based goals and issues?
• How are project implementers supported to work within a culturally responsive educational context?
• To what extent are the products and activities developed with consideration for the principles of UDL?

Critical questions to be considered for each project will vary and will require qualitative approaches to evaluation, as well as the quantitative metrics approach already in use. Moreover, the topic of broadening participation should be part of the overall visioning process and evaluation strategy (discussed in Chapter 6).

NETWORKS

When SciAct stakeholders describe the portfolio, it is regularly characterized as a network. As mentioned earlier, SciAct regularly refers to the “network” element of the SciAct portfolio as one of the strengths of the current program model, and SciAct stakeholders express interest in continuing to support this feature of the work. In practice, networks can exist in many forms, with a wide range of goals and purposes. In this section, we describe some of the relevant research on networks, and then apply this research to

help us understand what kind of network SciAct currently represents, as well as what the program should consider if it wants to develop and expand its network capabilities in the future.

Principles to Consider for Building Networks

Intentionally designed networks are one strategy for collections of organizations and stakeholders to pursue a wide range of goals. In the field of education, networks may be used to support coherent policy implementation, school district change, professional learning, and development and dissemination of novel educational innovations, among other desired ends (Bryk et al., 2015; Coburn et al., 2013; Cooper, Slavin, and Madden, 1998; Culatta, 2012; Kania and Kramer, 2011; Lieberman and Wood, 2003; McLaughlin and O’Brien-Strain, 2008; Russell et al., 2015; Santo, 2017). Network development can be an intentional strategy to meet the goals held by a set of actors.

Given the number of organizational stakeholders involved in the SciAct portfolio, it is fair to characterize SciAct as an “interorganizational network.” Russell and colleagues (2015, p. 93) offer the following definition for a designed interorganizational network: “An arrangement of public and private organizations, agencies, and departments that have been explicitly constituted to facilitate collective action.” Such interorganizational networks seek to achieve outcomes that go beyond individual organizations or cross-organizational projects (Provan, Fish, and Sydow, 2007). An interorganizational network can pursue a range of goals, including the development of new knowledge and innovations through interorganizational coordination and design activities (Powell and Grodal, 2005; Santo, 2017; Tödtling and Trippl, 2005; Von Hippel, 2001), dissemination of innovations (Coburn et al., 2013; Rogers, 2010), sustaining innovations once they have spread (Coburn and Russell, 2008), and building capacity for problem identification within collectives (Hargadon, 2002; Santo et al., 2017).

Moreover, many networks are developed specifically to support the development of collective knowledge, innovation, and continuous improvement. These networks are known as “learning networks.” Coburn and colleagues (2013) summarize a number of structural features common to learning networks. First, learning networks must be able to incorporate new (and potentially valuable) knowledge into a coordinated network “core,” and this core must be set up to engage in sharing and inquiry that promotes innovation and improvement. Additionally, learning networks must include rules or norms that allow stakeholders to engage in actions that contribute to overall network learning and improvement.

In order to function as a learning network, the network needs a carefully designed and strategically implemented infrastructure (also referred to as the “backbone”) that coordinates network activities and supports

network goals (Kania and Kramer, 2011). A learning network infrastructure should include an intended structure (e.g., What do network stakeholders do and how they do it? What are shared expectations for members?); mechanisms for developing, promoting, and sustaining that structure (e.g., How will members communicate? How will they participate in the network?); and a governance model that facilitates implementation of those mechanisms (Provan, Fish, and Sydow, 2007). Common mechanisms for coordination within networks include

• communications channels (e.g. newsletters, blogs, listservs, online portals, “chats” on social media platforms),
• project and/or member directories,
• convening spaces (in-person or online),
• subgroup structures (e.g. working groups, committees, advisory groups),
• approaches for shared measurement, and
• knowledge bases that capture learning that is beneficial to the whole network (Coburn et al., 2013; Santo et al., 2016).

Developing a learning network’s infrastructure is a substantial effort, so it is important to understand why networks should take on the task. Without a clear rationale for why a network is building a collective learning infrastructure, it is unlikely that the network will succeed in its efforts. Table 5-1 lists various rationales for why networks would want to build an infrastructure, and then describes their projected impacts. Of course, these projected impacts are contingent on the nature of the design and implementation of various coordination mechanisms within the network.

The rationales and projected impacts outlined in Table 5-1 do not operate independently of one another. Participation in collective learning infrastructure can lead to greater clarity around collective goals and trust among stakeholders, which support alignment around shared goals, which provides the grounds for innovation and improvement and in turn supports network impact and sustainability. At the same time, collective improvement, innovation, and impact across the network supports clarity around network goals and deeper trust among actors. All of these outcomes can foster willingness of network stakeholders to participate in activities and efforts that constitute the network infrastructure.

The SciAct Portfolio as a Network

As noted in Chapter 3, the existing SciAct Program has elements that align with a number of network types. SciAct might be considered a network of networks in that a number of the awardees themselves represent,

TABLE 5-1 Rationales for Building Infrastructure to Support Collective Learning and Their Projected Impacts

lead, or partner with large networks of organizations and individuals across the country. Also, SciAct acts as a dissemination network: SciAct projects mobilize existing NASA Science Mission Directorate (SMD) assets, including subject matter experts, datasets, research findings, sensors, and platforms, so that they can reach broad audiences across the United States. SciAct also acts as an innovation network: Projects engage in substantive design and development work in order to transform NASA assets into tools that are useful and relevant for learners, learning environments, and education settings. By developing new approaches to teaching and learning in NASA-related STEM disciplines and circulating those approaches within the context of SciAct projects, SciAct is both innovating and disseminating.

The existing innovation and dissemination functions of the SciAct portfolio, however, do not speak to the degree of coordination among the many projects and whether the projects in the portfolio function as a learning network (see the preceding section for a description of the characteristics and value of a learning network). The committee found evidence of some level of cross-project coordination and collective learning, most notably in the annual principal investigators (PI) meeting. The PI meeting with all awardees is a 5-day convening that includes a number of activities intended to promote collective learning, collaboration, and capacity building, including:

• sharing ongoing work and approaches taken across projects,
• engaging outside experts in presentations and capacity-building workshops,
• providing opportunities for interest-driven conversations and topical sessions,
• communicating program accomplishments and future directions,
• providing opportunities for strategic planning around network-wide coordinated activities (e.g., 2017 solar eclipse and the 50th anniversary of the Apollo Moon Landing),
• disseminating updates and information from topical working groups meeting with the broader PI group.

Beyond the annual PI meeting, the committee found evidence of additional cross-project sharing and coordination within SciAct. First, SciAct convenes topical working groups where representatives from multiple projects explore issues of shared interest, including Girls in STEM, maker education, educational technology, and use of visualizations in educational programs. Secondly, projects have leveraged major events, such as the 2017 solar eclipse, to coordinate programming, share resources, and aggregate data on reach and impact. Lastly, project stakeholders are able to engage with one another in more open sharing using SciAct’s online Web portal, SMDEPO (described in Chapter 3). However, despite these mutually reinforcing activities, the committee did not find evidence that sufficient connections exist to allow for and incentivize optimal sharing across the network. Furthermore, while much of the substantive communication in the network happens at the annual PI meeting and in the topical working groups, it is not clear that all members of the teams are privy to all communications. These limited mechanisms for cross-project communication constrain how much all network stakeholders are able to benefit from the information sharing and capacity building. Finally, as mentioned above, SciAct could be considered a network of networks; however, under its current structure, there is untapped potential for leveraging the existing partnerships to tap into and activate the collective reach of the SciAct partners and awardees.

Considerations for Future Planning

As the NASA SciAct Program moves into Phase 2, there is an opportunity to support cross-portfolio innovation and improvement proactively through development of a learning network infrastructure. Currently, projects within the SciAct portfolio can be understood as constituting a dissemination and innovation network, insofar as students, teachers, families, life-long learners, and others are able to engage with the educational opportunities offered by the SciAct projects, and the projects are able to transform NASA’s assets into useful and relevant educational tools. However, the committee sees potential for SciAct to build on this existing network to become a true learning network. In order to do this, however, SciAct would need to develop mechanisms that would (1) facilitate active coordination across projects in order to build knowledge, (2) engage projects and project stakeholders in continuous improvement, and (3) surface challenges and opportunities that lead to deeper outcomes and more robust impacts.

Decisions around what kind of network mechanisms to implement, and at what point, should be contingent on overall network goals. As noted in Chapter 2, the existing SciAct Program goals likely need to become more specific if they are going to be used to guide substantive cross-project learning and improvement through a learning network approach. Until program goals become more targeted and specific, it may be difficult to determine just what kind of network approach and associated infrastructure are appropriate with regard to cross-network learning.

As the committee describes in previous sections of this chapter, building a learning network infrastructure requires a nontrivial investment of resources and capital. While the committee believes that a learning network is one way to share and develop best practices and build capacity, we note that SciAct may not need to devote a full suite of financial and personnel-based resources to achieve all its desired and stated ends. (For the committee’s suggestions on how SciAct should design and implement a program model aligned to its vision, see the final chapter of this report.) However, based on SciAct’s stated commitment to the network aspect of the portfolio’s work, the committee believes it is important to offer considerations for future planning, should the development of a learning network be a priority for Phase 2. Below, the committee offers three possible approaches to establishing collective learning infrastructure for consideration by NASA SciAct staff, and ideally by project stakeholders, as the program moves into Phase 2.

Participatory Knowledge-Building Groups

As evidenced by some topical working groups within the SciAct network, opportunities exist to build collective knowledge through more intentional development of topical collectives that deliberately engage in

surfacing, synthesizing, and sharing knowledge related to particular issues. Groups might be organized along a number of lines in terms of topics. For instance, one set of groups might be organized around content and pedagogy (e.g., teaching climate science, doing citizen science, creating inclusive and equitable learning environments) while others might be organized around process (e.g., effective approaches to creating scalable curricula, engaging in community-based partnerships, utilizing formative evaluation data). Groups would then be intentionally structured to surface knowledge from member projects that is applicable across the network (e.g., design principles, tip sheets, practice briefs, measurement repositories).

Developing, facilitating, and sustaining participation in such groups is, of course, a central challenge. One approach could involve cross-portfolio evaluation teams being responsible for these functions, another could involve the development of microgrants that support self-organization and staff time for participation among project members, or both. Most importantly, the knowledge-building process itself is as important as the direct outcomes or “content deliverables” of such groups. It is in the process of participating in knowledge development that understandings across project teams circulate, that trust is built, and that clarity and alignment across projects develop.

Strengthened Infrastructure for Cross-Network Communications

Effectively supporting learning in a network requires careful attention to communication mechanisms and routines. It can be worthwhile for the network to consider how its current communication infrastructure is operating and how it might be improved—for instance, looking at the nature and frequency of in-person cross-network convenings, communications channels (e.g., listservs, portals, project directories), and the possibility of regular, targeted SciAct virtual community calls that would elevate ongoing project approaches and challenges. In general, a key function of such communications infrastructure is to support network projects to “work in the open” (Santo et al., 2016). Open work, drawing on practices from the free/open source software community, includes practices that value transparency, collaboration, and sharing within communities of experimentation. Communications platforms can support practices of “open work,” including (1) public storytelling and context setting, (2) rapid prototyping “in the wild,” (3) community contribution, (4) public reflection and documentation, and (5) creation of remixable work products and data.

Shared Measurement Infrastructure

A final approach to be considered is the development of shared measurement approaches across the SciAct portfolio. This approach would need to be taken with careful consideration, and, given the complexity of

implementing any form of shared measurement across network projects, would require deep knowledge of the nature of projects in terms of approaches taken and where there are in fact shared goals that lend themselves to shared measurement. Shared measurement might be viable at a macro level across all projects. However, it is more likely that supporting improvement and innovation across such varied projects, as is the case for the SciAct portfolio, can be accomplished by identifying subgroups within the portfolio that are similar enough to benefit from shared measurement on inputs and outcomes, and also considering the stage of each project, given that outcomes may be a shifting target in the case of projects that are in the design phase. For more information on this approach, SciAct could consider drawing on models of networked improvement communities that center on the use of shared measurement in service of improving outcomes on highly specified problems of practice (Bryk et al., 2015).

SUMMARY

The committee supports SciAct’s continued efforts to broaden participation, and believes attention to making this commitment explicit is an important strategy for SciAct going forward. In the committee’s view, the second phase of the SciAct Program is an opportunity to continue its pursuit of diversity, equity, inclusion, and accessibility across the portfolio, and to build upon the existing expertise and experience of the awardees. This phase could also be used to build capacity for successfully attending to broadening participation throughout the network.

Similarly, the committee notes that while there are some limited existing opportunities for sharing, exploring shared interests, and coordinating across projects, SciAct has yet to formally build an infrastructure for supporting collective learning. Phase 2 is a potential opportunity to support the development of a learning network, if that is in fact a desired goal.

CONCLUSION 12: While broadening participation is a stated intention of Science Activation (SciAct), it is not clearly defined nor is there evidence that awardee activities have uniformly had an impact in this area. Integrating goals related to broadening participation throughout SciAct projects would require explicit assessment beyond counting the numbers of participants from various groups.

CONCLUSION 13: The projects that are part of the Science Activation portfolio use a variety of design strategies to translate the National Aeronautics and Space Administration’s assets—subject matter experts, media assets, scientific instruments, datasets, etc.—to support learning in science, technology, engineering, and mathematics. However, there

are limited mechanisms for gathering, synthesizing, and sharing these innovations across the portfolio or for learning from cases of success or failure.

CONCLUSION 14: The Science Activation Program is at an important inflection point in its history. The second phase of the program presents an opportunity for iterative improvement and refocusing on both the individual project level and the portfolio as a whole.

CONCLUSION 15: While continuing existing awards may allow for continuity and support an environment of collaboration and partnership among existing awardees, lack of competition or opportunities to fund new projects may stifle the evolution of the portfolio.

This page intentionally left blank.