Skip to main content

Currently Skimming:

Evaluating Outcomes and Generating New Knowledge
Pages 31-40

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 31...
... In this chapter, we describe the complex nature of evaluating the outcomes of out-of-school programs, and what can be done to provide a clearer picture of what programs work best under what circumstances for whom, and how the programs fit into the larger STEM learning ecosystem. The chapter provides a framework to guide evaluation efforts.p The Role of Evaluation Evaluation has many purposes, including for continuous improvement, accountability, informing management, and demonstration of value.
From page 32...
... Consequently, evaluations of education pro grams typically focus on individual learning assessments, where learning is defined in terms of gains in specific knowledge or skills.q How these out comes are measured depends to a considerable degree on how a program's designers have defined learning outcomes and the factors affecting them. How do young people learn STEM?
From page 33...
... Accumulation of Experiences The success of out-of-school STEM programs depends on the possibilities they create for young people to expand, deepen, and reinforce their cumulative STEM experiences. Since a wide array of activities, people, programs, material resources, and facilitators sustain engagement, the accumulation of learning opportunities usually accounts for development of expertise and interest (though, occasionally, one powerful experience is transformative)
From page 34...
... It is also essential that evaluators understand the cultural patterns of social discourse of participating communities so that evaluations accurately capture a program's effects. Additional Evaluation Challenges There are many additional challenges to evaluating STEM learning in out-of-school programs.
From page 35...
... A Three-Level Approach to Evaluating the STEM Learning Ecosystem From an ecosystem perspective on learning, a comprehensive out-of-school STEM program evaluation includes measurement at three interrelated levels: individual, program, and community. At the individual level, evaluation of the quality of an out-of-school program would include measures of an individual's intellectual development in STEM; positive STEM identity and dispositional development; and expansion of an individual's horizons (awareness, connections, and choices)
From page 36...
... ; the ways in which a given program is synergistic with the resources within a community and across settings; and the ways in which a program affects the community by expand ing learning opportunities and brokering additional engagement in STEM learning across differ ent community settings. Community-level indicators signal the extent to which community-level resources are in place to support effective out-of-school STEM programming, to support connec tions among in-school and out-of-school learning, and to identify any need for action.
From page 37...
... Community-Level Evaluation To better support the development and coordi nation of its ecosystem of STEM learning opportunities, the Mozilla Hive NYC Learning Network works with Hive NYC partners and the Hive Research Lab to capture and share best practices and collective wisdom. For example, in 2014, Hive NYC members and stakeholders convened meetings to develop principles and guidelines for "working open" -- a model for reflective, evaluative practice to support the continuous improvement of programs and outcomes at the community level.
From page 38...
... , and comparative evaluation interact. Existing measurest and program evaluationsu are typically con ducted at the individual or program level and typically focus on the short-term outcomes.
From page 39...
... Some researchers have pursued the development of standard metrics for measuring STEM interest and motivation across the range of out-of-school STEM environments.118 Others use qualitative means to probe and document the way that out-of-school experiences shape young people's life trajectories, as evidenced by choices, pathways, and "ways of being" -- for example, interacting with phenomena or appraising ideas, designs, and products.119 The most common approaches to research and evaluation focus on near-term measures that are easy to administer and score. Well-designed tools of this kind are an important component of an evaluation toolkit, and there are several ambitious initiatives under way to develop suites of tools that can be shared across projects: • the Youth, Engagement, Attitudes and Knowledge (YEAK)
From page 40...
... However, there are significant concerns about the ways that common measurements may constrict educational opportunities and approaches in schools and otherwise negatively affect learning in out-of-school settings. Clearly there are benefits and limitations of common metrics, and this is an area of work that deserves careful investment and study over time.121 Common approaches to measurement of youth outcomes are generally meant to document the contributions of out-of-school programs to STEM learning122 or to determine whether the contri butions to STEM learning vary for different populations of young people.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.