NOAA's Education Program Review and Critique (2010) / Chapter Skim
Currently Skimming:
5 Current Evaluation Framework and Existing Evaluation Efforts
5 Current Evaluation Framework and Existing Evaluation Efforts
Pages 111-132
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 111... ...
Evaluation is more prominent in the 2009-2029 education strategic plan than it was in the 2004 plan. Evaluation is primarily the responsibility of each individual education program, and the Education Council (EC)
|
From page 112... ...
Therefore, rather than simply reviewing past evaluations, this analysis takes a forward-looking perspective, identifying the opportunities and challenges of linking existing formative and summative evaluation practice with the new strategies and goals set for NOAA's education initiatives. We begin our review by examining the quality of individual program evaluations conducted in recent years.
|
From page 113... ...
Second, in 2008 the EC led an agencywide initiative to develop a new strategic plan to guide the organization, coordination, implementation, and evaluation of the many education programs housed in NOAA. CURRENT AND PAST PRACTICES IN PROGRAM EVALUATION In this section we review the state of evaluation for education and outreach programs, based on evaluations that NOAA provided.
|
From page 114... ...
Nancy Foster Scholarship Program No evaluation available Sea Grant Evaluation: local Other Initiatives and Interagency Collaborations Climate Communication and Education Program No evaluation available National Weather Service No evaluation available Ocean Hall/Ocean Kiosk No evaluation available Ocean Science Bowl Evaluation: programwide NOTE: An indication of "no evaluation available" should not be read as an absence of evaluation activity, but rather there was no evidence for making an assessment of the current practice. participants.
|
From page 115... ...
Although this suggests some room for improvement across programs, the norm for NOAA programs is to have acceptable levels of transparency in evaluation questions. In each quadrant of Table 5.2 the questions are aimed at different stakeholder communities important to the NOAA education mission.
|
From page 116... ...
This assessment offers some optimism that NOAA education programs will adapt to the new mission and develop evaluation questions to serve the needs of the strategic goals for environmental literacy and workforce development. The adoption of the TOP model, described in detail below, requires that evaluations now be guided by a different set of evaluation questions that align with the need to address the 2008 strategic education plan goals.
|
From page 117... ...
The 2007 evaluation is also notable because it is the most rigorous evaluation design employed among the NOAA evaluation programs. During the development of this report pressure was growing in the federal government to incorporate more rigorous designs into all program evaluations.
|
From page 118... ...
In public documents NOAA has touted the B-WET evaluation as an example of the responsiveness of the agency to the national policy initiatives emphasizing rigorous evaluation. Sea Grant Program Sea Grant, the oldest initiative in the EC, submitted 40 reports for our review of evaluation practices, but most were either assessments of a state's Sea Grant Program (with a description of what education activities occur and who is served)
|
From page 119... ...
Thus, the evaluations of Sea Grant include all research and extension-related activities, which are beyond the purview of this report. Sea Grant leadership indicates that the most common use of evaluations is as a performance management tool that provides Sea Grant program officers with up-to-date information on
|
From page 120... ...
Sea Grant leadership reports the following challenges associated with the current evaluation process after two rounds of program assessments. First, program assessments are broadly focused on the entire research program.
|
From page 121... ...
For example, practices that are worthy of replication include recruiting a comparison group when useful and appropriate; creating an information infrastructure to integrate planning, implementation, and evaluation; collecting information from multiple sources; aligning evaluation questions with program goals; collecting both quantitative and qualitative data; and using literature reviews in early stages to understand best practices in program design and interpret program implementation findings. Until NOAA articulates measurable goals and outcomes for its education programs, it will be difficult to design evaluation questions that align goals and outcomes or produce any summative results at the highest level.
|
From page 122... ...
The 2008 strategic goals pose several challenges to existing evaluation practices. Perhaps the most significant challenge is establishing evaluation processes through which NOAA can assess the cumulative impact of education programs toward achieving the strategic national goals.
|
From page 123... ...
• Barriers to having common cost-benefit standards for determining the effectiveness of programs. TABLE 5.3 Common Logic Model for NOAA Instructional Programs Logic Model Elements Corresponding NOAA components Inputs Educational goals in a research agency provide guidance for: Educational management that creates and administers: Activities Instructional activities directed at: An audience (or audience clusters)
|
From page 124... ...
Collaborations between education staff and scientists and engineers have the potential to lead to higher quality education programs and resources than would be possible without such collaborations. Thus, just as the contributions of NOAA education staff
|
From page 125... ...
As in the challenge posed by partnerships, there is a significant gap between current evaluation practice and the goal of using common performance metrics. NOAA is making important investments in conducting the baseline research for constructing such metrics, as evidenced in the sponsorship of the National Assessment of Environmental Literacy conducted by the North American Association for Environmental Education (see McBeth et al., 2008)
|
From page 126... ...
TOP documentation presents itself as based on a hierarchy, integrating program evaluation in the program development process using a simple framework to target specific outcomes in program development to existing resources and outputs, and then to assessing the degree to which the outcome targets are reached. As with most evaluation frameworks, the TOP model does not provide guidance on specific methods or metrics for implementing individual evaluations.
|
From page 127... ...
* KASA Reactions Reactions Participation Participation Activities Activities Resources Resources FIGURE 5.1 Bennett TOP model.
|
From page 128... ...
Current evaluation standards from the Joint Committee on Standards for Educational Evaluation (1994) 1 include 1An updated version of the Standards for Educational Evaluation Program will be released in 2010.
|
From page 129... ...
In addition to the standards, Daniel Stufflebeam, the first chair of the Joint Committee on Standards for Educational Evaluation, recently provided an assessment of evaluation models using the 30 standards (Stufflebeam and Shinkfield, 2007)
|
From page 130... ...
Brackett notes that many of the evaluation reports submitted were conducted either prior to or coinciding with the adoption of TOP model by the EC and the issuance of any guidance to the individual programs. Although only a small proportion of education initiatives have been evaluated, "17 of the 18 [evaluation]
|
From page 131... ...
SUMMARY Evaluation of federally funded education programs is evolving rapidly, and at the same time the expectations of NOAA programs have changed quickly. The agency has responded and in some cases has done exemplary work.
|
From page 132... ...
• Evaluating both the education programs and the line offices on how effectively the ideas, insights, knowledge, understanding, and passion of the agency's scientists and engineers, as well as other scientists and engineers in the relevant disciplines, are incorporated into educational materials and programs. • Evaluating the appropriateness and effects of their partnerships.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.