Evaluating Occupational Health and Safety Research Programs Framework and Next Steps (2009) / Chapter Skim
Currently Skimming:
4 Improving the Evaluation Process
4 Improving the Evaluation Process
Pages 75-92
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 75... ...
Those recommendations and the accompanying detailed evaluations of each program are The chapter draws on the November 24, 2008, workshop organized by the framework committee and held at the National Academies, "Evaluating NIOSH Programs: Lessons Learned and Next Steps." Workshop participants included NIOSH program and senior staff, members of the NIOSH Board on Scientific Counselors, evaluation committee members, framework committee members, and National Academies' staff. Findings also reflect the experiences relayed throughout the eight studies by evaluation committee members and NIOSH staff.
|
From page 76... ...
Evaluation and framework committee members give NIOSH a great deal of credit for holding their research programs accountable for real out comes that affect life and health. Evaluations of research supported by other federal agencies often focus on output productivity and intermediate outcomes and do not hold the agency accountable for real-world impacts.
|
From page 77... ...
The framework committee views reports from external stakeholders as the most accessible source of this information. In future evaluations, the questions posed by evaluation committees to relevant stakeholders could include this larger set of intermediate outcomes.
|
From page 78... ...
How ever, during the November workshop, NIOSH staff reported that the prospect of receiving impact and relevance scores had a motivational effect, in general, on the NIOSH program staff who were assembling the evidence packages and responding to additional requests for information from the evaluation committees. Throughout the evolution of the framework, the scoring criteria have been clarified.
|
From page 79... ...
In revisiting the scoring issues at the end of the eight evaluations, the framework committee decided to stay with the integer approach using a 5-point scale because it offers greater opportunity for consistency. Rating Quality During the workshop, participants discussed the question of whether a specific numerical rating for quality should be added to the task of future NIOSH program evaluations.
|
From page 80... ...
Some evaluation committees adhered rather closely to the framework guidelines and structure, and others used it more as a starting point and reference document. Some evaluation committees thought the structure and format presented by the framework for evaluation was rather rigid, while others found that the framework provided committee members with the latitude needed to individualize the evalu ation to fit a specific NIOSH program.
|
From page 81... ...
Greater emphasis on self-assessment prior to external evaluation could provide valuable information and facilitate a greater understanding of each program. Evidence Packages The term evidence package refers to the collection of information provided by NIOSH staff to each evaluation committee.
|
From page 82... ...
Several NIOSH evidence packages included such a summary that was helpful to the committee. Because most evaluation committees divided into working groups that focused on specific subgoals, organizing the evidence package in a similar manner would be particularly useful.
|
From page 83... ...
Quantity and Quality of the Materials As noted above, the evaluation committees found a great deal of useful information in the evidence packages. During the November workshop, breakout session participants discussed the need for NIOSH staff to be more consistent in categorizing a program's efforts according to the components of the logic model, particularly in differentiating activities and intermediate outcomes.
|
From page 84... ...
. Research Prioritizing and Planning Process Evaluation committee members expressed the need for evidence packages to include more complete discussions of the prioritization of research agendas and the planning processes used by the NIOSH program to establish priorities.
|
From page 85... ...
This was particularly helpful for the latter evaluations as NIOSH staff became more aware of the critical materials and the best ways of organizing the data. As evaluation becomes an integral part of the ongoing work process at NIOSH, the framework committee hopes that systematic data collection and perhaps online tools will be available to collect and analyze data for future evaluations and their evidence packages.
|
From page 86... ...
However, given the distributed nature of some programs with many locations and no one central facility, it was not practical for some evaluation com mittees to conduct a site visit. The site visits highlighted the importance to the evaluation committees of in formal interactions with NIOSH staff members including opportunities to observe program staff in their work environments, ask questions, and discuss issues outside of a formal presentation setting.
|
From page 87... ...
Some evaluation committees provided the opportunity for stakeholders to submit input anonymously through online questionnaires. Discussions on stakeholder input during the November 2008 workshop indicated that although all agreed that meeting time devoted to stakeholder input was quite valuable, workshop participants had a mixed reaction regarding the value of seeking comments through a website or online mailing.
|
From page 88... ...
The framework committee noted that the evaluation framework provides a suggested table of contents for evaluation reports, but leaves decisions on the organization and format to the discretion of the individual evaluation commit tees. The framework committee believes any efforts to further prescribe the report format might jeopardize the creativity and focus of the evaluation committees.
|
From page 89... ...
Although NIOSH staff appreciated the confidence in the programs implied by these recommendations, they also expressed interest in receiving more guidance from the evaluation committees on where and how to set priorities, particularly if faced with static or declining resources. A suggestion was made that future evaluation committees provide recommendations from both perspectives (increases in resources and flat or declining resources)
|
From page 90... ...
Increased opportunities, particularly informal opportunities, for NIOSH staff to discuss issues with evaluation committees would be helpful, as would increased attention by evaluation committees to some of the more indirect measures of intermediate outcomes. This chapter has addressed both the framework and its application.
|
From page 91... ...
2009. Demonstrating and communicating research impact: Preparing NIOSH programs for external review.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.