Evaluating Occupational Health and Safety Research Programs Framework and Next Steps (2009) / Chapter Skim
Currently Skimming:
2 The Program Evaluation Context
2 The Program Evaluation Context
Pages 25-36
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 25... ...
At the conclusion of some of the overview sections throughout the chapter, the committee's findings specific to the evaluation process used by the framework and evaluation committees are included in bold and italicized text. PROGRAM EVALUATION Although formal program evaluations, especially of educational programs, preceded World War I, the profession as currently practiced in the United States 1This chapter draws on background papers commissioned by the committee from Sonia Gatchair, Georgia Institute of Technology, and Monica Gaughan, University of Georgia.
|
From page 26... ...
Evaluation is now an established profes sional practice, reflected through organizations such as the American Evaluation Association and the European Evaluation Society (AEA, 2009; EES, 2009)
|
From page 27... ...
Prior to the work of the evaluation committees, NIOSH contracted with RAND Corporation to provide operational and analytical assistance with compiling the evidence packages for the reviews and developing the logic models; a detailed description of that effort can be found in a recent RAND report (Williams et al., 2009)
|
From page 28... ...
In ad dition, ongoing dialogues with external stakeholders shape research activities and spread research knowledge in ways that are hard to track. Program evaluations that solely rely on the logic model almost inevitably miss information on some of the nonlinear effects of program activities.
|
From page 29... ...
Expert panel review is the "bread-and-butter" approach worldwide, but there is also a long track record of evaluation studies, in which external consultants gather and analyze primary data to inform the expert deliberations. Within the range of evaluation approaches for research programs, the National Academies' evaluations of NIOSH programs clearly fall among expert panel evaluations, rather than evaluation studies.
|
From page 30... ...
Although expert panel reviews work to balance conflicting values, objectives, or viewpoints, they also may lead to tensions in the same areas they are expected to resolve. As noted above, the review process may be broadened to include other stakeholders beyond "experts" or "peers." Expert panels usually operate with an evaluation protocol developed by an outside group, including evaluation proce dures, questions to be answered, and evaluation criteria (e.g., the evaluation of the Sea Grant College Program, Box 2-1)
|
From page 31... ...
Rating scales capture qualitative judgments on ordinal scales and allow for descriptions of performance at the various levels. Characteristics that are sought in expert panel reviews include a panel with a balanced set of expertise and credibility among various stakeholder groups and independence and avoidance of conflict of interest among panel members to the extent possible.
|
From page 32... ...
The National Academies committees follow a thorough bias and conflict of-interest process that includes completion of disclosure forms and the bias and conflict-of-interest discussion held at the first meeting. Other Methods of Evaluating Research Programs Other types of evaluations generally involve hiring consultants to provide analyses of specific outputs of the program.
|
From page 33... ...
The National Academies' evaluation of NIOSH research programs used one of the most common approaches: expert panel review. As is common in evaluations of applied research programs, this process involved stakeholders as members of the evaluation committees and also sought external stakeholder input.
|
From page 34... ...
The process did not, however, expand into an evaluation study by gathering new data or extensively analyzing external data sources. The evaluations of NIOSH programs fall well within the range of accept able practice in evaluating research programs and are compiled in comprehensive reports that went through peer review under the National Academies' report review process.
|
From page 35... ...
1994. A review of the NOAA National Sea Grant College Program.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.