Skip to main content

Currently Skimming:

3 Evaluation Framework
Pages 37-74

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 37...
... developed an initial evaluation framework that was honed and refined to improve its utility, clarity, and emphasis as the eight evaluations proceeded. The framework presented in this chapter differs slightly from the versions that were used by the evaluation committees. Insights gained in the evaluation of the eight NIOSH programs (see Chapter 4)
From page 38...
... The following major components of each NIOSH program were assessed by the evaluation committees: • Major occupational safety and health challenges in the program area. • Goals and objectives as defined by NIOSH.
From page 39...
... OSHA, MSHA, other federal agencies; -Recommendations, Pilot and/or NIOSH reports, publications, Transfer: market-ready programs; workshops, databases, -Translation of technologies, Employees, Congress; state conferences; research into training and employers, and local practice, products, education industry, agencies; -Training and education and technologies; programs, educators, standards materials and -Information guidance, regulators bodies; labor, demonstration programs, dissemination; regulations, who reduce or trade, and trained professionals; -Capacity building standards, prevent professional through technical trade and hazardous associations; -Tools and methods, best assistance major media exposures or technology practices, developmental (HHEs) , training, releases, conditions developers and technologies, licenses, and education websites manufacturers; patents other researchers; SH practitioners Conduct surveillance and evaluate intervention effectiveness FIGURE 3-1  The National Institute for Occupational Safety and Health logic model.
From page 40...
... , but the framework committee found more value in focusing on the responses to these outputs as intermediate outcomes in the evaluation. Some NIOSH programs are organized using a matrix management approach as they span several NIOSH divisions or laboratories.
From page 41...
... ponents of the logic model would be beneficial at the inception of the evaluation process both for NIOSH staff as they assemble the evidence packages and for evaluation committee members as they begin their assessment of the program. Drawing on the program logic model, the evaluation framework, and the evaluation committee members' expertise, the evaluation committees began by examining important inputs and external factors affecting the NIOSH research program's agenda.
From page 42...
... Because of the uniqueness of each NIOSH program, each evaluation committee determined the most reasonable way to apply the evaluation criteria. EVALUATION COMMITTEES The individual evaluation committees were formed in accordance with the rules of the National Academies that focus on ensuring a balanced committee.
From page 43...
... In addition to the information provided by the NIOSH program, the evaluation committees independently collected additional information as deemed necessary for the evaluation, such as the perspectives of external stakeholders, including the Occupational Safety and Health Administration (OSHA) , the Mine Safety and Health Administration (MSHA)
From page 44...
... o  Results of previous program reviews (e.g., annual review by NIOSH leadership team or external scientific program reviews) o External factors affecting the program • Interactions with external stakeholders and with other NIOSH programs: o  role of program research staff in NIOSH policy setting, OSHA and MSHA stan The dard setting, voluntary standard setting, and other government policy functions o Interactions and working relationships with other NIOSH programs o  Identification of other institutions and research programs with similar portfolios and an explanation of the relationship between NIOSH activities and those of other institutions o  Key partnerships with other government agencies, employers, labor, academic institutions, nonprofit organizations, and international organizations • Program inputs: o Production inputs (program resources)
From page 45...
... : o Intramural: ß Peer-reviewed publications, agency reports, alerts, and recommendations ß  Databases, websites, tools, and methods (including education and training materials) ß Technologies developed and patents ß Sponsored conferences and workshops o Extramural: ß Program announcements ß Requests for applications • Intermediate outcomes: o  Standards or guidelines issued by other agencies or organizations based in whole or in part on NIOSH research o  Adoption and use of control or personal protective technologies developed by NIOSH o  Evidence of industry, employer, or worker behavioral changes in response to re search outputs o  of NIOSH products by workers, industry, occupational health and safety profes Use sionals, healthcare providers, and others (including internationally)
From page 46...
... Informa tion regarding external factors was also sought from other NIOSH program and management staff, OSHA and MSHA staff, and from other external stakeholders. Additionally, each evaluation committee chose other approaches to assess external factors.
From page 47...
... • NIOSH resources are inadequate to tackle key questions. Evaluation of the impact of NIOSH research outputs on worker health and safety also required consideration of external factors that might impede or aid implementation, measurement, and so on.
From page 48...
... For purposes of the eight reviews al ready completed, the evaluation committees considered three general timeframes: 1970–1995 (pre-NORA period) , the period from the founding of NIOSH to the initiation of the National Occupational Research Agenda (NORA)
From page 49...
... The evaluation committees recognized that NIOSH research priorities are sometimes circumstantial (e.g., congressionally mandated) rather than based on NIOSH's assessment of the state of knowledge.
From page 50...
... Sources of qualitative inputs for NIOSH included the following: • Other NIOSH programs; • Federal advisory committees, such as the Board of Scientific Counselors, the Mine Safety and Health Research Advisory Committee, and the Na tional Advisory Committee on Occupational Safety and Health; • NORA research partners and stakeholders, NORA strategic research plans, and the NORA Liaison Committee and federal liaison committee recommendations; • Industry, labor, academe, professional associations, industry associations, and the Council of State and Territorial Epidemiologists; and • OSHA and MSHA strategic plans and other federal research agendas. Attention was given to how comprehensive the inputs have been and to what extent gaps in input have been identified and considered by the program being evaluated.
From page 51...
... for joint requests for applications or program announcements and to OSHA, MSHA, and DoD contracts as well as collaborations with other NIOSH programs. Using this evaluation model, assessment of inputs included the evaluation committee's consideration of the degree to which allocation of funding and personnel was commensurate with the resources needed to conduct the research and the extent to which funding for the relevant intramural research activity has been limited by lack of discretionary spending beyond salaries, such as travel, supplies, and external laboratory services.
From page 52...
... Activities are defined as the efforts of program staff, grantees, and contrac tors. Activities of the NIOSH program under review were divided into research and transfer activities.
From page 53...
... hazards Safety design and safety systems research Intervention research: Control technologies Engineering controls and alternatives Administrative controls Personal protective equipment Work organization Community participation Policy (e.g., alternative approaches to targeting inspections) Design for safety Emergency preparedness and disaster response Diffusion and dissemination research: Training effectiveness Information dissemination effectiveness Diffusion of technology Health services and other research: Access to occupational health care Infrastructure -- delivery of occupational health services Socioeconomic consequences of work-related injuries and illnesses Workers' compensation Technology transfer and other transfer activities: Information dissemination Training programs Marketing analysis Technical assistance
From page 54...
... Highly relevant information and technology transfer activities include plans for transfer to all appropriate worker populations, including those considered vulnerable. Training should be incorporated into the strategic goals of all research fields where appropriate.
From page 55...
... Assessment of the transfer activities included considerations of program planning, coherence, and impact. The evaluation committee also considered the incorporation of international research results into knowledge-transfer activities conducted by the NIOSH program for U.S.
From page 56...
... and citations • Collaboration with other government or academic researchers NIOSH reports: • Number of written reports and citations Sponsored conferences and workshops: • Number of sponsored conferences • Number of sponsored workshops • Description of conferences and workshops (title, date, sponsors, target audience, number of participants, and resulting products) Databases: • Number of major databases created by NIOSH staff • Number of major databases created by external researchers funded by NIOSH grants • Description of databases: o Title, objective (in one to four sentences)
From page 57...
... o Complete citation o Percentage of target audience that has used product (up to 10 years after release) o  to three examples of implementation in the field Up Miscellaneous: • Any other important program outputs
From page 58...
... The evaluation committees con sidered coauthorship while determining the importance of research by the NIOSH program to the broader research community. The NIOSH program was asked to provide information on all relevant outputs of the program under review that were produced during the chosen period.
From page 59...
... Intermediate outcomes in the NIOSH evaluations included the production of guidelines or regulations based wholly or partly on NIOSH research by those outside of NIOSH (products adopted as public policy or as policy or guidelines by private organizations or industry) ; contributions to training and education programs sponsored by other organizations; use of publications or other materials by workers, industry, and occupational safety and health professionals in the field; secondary dissemination of program activities and outputs through trade and mass media coverage; and citations of NIOSH research by industrial and academic scientists.
From page 60...
... Assessment of intermediate outcomes.  The evaluation committees were asked to provide a qualitative assessment of product development, usefulness, and impact with consideration given to the relative value of intermediate outcomes (the frame work committee recommended applying the well-accepted hierarchy-of-controls model)
From page 61...
... The evaluation committees described the realized or potential benefits of the NIOSH program. Assessing the causal relationship between NIOSH research and specific occupational safety and health outcomes was seen as a major challenge because NIOSH does not have direct responsibility or authority for implementing its research findings in the workplace.
From page 62...
... The evaluation committees were asked to consider the strengths and weak nesses of outcome data sources. Quantitative accident, injury, illness, and employ ment data and databases are subject to error and bias and should be used by the evaluation committees only for drawing inferences after critical evaluation and ex amination of available corroborating data.
From page 63...
... 4. To what degree is the NIOSH program or subprogram responsible for improvement in occupational safety or health?
From page 64...
... Assessment of other outcomes.  The evaluation committees were asked to con sider other outcomes, including beneficial changes that are expected to occur; social, economic, security, or environmental outcomes; and the impact that the program has had on international occupational safety and health. 8.  Determine Scores for Relevance and Impact and Provide the Rationale The evaluation committees assigned an integer score for the relevance of the research program to the improvement of occupational safety and health and another integer score for the impact of the program on such improvements.
From page 65...
... 3.  Contributions of the NIOSH program's research and transfer activities to improve ments in work-related practices.
From page 66...
... The scoring criteria focus on the evaluation committee's assess ment of whether the program appropriately set priorities among research needs as well as how engaged the program was in appropriate transfer activities to move research findings into the workplace. Since the evaluation of NIOSH programs in cluded assessment of research activities and knowledge transfer activities, both are considered in the final relevance score.
From page 67...
... The committee could say that the NIOSH program made "major contributions" if the improvements would not have occurred when they did without the program's efforts. The framework committee had some concern that the imposed scoring criteria for impact might be considered a promotion of the conventional occupationalhealth research paradigm that focuses on health-effects and technology research
From page 68...
... The evaluation committees considered whether some of those issues need to be addressed and considered as external factors that facilitate or limit applica tion of more traditional research findings. Given the rapidly changing nature of work and the workforce and the intractable problems in manufacturing, mining, and other fields, the evaluation committees were encouraged to think beyond the conventional paradigm.
From page 69...
... The evaluation committees reviewed the procedures that the NIOSH program has in place to identify needed research relevant to the NIOSH mission and reviewed the success that the NIOSH program has had in identifying and addressing research related to emerging issues. For example, the program should be involved in examining leading indicators from other federal agencies (e.g., EPA, Department of Labor, National Institute of Standards and Technology, NIH, DoD, and Department of Commerce)
From page 70...
... 10.  Prepare Report by Using the Template Provided as a Guide Consistency and comparability among evaluation committee report formats was desirable, but the framework committee recognized that each NIOSH research program is different and that each evaluation committee was independent. The outline provided in Box 3-9 flows from the framework committee's review of the generalized NIOSH logic model (Figure 3-1)
From page 71...
... A. Evaluation summary: should include a brief summary of the evaluation with respect to impact and relevance, scores for impact and relevance, and summary statements B. Strategic goals and objectives: should describe assessment of the extent to which program strategic plans reflect program relevance C. Assessment of inputs: should describe adequacy of inputs to achieve goals D. Assessment of activities: should describe assessment of the relevance of the activities E. Assessment of research program outputs: should describe assessment of rel evance and potential usefulness of the research program's outputs F. Assessment of intermediate outcomes and causal impact: should describe as sessment of the intermediate outcomes and the program's contribution to them; should include the likely impacts and recent outcomes in the assessment G. Assessment of end outcomes: should describe the end outcomes related to health and safety and provide an assessment of the type and degree of attribution to the NIOSH program H. Assessment of other outcomes: should discuss health and safety impacts that are expected to occur; beneficial social, economic, and environmental outcomes; and international dimensions and outcomes I. Summary of ratings and rationale III. NIOSH Targeting of New Research and Identification of Emerging Issues (Charge 2)
From page 72...
... No. 6, Reviews of Research Programs of the National Institute for Occupational Safety and Health.
From page 73...
... 2009. Demonstrating and communicating research impact: Preparing NIOSH programs for external review.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.