Respiratory Diseases Research at NIOSH Reviews of Research Programs of the National Institute for Occupational Safety and Health (2008) / Chapter Skim
Currently Skimming:
Appendix A: Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health
Appendix A: Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health
Pages 173-216
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 173... ...
It is a working document that will be subject to change by the Framework Committee aimed at improving its relevance on the basis of responses received from evaluation committee members, NIOSH, stakeholders, and the general public before and during the course of the assessments conducted by independent evaluation committees of up to 15 research programs and healthoutcomes programs.
|
From page 174... ...
174 R e s p i r ato ry D i s e a s e s R e s e a r c h at NIOSH All public comments submitted to the Committee for the Review of NIOSH Research Programs will be included in the Public Access File for this study as pro vided in the National Academies Terms of Use (www.nationalacademies.org/legal/ terms.html)
|
From page 175... ...
IV. Evaluation Committee Report Template V. Framework Committee Final Report Figure 1 The NIOSH operational plan presented as a logic model Figure 2 Flow chart for the evaluation of the NIOSH research program
|
From page 176... ...
176 R e s p i r ato ry D i s e a s e s R e s e a r c h at NIOSH Table 1 NORA High-Priority Research Areas by Category Table 2 Examples of NIOSH Program Research and Transfer Activities Table 3 Examples of a Variety of Scientific Information Outputs Table 4 Evaluation Committee Worksheet to Assess Research Programs and Subprograms
|
From page 177... ...
A pp e n d i x A 177 ACRONYMS ABLES Adult Blood Lead Epidemiology and Surveillance ACOEM American College of Occupational and Environmental Medicine AOEC Association of Occupational and Environmental Clinics BLS Bureau of labor Statistics CDC Centers for Disease Control and Prevention EC Evaluation Committee FACE Fatality Assessment Control and Evaluation FC Framework Committee HHE Health Hazard Evaluations MSHA Mine Safety and Health Administration NEISS National Electronic Injury Surveillance System NIOSH National Institute for Occupational Safety and Health NORA National Occupational Research Agenda NORA1 National Occupational Research Agenda 1996-2005 NORA2 National Occupational Research Agenda 2005-forward OSHA Occupational Safety and Health Administration OSHAct Occupational Safety and Health Act of 1970 OSH Review Occupational Safety and Health Review Commission Commission PART Performance Assessment Rating Tool PEL Permissible Exposure Limits SENSOR Sentinel Event Notification System of Occupational Risks TMT Tools, Methods, or Technologies
|
From page 178... ...
Evaluation committee assessment of progress in reducing workplace illnesses and injuries facilitated by occupational safety and health research through (a) an analysis of relevant data about workplace illnesses and injuries for the program activity, and (b)
|
From page 179... ...
Those three charges constitute the scope of work of the individually appointed, independent ECs formed by the National Academies. I.A. NIOSH Strategic Goals and Operational Plan As a prelude to understanding the NIOSH strategic goals and operational plan, NIOSH research efforts should be understood in the context of the Occupational Safety and Health Act (OSHAct)
|
From page 180... ...
In 1994, NIOSH embarked on a national partnership effort to identify research priorities to guide occupational health and safety research for the next decade. The National Occupational Research Agenda (NORA)
|
From page 181... ...
NIOSH will provide similar logic models relevant to each research program evaluated by an EC. I.B. Information from Other Evaluations The FC is aware that several NIOSH programs have already been subjected to evaluation by internal and external bodies.
|
From page 182... ...
, training science stakeholder inputs, extramural, including releases technologies, licenses, and education community, SH surveillance and domestic and patents practitioners A-1.eps intervention international efforts, effectiveness data, such as work earmarks, conducted at ERCs, risk assessments ARCs and WHO Global Network of Collaborating Centers Transfe r Prevention through effective research, transfer, and evaluation External Factors: Economic and social conditions and regulatory environment FIGURE 1 The NIOSH operational plan presented as a logic model
|
From page 183... ...
♦ Process to solicit and approve intramural research proposals. PART focuses on assessing program-level performance and is one of the measures of success for the Budget and Performance Integration initiative of the president's management agenda (see CDC Occupational Safety and Health at http://www.whitehouse.gov/omb/budget/fy2006/pma/hhs.pdf)
|
From page 184... ...
♦ Interventions, recommendations, and information-dissemination and technology-transfer activities designed to get research findings used to improve occupational safety and health. ♦ Outcomes of research, alerts, standard-setting, investigations, and consultations; for example -- documented reductions in risk after program-supported interventions, employer and industry behav ior changes made in response to research outputs, and worker behavior changes in response to research outputs.
|
From page 185... ...
. • Intermediate outcomes (responses by NIOSH stakeholders to NIOSH products, such as public or private policy change, training and educa tion in the form of workshop or seminar attendance, self-reported use or repackaging of NIOSH data by intermediary stakeholders, adop tion of technologies developed by NIOSH, implemented guidelines, licenses, and reduction of workplace hazardous exposures and other risk factors)
|
From page 186... ...
186 External Factors A B Analysis of Review and C Major Program Strategic Goals and Assessment Review and Area Challenges Objectives Driving of Inputs Assessment Determined by EC Current Program of Activities Planning: surveillance and intervention data; Surveillance, Independent assessment Assessment of NIOSH stakeholder inputs by EC members to process to select health effect research, compare with NIOSH program goals, intervention research, Production: intra- and technology transfer program area goals evaluation of goals extramural funding, activities, health services selected by NIOSH, staffing, physical facilities, and other research comparison with EC management structure assessment of challenges A-2.eps E F Review and D Review and Assessment of Review and Assessment of Intermediate Assessment End Outcomes Outcomes of Outputs Reduced injuries, Public policy impact, Publications, reports, illnesses, risk factors training/education, databases, tools, in the workplace self-reported use and/or methods, guidelines, repackaging by stakeholders, recommendations, implemented guidelines licences and patents External Factors FIGURE 2 Flow chart for the evaluation of the NIOSH research program
|
From page 187... ...
. For purposes of this review, the results of inputs and external factors are the program research activities, outputs, and associated transfer activities that may result in intermediate outcomes and possibly eventual end outcomes.
|
From page 188... ...
10. Identify significant emerging research areas (Section III.C)
|
From page 189... ...
A relevant research program is more than a set of unrelated research projects; it is an integrated program involving an interrelated set of surveillance, research, and transfer activities. • Appropriate consideration by NIOSH of stakeholder inputs.
|
From page 190... ...
External factors may best be assessed through the expert judgment of EC members regarding the knowledge base, the research program, and implementation of interventions as these relate to the needs in the occupational health or safety area targeted by the research program. The ECs, however, may choose additional approaches to assess external factors.
|
From page 191... ...
Evaluation of the impact of NIOSH research outputs on outcomes may require consideration of external factors that might have impeded or aided implementation, measurement, and so on. For example, evaluators might consider whether • Regulatory end points are unachievable because of obstacles to regu lation or differing priorities of the regulatory agencies.
|
From page 192... ...
. It will be important for the ECs to get a general sense of the history of the NIOSH research program and its impact, but their efforts should be focused on the impact and relevance of NIOSH programs from 1996 on.
|
From page 193... ...
Research program relevant evaluations that should be requested include the NIOSH annual program review by the Leadership Team; the NORA research program proposal pre-award external review, NORA post-award program external review, and external scientific program review.
|
From page 194... ...
Production inputs include intramural and extramural funding, staffing, management structure, and physical facilities. Inputs for program evaluation include existing intramural and extramural information and, potentially, surveys or case studies that might have been devel oped specifically to assess progress in reducing workplace illnesses and injuries and to provide information relevant to targeting research appropriately to future
|
From page 195... ...
Planning Inputs Planning inputs can be qualitative or quantitative. Sources of qualitative inputs include • Federal Advisory Committee Act panels (Board of Scientific Counsel ors, Mine Safety and Health Research Advisory Committee, National Advisory Committee on Occupational Safety and Health, and so on)
|
From page 196... ...
Production Inputs For each research program under review, NIOSH should specify an identifiable portion of the NIOSH intramural budget, staff, facilities, and management that has been allocated by divisions and offices that play a major role in the research program. Production inputs should be described primarily in terms of intramural research projects and staff, relevant extramural projects (particularly cooperative agreements and contracts)
|
From page 197... ...
. Transfer activities include information dissemination, training, technical assistance, and education designed to translate research outputs into content and formats designed for application in the workplace to produce improvements in occupational safety and health.
|
From page 198... ...
Diffusion and dissemination research Training effectiveness Information-dissemination effectiveness Diffusion of technology Health-services and other research Access to occupational health care Infrastructure research -- delivery of occupational-health services, including international health and safety Socioeconomic consequences of work-related injuries and illnesses Worker compensation Technology-transfer and other transfer activities Information dissemination Training programs The ECs should review the list of research and transfer activities (projects) for the research program under review that have beentransfer activities (projects)
|
From page 199... ...
Questions to Guide the Evaluation Committee in Assessing Transfer Activities 1. Is there a coherent planned program of transfer activities?
|
From page 200... ...
Assessment of the transfer activities must include considerations of program planning, coherence, quality, and impact. III.B.6. Review of Outputs (Box D in Figure 2)
|
From page 201... ...
Peer-reviewed publications by external researchers funded by NIOSH Total number of NIOSH-funded original research articles by external researchers Total number of NIOSH-funded review articles by external researchers (including best-practices articles) Complete citation for each written report Complete copies of the "top five" articles Collaboration with other government or academic researchers NIOSH reports in the research program Total number of written reports Complete citation for each written report Complete copies of the "top five" reports Sponsored conferences and workshops Total number of sponsored conferences Total number of sponsored workshops For each sponsored conference or workshop, describe: Title, date, and location Partial vs complete sponsorship (if partial, who were cosponsors?
|
From page 202... ...
Complete citation Percentage of target audience that has used product 1, 5, and 10 years later Up to three examples of implementation in the field Identification of "top five" patents to date Miscellaneous Any other important program outputs The EC should ask NIOSH to provide information on all relevant outputs for the specific program for the chosen time period. Questions to Guide the Evaluation Committee 1.
|
From page 203... ...
; attendance at training and education programs sponsored by other organizations; use of publications by workers, industry, and occupational safety and health professionals in the field; and citations of NIOSH research by industrial and academic scientists. More difficult-to-collect intermediate outcomes that may be valid indicators of quality or utility include self-report measures by users and relevant nonusers of NIOSH outputs.
|
From page 204... ...
4. Has the program resulted in new personal protective equipment that is feasible for use or has been adopted in the workplace to reduce risk factors or exposures?
|
From page 205... ...
End outcomes are defined by measures of health and safety and of impact on process and programs. The FC recognizes that a major challenge in assessing the causal relationship between NIOSH research and specific occupational health and safety outcomes is that NIOSH does not have direct responsibility or authority for implementing its research findings in the workplace.
|
From page 206... ...
. • NIOSH intramural surveillance systems, such as the National Elec tronic Injury Surveillance System (NEISS)
|
From page 207... ...
4. To what degree has the NIOSH program or subprogram been respon sible for improvements in occupational safety or health?
|
From page 208... ...
Assessment Evaluation by the EC may consist of a discussion of other outcomes, includ ing positive changes that have not yet occurred; other social, economic, security, or environmental outcomes; and the impact that NIOSH has had on international occupational safety and health. It might also consider the incorporation of in ternational research results into the NIOSH program of knowledge transfer for industry sectors.
|
From page 209... ...
and evidence that external factors affected reduction Subprogram Activity Addressing Charge 1 Category Program 1 … … n 1.3 Contributions of NIOSH research Research and transfer activities to changes in work-related practices Transfer 1.4 Contributions of NIOSH research Research and transfer activities to reductions in workplace exposure, Transfer illness, or injuries 1.5 Evidence of external factors Research preventing application of NIOSH research results Transfer 1.6 Contribution of NIOSH research Research to enhancement of capacity in government or other research Transfer institutions 1.7 Contributions of NIOSH research Research to productivity, security, or environmental quality (beneficial Transfer side effects) Addressing Charge 2 2.1 Relevance of current and recently Research completed research and transfer activities to future improvements Transfer in workplace safety and health 2.2 Progress in targeting research to Research areas of study most relevant to future improvements in Transfer occupational safety and health
|
From page 210... ...
and charge 2 (relevance) , the ECs will use their expert judgment, their responses to the questions in Table 4, and any other appropriate information to arrive at one overall rating for the impact of the research program and one for its relevance to the improvement of occupational safety and health.
|
From page 211... ...
needed to effect change. Clearly, not all intermediate outcomes occur in the workplace.
|
From page 212... ...
1 = Research in the research program is an ad hoc collection of projects, is not integrated into a program, and is not likely to improve workplace safety or health. III.C. Identifying Significant Emerging Research (Addressing Charge 3)
|
From page 213... ...
Questions to Guide the Evaluation Committee 1. What information does NIOSH review to identify emerging research needs?
|
From page 214... ...
IV. EVALUATION COMMITTEE REPORT TEMPLATE The following outline flows from the FC's review of the generalized logic model prepared by NIOSH, the request for information from NIOSH programs, and the assessment model described earlier in this report. I. Introduction: This section should be a brief descriptive summary of the history of the program (and subprograms)
|
From page 215... ...
: The EC should assess the progress that the NIOSH program has made in targeting new research in the fields of occupational safety and health. There should be a discussion of the assessment process and results.
|
From page 216... ...
E. Coordinating NIOSH research activity with respect to other public and private research programs. F. Impact on occupational safety and health.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.