Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health* This is a document prepared by the National Academiesâ Committee for the Review of NIOSH Research Programs, also referred to as the Framework Com- mittee. This document is not a formal report of the National Academiesârather, it is a framework proposed for use by a number of National Academies committees that will be reviewing research in various research programs and health-outcomes programs. This version will be posted on the website of the National Academies and NIOSH for review. It is a working document that will be subject to change by the Framework Committee aimed at improving its relevance on the basis of responses received from evaluation committee members, NIOSH, stakeholders, and the general public before and during the course of the assessments conducted by independent evaluation committees of up to 15 research programs and health- outcomes programs. *Version of 12/19/05. âMembers of the National Academiesâ Committee for the Review of NIOSH Research Programs include: David Wegman (Chair; University of Massachusetts Lowell School of Health and Environ- ment), William Bunn, III (International Truck and Engine Corporation), Carlos Camargo (Harvard Medical School), Susan Cozzens (Georgia Institute of Technology), Letitia Davis (Massachusetts De- partment of Public Health), James Dearing (Kaiser Permanente), Fred Mettler, Jr. (University of New Mexico School of Medicine), Franklin Mirer (Hunter School of Health Sciences), Jacqueline Nowell (United Food and Commercial Workers International Union), Raja Ramani (Pennsylvania State University), Jorma Rantanen (Finnish Institute of Occupational Health), Rosemary Sokas (University of Illinois at Chicago School of Public Health), Richard Tucker (Tucker and Tucker Consultants, Inc. and University of Texas at Austin), and James Zuiches (North Carolina State University). 231
A pp e n d i x A 233 CONTENTS Acronyms used in this document I. Overview of Charge I.A. NIOSH Strategic Goals and Operational Plan I.B. Information from Other Evaluations I.C. Evaluation Committees I.D. Evaluation Committeesâ Information Needs II. Summary of Evaluation Process II.A. The Evaluation Flow Chart (Figure 2) II.B. Steps in Program Evaluation II.C. Assessing Relevance II.D. Assessing Impact I II. Evaluation of NIOSH Research Programsâthe Process III.A. Analysis of External Factors Relevant to the NIOSH Program III.A.1. Overview III.A.2. Considerations for Discussion III.B. Evaluating NIOSH Research Programs (Addressing Charges 1 and 2) III.B.1. Identifying Period of Time to be Evaluated III.B.2. Identification of Major Challenges (Circle in Figure 2) III.B.3. Analysis of NIOSH Program Strategic Goals and Objectives (Box A in Figure 2) III.B.4. Review of Inputs (Box B in Figure 2) III.B.5. Review of Activities (Box C in Figure 2) III.B.6. Review of Outputs (Box D in Figure 2) III.B.7. Review of Intermediate Outcomes (Box E in Figure 2) III.B.8. Review of End Outcomes (Box F in Figure 2) III.B.9. Review of Other Outcomes III.B.10. Summary Evaluation Ratings and Rationale III.C. Identifying Significant Emerging Research (addressing Charge 3) IV. Evaluation Committee Report Template V. Framework Committee Final Report Figure 1 The NIOSH operational plan presented as a logic model Figure 2 Flow chart for the evaluation of the NIOSH research program
234 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH Table 1 NORA High-Priority Research Areas by Category Table 2 Examples of NIOSH Program Research and Transfer Activities Table 3 Examples of a Variety of Scientific Information Outputs Table 4 Evaluation Committee Worksheet to Assess Research Programs and Subprograms
A pp e n d i x A 235 ACRONYMS ABLES Adult Blood Lead Epidemiology and Surveillance ACOEM American College of Occupational and Environmental Medicine AOEC Association of Occupational and Environmental Clinics BLS Bureau of labor Statistics CDC Centers for Disease Control and Prevention EC Evaluation Committee FACE Fatality Assessment Control and Evaluation FC Framework Committee HHE Health Hazard Evaluations MSHA Mine Safety and Health Administration NEISS National Electronic Injury Surveillance System NIOSH National Institute for Occupational Safety and Health NORA National Occupational Research Agenda NORA1 National Occupational Research Agenda 1996-2005 NORA2 National Occupational Research Agenda 2005-forward OSHA Occupational Safety and Health Administration OSHAct Occupational Safety and Health Act of 1970 OSH Review Occupational Safety and Health Review Commission Commission PART Performance Assessment Rating Tool PEL Permissible Exposure Limits SENSOR Sentinel Event Notification System of Occupational Risks TMT Tools, Methods, or Technologies
236 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH In September 2004, the National Institute for Occupational Safety and Health (NIOSH) contracted with the National Academies to conduct a review of NIOSH research programs. The goal of this multiphase effort is to assist NIOSH in increas- ing the impact of its research efforts in reducing workplace illnesses and injuries and improving occupational safety and health. The National Academies agreed to conduct this review and assigned the task to the Division on Earth and Life Studies and the Institute of Medicine. The National Academies appointed a committee of 14 members, including persons with expertise in occupational medicine and health, industrial health and safety, industrial hygiene, epidemiology, civil and mining engineering, sociology, program evaluation, communication, and toxicology; representatives of industry and of the workforce; and a scientist experienced in international occupational- health issues. The Committee on the Review of NIOSH Research Programs, re- ferred to as the Framework Committee (FC), held meetings during 2005 on May 5-6 and July 7-8 in Washington, DC, and on August 15-16 in Woods Hole and Falmouth, MA. This document is not a report of the National Academies; rather, it presents the evaluation framework developed by the FC to serve as a guideline and structure for NIOSH program reviews by Evaluation Committees (ECs) to be appointed by various divisions and boards of the National Academies. The ECs will use this framework in reviewing as many as 15 NIOSH research programs during a 5-year period. This is a working document. It is shared with NIOSH and the public. The framework and criteria may be modified by the FC on the basis of responses it receives from the ECs and other sources. It is incumbent upon the ECs to consult with the FC if portions of the evaluation framework presented here are inappropri- ate for the specific program under review. I.â OVERVIEW OF CHARGE At the first meeting of the FC, Lewis Wade, NIOSH senior science advisor, emphasized that the reviews should focus on evaluating NIOSHâs research pro- grams impact and relevance to health and safety in the workplace. In developing a framework, the FC was asked to address the following: 1. Evaluation committee assessment of progress in reducing workplace illnesses and injuries facilitated by occupational safety and health research through (a) an analysis of relevant data about workplace illnesses and injuries for the program activity, and (b) an evaluation of the effect that NIOSH research has had in reducing illnesses and injuries. The evaluation committees will rate the performance of each
A pp e n d i x A 237 program for impact of the program in the workplace. Impact may be assessed directly or, as necessary, using intermediate outcomes to esti- mate impact. Qualitative narrative evaluations may also be appropriate under certain circumstances. 2. Evaluation committee assessment of progress in targeting new research to the areas of occupational safety and health most relevant to future improvements in workplace protection. 3. Evaluation committee identification of significant emerging research areas which appear especially important in terms of their relevance to the mission of NIOSH. Those three charges constitute the scope of work of the individually appointed, independent ECs formed by the National Academies. I.A.â NIOSH Strategic Goals and Operational Plan As a prelude to understanding the NIOSH strategic goals and operational plan, NIOSH research efforts should be understood in the context of the Occupational Safety and Health Act (OSHAct) under which it was created. The OSHAct identi- fies workplace safety and health to be a national priority and gives employers the responsibility for controlling hazards and preventing workplace injury and illness. The act creates an organizational framework for doing this, with complementary roles and responsibilities assigned to employers and employees, OSHA, the States, the OSH Review Commission, and NIOSH. As one component of a national strategy the act recognizes NIOSHâs roles and responsibilities to be supportive and indirectâNIOSHâs research, training programs, criteria and recommendations are all intended to be used to inform and assist those actually responsible for hazard control (OSHAct Section 2b and Sections 20 and 22). Section 2b of the OSHAct describes thirteen interdependent means of accom- plishing the national goal, one of which is âby providing for research . . . and by developing innovative methods . . . for dealing with occupational safety and health problems.â Sections 20 and 22 give the responsibility for this research to NIOSH. In addition, NIOSH is given related responsibilities including: the development of criteria to guide prevention of work-related injury or illness, development of regulations reporting on the employee exposures to harmful agents, the establish- ment of medical examinations programs or tests to determine illness incidence and susceptibility, publication of a list of all known toxic substances, the assessment of potentially toxic effects or risk associated with workplace exposures in specific settings, the conduct of education programs for relevant professionals to carry out
238 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH the OSHAct purposes, and assisting the Secretary of Labor regarding education programs for employees and employers in hazard recognition and control. The NIOSH mission is âto provide national and world leadership to prevent work-related illness, injury, disability, and death by gathering information, con- ducting scientific research, and translating the knowledge gained into products and servicesâ. To fulfill its mission, NIOSH has established the following strategic goals: â¢ Goal 1: Conduct research to reduce work-related illnesses and injuries. â¦ Track work-related hazards, exposures, illnesses, and injuries for prevention. â¦ Generate new knowledge through intramural and extramural re- search programs. â¦ Develop innovative solutions for difficult-to-solve problems in high-risk industrial sectors. â¢ Goal 2: Promote safe and healthy workplaces through interventions, recommendations, and capacity-building. â¦ Enhance the relevance and utility of recommendations and guidance. â¦ Transfer research findings, technologies, and information into practice. â¦ Build capacity to address traditional and emerging hazards. â¢ Goal 3: Enhance global workplace safety and health through inter- national collaborations. â¦ Take a leadership role in developing a global network of occupa- tional health centers. â¦ Investigate alternative approaches to workplace illness and injury reduction and provide technical assistance to put solutions in place. â¦ Build global professional capacity to address workplace hazards through training, information sharing, and research experience. In 1994, NIOSH embarked on a national partnership effort to identify research priorities to guide occupational health and safety research for the next decade. The National Occupational Research Agenda (NORA) identified 21 high-priority research areas (see Table 1). NORA was intended not only for NIOSH but for the entire occupational health community. Approaching the 10-year anniversary of âSee also http://www.cdc.gov/niosh/docs/strategic/.
A pp e n d i x A 239 TABLE 1 NORA High-Priority Research Areas by Category Category Priority Research Area Disease and injury Allergic and irritant dermatitis Asthma and chronic obstructive pulmonary disease Fertility and pregnancy abnormalities Hearing loss Infectious diseases Low-back disorders Musculoskeletal disorders of upper extremities Trauma Work environment and workforce Emerging technologies Indoor environment Mixed exposures Organization of work Special populations at risk Research tools and approaches Cancer research methods Control technology and personal protective equipment Exposure-assessment methods Health-services research Intervention-effectiveness research Risk-assessment methods Social and economic consequences of workplace illness and injury Surveillance research methods NORA, NIOSH is working with its partners to update the research agenda. In the second decade of NORA, an approach based on industry sectors will be pursued. NIOSH and its partners will form sector research councils that will work to es- tablish sector-specific research goals and objectives. Emphasis will be placed on moving research to practice in workplaces through sector-based partnerships. Figure 1 is the NIOSH operational plan presented as a logic model of the path from inputs to outcomes for each NIOSH research program. The FC adapted the model to develop its framework. NIOSH will provide similar logic models relevant to each research program evaluated by an EC. I.B.â Information from Other Evaluations The FC is aware that several NIOSH programs have already been subjected to evaluation by internal and external bodies. Those evaluations range from overall assessments of NIOSH, such as the recent 2005 Performance Assessment Rating âDeveloped by NIOSH with the assistance of the RAND Corporation.
Mission: To Provide National and World Leadership to Prevent Work-Related Illness and Injuries 240 Customer Activities and Outputs Inputs Activities Outputs Intermediate customers Outputs Final customers Outcomes (transformation) (implementation) Research*: Surveillance, epidemiological and Intra- and Production behavioral studies, interagency inputs: intervention studies, entities, state, budget, staff, laboratory and field local and studies, -Recommendations, international facilities, exposure reports, publications, Transfer: Pilot and/or entitites, NGOs managerial, measurements and workshops, databases, -translation of market ready (e.g., standard risk assessment, conferences; research into technologies, Employees, infrastructure setting bodies Improvements control studies and practice, products training and employers, and labor, trade, development, -training and education and technologies; education in safety and materials and -information and professional industry, PPE studies and programs, development demonstration programs, dissemination; associations), educators, health in guidance, Planning trained professionals; -capacity building technology regulators workplaces regulations, inputs: through technical developers and standards, -tools and methods, best assistance mfrs, OSH customer/ media *Intramural and practices, developmental (HHEâs), training science stakeholder inputs, extramural, including releases technologies, licenses, and education community, SH surveillance and domestic and patents practitioners A-1.eps intervention international efforts, effectiveness data, such as work earmarks, conducted at ERCs, risk assessments ARCs and WHO Global Network of Collaborating Centers Transfe r Prevention through effective research, transfer, and evaluation External Factors: Economic and social conditions and regulatory environment FIGURE 1â The NIOSH operational plan presented as a logic model
A pp e n d i x A 241 Tool (PART) review, to evaluation of research program elements such as any ex- ternal scientific program reviews. The ECs should review all available prior reviews. Although it is important to consider all prior reviews in the present evaluation to aid in understanding the evolution of the programs and program elements, the ECsâ evaluations of NIOSHâs programs are independent of the prior reviews and evaluations. I.C.â Evaluation Committees Individual ECs will be formed through a process consistent with the rules of the National Academies for the formation of balanced committees. The commit- tees will be composed of persons with expertise appropriate to evaluating specific NIOSH research programs and may include representatives of stakeholder groups (such as labor unions and industry) and experts in technology transfer and pro- gram evaluation. The committees will conduct appropriate information-gathering sessions to obtain information from the sponsor (a NIOSH research program), stakeholders affected directly by the NIOSH research, and relevant independent parties. Each EC will consist of about 10 members, will meet about three times, and will prepare a report. The National Academies will deliver the report to NIOSH within 9 months after the individual EC is formed. EC reports will be subjected to the National Academies report-review process. I.D.â Evaluation Committeesâ Information Needs The ECs are expected to conduct information-gathering as appropriate on â¢ Background and resources of the program: â¦ History of program, including results of previous reviews. â¦ Program funding, by year, for the current year and the last 10 years. â¦ Program funding, by objective or subprogram. â¦ Extramural-grant awarding, cooperative agreement and contract- ing process, solicitation of research ideas, and advisory activities. â¢ Program goals and objectives. â¢ Internal NIOSH processes and research: â¦ Intramural surveillance, research, and transfer activities. â¦ Process to solicit and approve intramural research proposals. âPART focuses on assessing program-level performance and is one of the measures of success for the Budget and Performance Integration initiative of the presidentâs management agenda (see CDC Occupational Safety and Health at http://www.whitehouse.gov/omb/budget/fy2006/pma/hhs.pdf).
242 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH â¢ NIOSH-funded extramural research: â¦ Requests for proposals, cooperative agreements and research con- tracts distributed. â¦ Awardee products, including close-out reports, surveillance, re- search, and transfer activities, peer-reviewed publications, and patents. â¢ Products and technology transfer: â¦ Data related to program publications, conferences, recommenda- tions, patents, and so on. â¦ Past and planned mechanisms for transferring outputs to outcomes. â¦ Interventions, recommendations, and information-dissemination and technology-transfer activities designed to get research findings used to improve occupational safety and health. â¦ Outcomes of research, alerts, standard-setting, investigations, and consultations; for exampleâdocumented reductions in risk after program-supported interventions, employer and industry behav- ior changes made in response to research outputs, and worker behavior changes in response to research outputs. â¢ Impact on worker safety and healthâdata necessary to evaluate pro- gram impact on health outcomes (work-related injuries and illnesses) and exposures. â¢ The most severe or most frequent adverse health and safety outcomes or exposures in the research program and the most accessible improve- ments with respect to health and safety. â¢ Interactions within NIOSH and with other stakeholders: â¦ The role of program research staff in NIOSH policy-setting, Oc- cupational Safety and Health Administration (OSHA) and Mine Safety and Health Administration (MSHA) standard-setting, and voluntary standard-setting and other government policy functions. â¦ Other institutions and research programs with overlapping or similar portfolios and an explanation of the relationship between the NIOSH work and staff and those of other institutions. â¦ Stakeholder perspectives (OSHA, MSHA, union and workforce, industry, and so on.) â¦ Key partnerships with employers, labor, other government orga- nizations, academic institutions, nonprofit organizations, and â¦ International involvement and perspective. â¢ Systems to identify emerging problems and emerging research, includ- ing plans.
A pp e n d i x A 243 II.â SUMMARY OF EVALUATION PROCESS The ECs are charged with assessing the relevance, quality, and impact of NIOSH research programs. In conducting their evaluations, the ECs should as- certain whether NIOSH is doing the right things (relevance) and doing them right (quality) and whether these things are improving health and safety in the workplace (impact). II.A. The Evaluation Flow Chart (Figure 2) To address its charges, the FC has developed a flow chart (Figure 2) that breaks the NIOSH logic model into discrete, sequential program components to be char- acterized or assessed by the ECs. The components to be assessed are as follows: â¢ Major program-area challenges. â¢ Strategic goals and objectives. â¢ Inputs (such as budget, staff, facilities, the instituteâs research manage- ment, the NIOSH Board of Scientific Counselors, the NORA process, and NORA work groups). â¢ Activities (efforts by NIOSH staff, contractors, and grantees, such as hazard and health-outcome surveillance, exposure-measurement re- search, health-effects research, intervention research, health services, other research, and technology-transfer activities). â¢ Outputs (the products of NIOSH activities, such as publications, re- ports, conferences, databases, tools, methods, guidelines, recommen- dations, education and training, and patents). â¢ Intermediate outcomes (responses by NIOSH stakeholders to NIOSH products, such as public or private policy change, training and educa- tion in the form of workshop or seminar attendance, self-reported use or repackaging of NIOSH data by intermediary stakeholders, adop- tion of technologies developed by NIOSH, implemented guidelines, licenses, and reduction of workplace hazardous exposures and other risk factors). â¢ End outcomes (such as reduction of work-related injuries or illnesses, or hazardous exposures in the workplace). Drawing on the program logic model, the flow chart, and EC membersâ exper- tise, the ECs will delineate important determinants of a NIOSH research programâs agenda and the consequences of the NIOSH research activity. Determinants are conceptualized as inputs and external factors. Examples of external factors are
244 External Factors A B Analysis of Review and C Major Program Strategic Goals and Assessment Review and Area Challenges Objectives Driving of Inputs Assessment Determined by EC Current Program of Activities Planning: surveillance and intervention data; Surveillance, Independent assessment Assessment of NIOSH stakeholder inputs by EC members to process to select health effect research, compare with NIOSH program goals, intervention research, Production: intra- and technology transfer program area goals evaluation of goals extramural funding, activities, health services selected by NIOSH, staffing, physical facilities, and other research comparison with EC management structure assessment of challenges A-2.eps E F Review and D Review and Assessment of Review and Assessment of Intermediate Assessment End Outcomes Outcomes of Outputs Reduced injuries, Public policy impact, Publications, reports, illnesses, risk factors training/education, databases, tools, in the workplace self-reported use and/or methods, guidelines, repackaging by stakeholders, recommendations, implemented guidelines licences and patents External Factors FIGURE 2â Flow chart for the evaluation of the NIOSH research program
A pp e n d i x A 245 the research activities of industry and other federal agencies and the political and regulatory environment, which can affect all components of the research program (Figure 2). For purposes of this review, the results of inputs and external factors are the program research activities, outputs, and associated transfer activities that may result in intermediate outcomes and possibly eventual end outcomes. The FC has used the NIOSH logic model to develop the flow chart to define the scope and steps of an EC evaluation. The FCâs vision of how a program evalu- ation should occur is incorporated in a summary manner in the flow chart and discussed extensively in later sections. For example, the FC identified two types of outcomes: (a) intermediate outcomes, which represent implementations (what external stakeholders, such as employers, do in reaction to the products of NIOSH work, including new regulations, widely accepted guidelines, introduction of con- trol technologies in the workplace, changes in employer or worker behaviors, and changes in diagnostic practices of health-care providers), and (b) end outcomes, which are improvements (reductions in work-related injuries, illnesses, and haz- ardous exposures). For the purpose of evaluation, the FC does not differentiate between NIOSHâs âintermediate customerâ and âfinal customerâ activities (Figure 1); instead it combines them into a single category (Box E, Review and Assessment of Intermediate Outcomes, Figure 2). Training and development programs were ap- propriately defined as outputs by NIOSH in the logic model, but the FC finds more value in focusing on response to such offerings as intermediate outcomes (Box E) in the flow chart. The number of workers exposed to training activities represents a type of implementation of NIOSH outputs in the workplace. In evaluating each program or major subprogram, the EC must collect, analyze, and evaluate infor- mation on items described in each of the boxes of Figure 2. Further details on the evaluation are described in Section III of this document. II.B.â Steps in Program Evaluation The FC has concluded that useful evaluation requires: (a) a disciplined focus on a small number of questions or hypotheses typically related to program goals, per- formance criteria, and performance standards; (b) a rigorous method for answering the questions or testing the hypotheses; and (c) a credible procedure for developing qualitative and quantitative assessments. The evaluation process developed by the FC is summarized here and described in detail in Section III of this document. 1. Gather appropriate information from NIOSH and other sources. 2. Determine timeframe that the evaluation will cover (see III.B.1). 3. Identify program-area major challenges and objectives (see III.B.2). All NIOSH research programs, whether based on health outcomes or sectors, are designed to be responsive to the safety and health problems
246 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH in todayâs or tomorrowâs workplace. In the NIOSH vision, mission, values, and goals, each research program should have its own objec- tives. The ECs will provide an independent assessment of the major program challenges and determine whether they are consistent with the research programâs stated goals and objectives. 4. Identify subprograms and major projects in the research program. It is important for each EC to determine how necessary it is to disaggregate a program to achieve a manageable and meaningful evaluation of its components and the total program. Each research program may need to be broken down into several recognizable subprograms or major projects if an effective evaluation is to be organized. It may be advan- tageous for an EC to disaggregate a program into subprograms that NIOSH identifies. 5. Evaluate the program and subprogram components sequentially as discussed in Section III, using the flow chart (Figure 2) as a guide (Sec- tions III.B.3 through III.B.8). This will involve qualitatively assessing each phase of a research program by using the questions and guidance provided by the FC and professional judgment. 6. Evaluate the research programâs potential outcomes not yet appreciated (Section III.B.9). 7. Evaluate and score the program outcomes and important subprogram outcomes specifically for contributions to improvements in workplace safety and health. A worksheet is provided with specific items for con- sideration (Section III.B.10). 8. Evaluate and score the overall program for impact (Section III.B.10). Final program ratings will consist of a numerical score and discussion of its rationale. 9. Evaluate and score the overall program for relevance (Section III.B.10). Final program ratings will consist of a numerical score and discussion of its rationale. 10. Identify significant emerging research areas (Section III.C). On the basis of the expert judgment of EC members and information gathered from stakeholders (such as, labor, industry, academe, and govern- ment agencies) and from appropriate NIOSH sentinel-event field- investigation activities, the EC will respond to Charge 3 by identifying and describing emerging research that appears especially important in its relevance to the mission of NIOSH. The EC will assess the extent to which NIOSHâs program is responsive to todayâs and tomorrowâs needs and determine whether there are any gaps in response. 11. Prepare report by using the template provided in Section IV as a guide.
A pp e n d i x A 247 II.C.â Assessing Relevance FC members identified numerous possible factors to consider in assessing the relevance of NIOSH research programs, such as: â¢ The severity, frequency, or both of the health and safety outcomes addressed and the number of people at risk (magnitude) for these outcomes. â¢ The extent to which NIOSH research programs have identified and addressed gender issues and the concerns related to vulnerable popu- lations. Vulnerable populations are defined as groups of workers who have (1) biological, social, or economic characteristics that place them at increased risk of developing work-related conditions and/or (2) inadequate data collected about them. Vulnerable populations include disadvantaged minorities, disabled individuals, low-wage workers, and non-English speakers for whom language or other barriers present health or safety risks. â¢ The extent to which NIOSH research programs have addressed the health and safety needs of small businesses. â¢ The âlife stageâ of the problems being addressed. As the health effects are understood, emphasis should shift to intervention research, and from efficacy to effectiveness to research on the process of dissemina- tion of tested interventions. Gaps in the spectrum of prevention need to be addressed; for example, research on exposure assessment may be necessary before the next intervention steps can be taken. â¢ The structure, in addition to the content, of the research program. A relevant research program is more than a set of unrelated research projects; it is an integrated program involving an interrelated set of surveillance, research, and transfer activities. â¢ Appropriate consideration by NIOSH of stakeholder inputs. II.D.â Assessing Impact Causal attribution is a major aspect of program evaluation. It is necessary for the ECs to assess, to the extent possible, NIOSHâs contribution to end outcomes. Data on reductions in work-related injuries, illnesses, and hazardous exposures will be available for some programs. In some cases, they may be quantifiable. It is possible, however, to evaluate the impact of a NIOSH research program whether the outcomes are intermediate outcomes or end outcomes. Intermediate outcomes may be used as proxies for end outcomes in assessing impact if there is no direct evidence of improvements in health and safety as long as the ECs qualify their find-
248 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH ings. The ECs will describe the realized or potential benefits of NIOSHâs programs. Examples of realized intermediate outcomes include: new regulations, widely ac- cepted guidelines, work practices, and procedures, all of which may contribute measurably to enhancing health and safety at the work place. The contribution of a NIOSH program to technology now in use or being implemented is another important part of impact assessment. NIOSHâs contribu- tion can be assessed as major or important, moderate, likely, limited, or none. If technology development is in progress or has been abandoned, for whatever reason, the benefits are only potential or consist of knowledge gain. III.â EVALUATION OF NIOSH RESEARCH PROGRAMSâTHE PROCESS III.A.â Analysis of External Factors Relevant to the NIOSH Research Program As depicted in the logic model (Figure 1), the end outcome of reduced inju- ries, illnesses, or exposures is effected through stakeholder activities and outputs. All those involve the use of NIOSH outputs by stakeholders in industry, labor, other government agencies, and so on. It is evident that actions beyond NIOSHâs controlâby industry, labor, and other entitiesâhave important bearings on the incorporation in the workplace of NIOSHâs outputs to enhance health and safety. The implementation of research findings may depend on existing or future policy considerations. III.A.1.â Overview External factors may be considered as forces beyond the control of NIOSH that may affect the evolution of the program. External factors dominate the evolution of the path from NIOSH inputs to occupational health and safety outcomes (Figure 1). External factors can also be considered inputs to the evaluation of each aspect (planning, implementation, transfer, and others) of NIOSH research programs (Figure 2). Identification of external factors by the ECs is essential to providing a context for NIOSH program evaluation. External factors may best be assessed through the expert judgment of EC members regarding the knowledge base, the research program, and implementation of interventions as these relate to the needs in the occupational health or safety area targeted by the research program. The ECs, however, may choose additional approaches to assess external factors. The FC recommends the ECs ask NIOSH to identify and describe external factors early in the evaluation sequence. Factors external to NIOSH might have been responsible for achieving some outcomes, and they might also have presented formidable obstacles. The ECs must address both possibilities.
A pp e n d i x A 249 III.A.2.â Considerations for Discussion Some external factors may involve constraints on research activity related to target populations, methodological issues, and resource availability. For example, evaluators might examine whether â¢ Projects addressing a critical health need are technologically feasible. A workforce with appropriate size and duration, magnitude, and dis- tribution of exposure for measuring a health effect may not exist. For example, no population of workers has been exposed for 30 years to formaldehyde at the current OSHA Permissible Exposure Level (PEL), so the related cancer mortality can not yet be directly assessed. â¢ Research is inhibited because NIOSH investigators are unable to access an adequate study population. Under current policy, NIOSH must ei- ther obtain an invitation by management to study a workplace or seek a judicial order to provide authority to enter a worksite. (Cooperation under court order may well be insufficient for effective research.) â¢ Research is inhibited because the work environment, materials, and historical records cannot be accessed even with management and workforce cooperation. â¢ Adequate or established methods do not exist for assessing the environment. â¢ Records needed for historical-exposure reconstruction cannot be ac- cessed or do not exist. â¢ Intervention research is inhibited because an appropriate employer partner cannot be identified to institute the intervention. â¢ The NIOSH contribution to a certain area of research is reduced be- cause other institutions are working in the same area. â¢ NIOSH resources are inadequate to tackle the key questions. Evaluation of the impact of NIOSH research outputs on outcomes may require consideration of external factors that might have impeded or aided implementa- tion, measurement, and so on. For example, evaluators might consider whether â¢ Regulatory end points are unachievable because of obstacles to regu- lation or differing priorities of the regulatory agencies. For example, recommendations for improved respiratory protection programs for health-care workers might not be implemented because of enforce- ment policies or lack of acceptance by the administration of health- care institutions. â¢ A feasible control for a known risk factor or exposure is not im-
250 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH plemented because the costs of implementation are too high or the economic incentives under current circumstances do not favor such actions. â¢ Improvements in end points are unobservable because baseline and ongoing surveillance data are not available. For example, the current incidence of occupational noise-induced hearing loss is not known although surveillance for a significant threshold shift is feasible. (NIOSH conducts surveillance of work-related illnesses, injuries, and hazards, but comprehensive surveillance is not possible with existing resources.) â¢ Reductions in adverse effects of chronic exposure cannot be measured. For example, 90% of identified work-related mortality is from diseases, such as cancer, that arise only after decades of latency from first expo- sure; therefore, effects of reducing exposure to a carcinogen cannot be observed in the timeframe of most interventions. â¢ A regulation is promulgated that requires a technology that was devel- oped but not widely used. III.B.â Evaluating NIOSH Research Programs (Addressing Charges 1 and 2) III.B.1.â Identifying Period of Time to Be Evaluated Through study of materials presented by the NIOSH research program and other sources, an EC will become familiar with the history of the research pro- gram being evaluated and its major subprograms, program goals and objectives, resources, and other pertinent information. It is useful for the ECs to consider three general timeframes in conducting their reviews: â¢ 1970-1995, the period from the founding of NIOSH to the initiation of the NORA process (pre-NORA period). â¢ 1996-2005 (NORA 1 period). â¢ Current period and forward (NORA 2 period). It will be important for the ECs to get a general sense of the history of the NIOSH research program and its impact, but their efforts should be focused on the impact and relevance of NIOSH programs from 1996 on. It is recognized that many of the intermediate and end outcomes since 1996 are the consequence of research outputs accomplished earlier. Both the relevance of the research program
A pp e n d i x A 251 targets of NORA 1 and the proposed NORA 2 objectives for the next decade should be considered. NIOSH is in the midst of a substantial restructuring of the NORA agenda, and expert judgment about relevance and prospective impact of current research pro- grams will be most useful to the agency. The timeframes provided here are only for general guidance; the exact dates of the period to focus on in reviewing programs will depend on the specific research program under review. III.B.2.â Identification of Major Challenges (Circle in Figure 2) Early in its assessment process, an EC should independently identify the major challenges for its research program. These would be the matters the EC believes should have priority in the research program being evaluated. In arriving at a list of challenges, the EC should rely on surveillance findings, including NIOSH investiga- tions of sentinel events (through health-hazard or fatality-assessment programs), and its own expert judgment. Those should be supplemented with determinations or recommendations by appropriate advisory sources regardless of whether these sources have contributed to NIOSH program deliberations. This process will al- low the EC to compare its assessment of challenges to be addressed by NIOSH with NIOSH program goals, and to evaluate the congruence between the two as a measure of relevance (Charge 2). III.B.3.â Analysis of Research Program Strategic Goals and Objectives (Box A in Figure 2) The research program goals and objectives should be evaluated, with a focus on how each research programâs goals are related to NIOSHâs agency-wide strategic goals and to the major current challenges and emerging problems identified in the step above. Differences may exist between the importance or relevance of an issue and the influence NIOSH-funded research might have in addressing the issue. The EC should recognize that NIOSH research priorities may be strategic rather than based on the assessment of the state of knowledge. Some aspects of the NIOSH research programâs strategic goals and objectives would have been already subjected to evaluation by internal or external bodies. Re- search program relevant evaluations that should be requested include the NIOSH annual program review by the Leadership Team; the NORA research program proposal pre-award external review, NORA post-award program external review, and external scientific program review.
252 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH Questions to Guide the Evaluation Committee 1. Are the strategic goals and objectives of the program well defined and clearly described? 2. In the last decade, how well were program goals and objectives aligned with NORA 1 priorities? 3. How do the current strategic goals and objectives of the program relate to the current NIOSH strategy, including NORA 2? 4. Are the research program goals, objectives, and strategies relevant to the major challenges in the research program and likely to address emerging problems in the research program (as determined by the EC)? a. Did past program goals and objectives (research and dissemina- tion/transfer activities) focus on the most relevant problems and anticipate the emerging problems in the research program? b. Are the current program goals and objectives targeted to the most relevant problems and likely to address emerging problems in the research program? 5. How does the program identify emerging research areas? a. What information is reviewed by NIOSH? b. What advisory or stakeholder groups are asked to identify emerg- ing areas? c. What new research areas have been identified in the program? d. Were important areas overlooked? Assessment The EC will provide a qualitative assessment discussing the relevance of the areaâs goals, objectives, and strategies as related to the research programâs major challenges and emerging problems. III.B.4.â Review of Inputs (Box B in Figure 2) Inputs are categorized as planning or production inputs in the NIOSH logic model. Planning inputs include stakeholder inputs, surveillance and intervention data, and risk assessments. Production inputs include intramural and extramural funding, staffing, management structure, and physical facilities. Inputs for program evaluation include existing intramural and extramural information and, potentially, surveys or case studies that might have been devel- oped specifically to assess progress in reducing workplace illnesses and injuries and to provide information relevant to targeting research appropriately to future
A pp e n d i x A 253 needs. The ECs should request the relevant planning and production inputs from NIOSH. Planning Inputs Planning inputs can be qualitative or quantitative. Sources of qualitative inputs include â¢ Federal Advisory Committee Act panels (Board of Scientific Counsel- ors, Mine Safety and Health Research Advisory Committee, National Advisory Committee on Occupational Safety and Health, and so on). â¢ NORA research partners, initial NORA stakeholder meetings, later NORA Team efforts (especially strategic research plans), and the NORA Liaison Committee and federal liaison committee recommendations. â¢ Other federal research agendas, industry, labor, academe, professional associations, industry associations, and Council of State and Territorial Epidemiologists. â¢ OSHA and MSHA strategic plans. Attention should be given to how comprehensive the inputs have been and to what extent gaps have been identified or considered. Sources of quantitative inputs include â¢ Intramural surveillance information, such as descriptive data on expo- sures and outcomes (appropriate data may be available from a number of NIOSH divisions and laboratories). â¢ Health Hazard Evaluations (HHEs). â¢ Reports from the Fatality Assessment Control and Evaluation (FACE) program. â¢ Extramural health-outcome and exposure-assessment data from (1) OSHA and MSHA (inspection data) and the Bureau of Labor Statis- tics, U.S. Department of Defense, and U.S. Department of Agriculture (fatality, injury, and illness surveillance data); (2) state government partners, including NIOSH-funded state surveillance programs, such as Sentinel Event Notification System of Occupational Risks (SEN- SOR), Adult Blood Lead Epidemiology and Surveillance (ABLES), and state-based FACE; and (3) non-government organizations, such as the Association of Occupational and Environmental Clinics (AOEC) and
254 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH the American College of Occupational and Environmental Medicine (ACOEM). â¢ Appropriate data from NIOSH-funded, investigator-initiated extra- mural research. Production Inputs For each research program under review, NIOSH should specify an identifiable portion of the NIOSH intramural budget, staff, facilities, and management that has been allocated by divisions and offices that play a major role in the research program. Production inputs should be described primarily in terms of intramural research projects and staff, relevant extramural projects (particularly cooperative agreements and contracts), and HHEs and related staff. Consideration should also be given to budget inputs for program evaluation and to leveraged funds provided by partners, such as National Institutes of Health and the Environmental Protec- tion Agency joint requests for applications or program announcements and OSHA, MSHA, and Department of Defense contracts with NIOSH to conduct work. Assessment of those inputs should include consideration of (1) the degree to which allocation of funding and personnel has been reasonably consistent with the resources needed to conduct the research and (2) the extent to which funding for the relevant intramural research program activity has been limited by lack of discre- tionary spending beyond salaries (travel, supplies, external laboratory services, and so on). The assessments, therefore, should consider the adequacy of the qualitative and quantitative planning inputs and the use and adequacy of production inputs, particularly (1) and (2) above. Questions as a Guide for the Evaluation Committee 1. Were the planning, production, and other input data adequate? 2. How well were the major planning, production, and other program inputs used to promote the major activities? 3. Were the sources of inputs and the amount and quality of inputs adequate? 4. Was input obtained from stakeholders representing vulnerable work- ing populations and small businesses? 5. Were production inputs (intramural and extramural funding, staffing, management, and physical infrastructure resources) consistent with goals and objectives of the program?
A pp e n d i x A 255 Assessment The EC will provide a qualitative assessment that discusses the quality, ad- equacy, and use of inputs. III.B.5.â Review of Activities (Box C in Figure 2) Activities are defined as the efforts and work of the program, its staff, and its grantees and contractors. For purposes of the present evaluation, activities of the NIOSH program under review should be divided into research and transfer activi- ties. Research activities may be further categorized as surveillance, health-effects research, intervention research, health-services research, and other research (see sample classification of research activities in Table 2). Transfer activities include information dissemination, training, technical assistance, and education designed to translate research outputs into content and formats designed for application in the workplace to produce improvements in occupational safety and health. De- pending on the scope of the program under review, activities may also be grouped by research program objectives or subprograms. Conventional occupational-health research focuses appropriately on health effects and technology. A focus on socioeconomic and policy research and on surveillance and diffusion research is also needed to effect change because not all relevant intermediate outcomes occur in the workplace. There are important out- comes farther out on the causal chain that NIOSH can affect and thereby influence health and safety in the workplace. Some examples of types of research that might also prove important in addressing NIOSHâs mission are â¢ Socioeconomic research on cost shifting between worker compensa- tion and private insurance. â¢ Surveillance research to assess the degree of significant and systematic underreporting of select injuries and illnesses on OSHA logs. â¢ Research on methods to build health and safety capacity in com- munity health centers that serve low-income and/or minority-group workers, and to improve recognition and treatment of work-related conditions. â¢ Transfer research to change health and safety knowledge in teenagers while they are in high school to improve the likelihood of reduced injuries when they enter the workforce. â¢ Community-based participatory research on differences between re- cently arrived immigrants and US-born workers regarding perceptions of acceptable health and safety risks to target programs to meet the workforce training needs of immigrant workers.
256 A P P E N D I XAA r i c u l t u r e , F o r e s t r y , g and Fishing Research at NIOSH 149 TABLE 2 Examples of NIOSH Program Research and Transfer Activities Surveillance (including hazard and health surveillance and evaluation of surveillance systems) Health-effects research Epidemiologic research Toxicologic research Laboratory-based physical and safety risk factor research Development of clinical screening methods and tools Exposure-assessment research Intervention research Control technology Engineering controls and alternatives Administrative controls Personal protective equipment Work organization research Community-based participatory research Policy research (such as alternative approaches to targeting inspections) Diffusion and dissemination research Training effectiveness Information-dissemination effectiveness Diffusion of technology Health-services and other research Access to occupational health care Infrastructure researchâdelivery of occupational-health services, including international health and safety Socioeconomic consequences of work-related injuries and illnesses Worker compensation Technology-transfer and other transfer activities Information dissemination Training programs The ECs should review the list of research and transfer activities (projects) for the research program under review that have beentransfer activities (projects) for The ECs should review the list of research and completed, are in progress, or have been planned. Surveillance activitieshave been completed, are in progress,EC the research program under review that should be included in this review. An or should request that Surveillance activities shouldreview provide this review. An EC have been planned. the NIOSH program under be included in a list of activities, grouping the projects into research activities as inreview2, and specify whether they should request that the NIOSH program under Table provide a list of activities, are intramural or extramural. For extramural projects, the2, and specify whether grouping the projects into research activities as in Table key organizations and principalintramural or extramural. For extramural projects, the key organizations they are investigatorsâ names should be requested, as should whether the projects were principal investigatorsâ names shouldor a request for as should whether the and in response to a request for proposal be requested, application. For an in- projects were in response to a request for proposal or a request for application. For
A pp e n d i x A 257 tramural project, the EC should ask NIOSH to provide a list of key collaborators (other government agency, academe, industry, and/or union partners). The ECs should evaluate each of the research activities outlined in Table 2 to the extent that each forms an important element of the program research. In the case of a sector research program (for example, mining, construction) in which health-effects research is not being reviewed, the ECs should determine what re- search inputs are being used by the program to develop its targets and then assess the value of the inputs. Questions to Guide the Evaluation Committee in Assessing Research Activities 1. What are the major subprograms or groupings of activities within the program? 2. Were the activities consistent with program goals and objectives? 3. Were the research activities relevant to the major challenges in the research program? a. Did they address the most serious outcomes? b. Did they address the most common outcomes? c. Did they address the needs of both genders, vulnerable working populations, and small businesses? 4. Were the research activities appropriately responsive to the input of stakeholders? 5. To what extent were partners involved in the research activities? 6. Are the resource allocations appropriate, and appropriate at this time, for the research activities? 7. To what extent did peer reviews (internal, external, and precourse or midcourse) affect the activities? 8. Is there adequate monitoring of quality assurance procedures to ensure credible research data, analyses, and conclusions? Questions to Guide the Evaluation Committee in Assessing Transfer Activities 1. Is there a coherent planned program of transfer activities? 2. Are the programâs information dissemination, training, education, technical assistance, or publications successful in reaching the work- place or relevant stakeholders in other settings? How widespread is the response? 3. To what extent did the program build research and education capacity (internal or external)?
258 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH Assessment For this part of the assessment, the EC will provide a qualitative assessment discussing relevance and quality. This evaluation must include consideration of the external factors identified in Section III.A that constrain choices of research projects. The EC will consider the appropriateness of resource allocations with respect to issuesâ importance and the extent to which the issue is being addressed. A highly relevant and high-quality program would be comprehensive, address high-priority needs, produce high-quality results, be highly collaborative, and be of value to stakeholders. Programs may be progressively less relevant or of lower quality as those key elements are not up to the mark or are missing. The discussion should cover those aspects in sufficient detail to arrive at a qualitative assessment of the activities. Assessment of the transfer activities must include considerations of program planning, coherence, quality, and impact. III.B.6.â Review of Outputs (Box D in Figure 2) As shown in Figure 1, research inputs and activities lead to outputs. An output is a direct product of a NIOSH research program that is logically related to the achievement of desirable and intended outcomes. Outputs are created for research- ers, practitioners, intermediaries, and end-users, such as consumers. Outputs can be in the form of publications in peer-reviewed journals, recommendations, reports, Web-site content, workshops and presentations, databases, educational materials, scales and methods, new technologies, patents, technical assistance, and so on. Outputs of NIOSHâs extramurally funded activities should also be considered. Examples of major outputs are provided in Table 3. Depending on the intended audience, outputs may be tailored to communicate information most effectively to increase the likelihood of comprehension, knowl- edge, attitude formation, and behavioral intent. The extent of use of formative evaluation data (data gathered prior to communication for the purpose of improv- ing the likelihood of the intended effects) or intended user feedback in the design of the output can be considered an indicator of output quality. In addition to outputs themselves, many related indicators of the production, reference to, and utility of outputs can be conceptualized and made operational. Examples include the extent of collaboration with other organizations in the determination of research agendas, the conduct of research, the dissemination of research results, and interorganizational involvement in the production of outputs. Coauthorship is a measure of the centrality of NIOSH researchers in the broader research community.
A pp e n d i x A 259 TABLE 3 Examples of a Variety of Scientific Information Outputs Peer-reviewed publications by NIOSH staff Total number of original research articles by NIOSH staff Total number of review articles by NIOSH staff (including best-practice articles) Complete citation for each written publication Complete copies of the âtop fiveâ articles Collaboration with other public- or private-sector researchers Publications in the field of interest with other support by investigators also funded by NIOSH (for example, ergonomic studies with other support by an investigator funded by NIOSH to do ergonomics work, in which case NIOSH should get some credit for seeding interest or drawing people into the field) Peer-reviewed publications by external researchers funded by NIOSH Total number of NIOSH-funded original research articles by external researchers Total number of NIOSH-funded review articles by external researchers (including best-practices articles) Complete citation for each written report Complete copies of the âtop fiveâ articles Collaboration with other government or academic researchers NIOSH reports in the research program Total number of written reports Complete citation for each written report Complete copies of the âtop fiveâ reports Sponsored conferences and workshops Total number of sponsored conferences Total number of sponsored workshops For each sponsored conference or workshop, describe: Title, date, and location Partial vs complete sponsorship (if partial, who were cosponsors?) Approximate number of attendees and composition of participants Primary âproductsâ of the event (such as publication of conference proceedings) NIOSHâs assessment of value or impact Databases Total number of major databases created by NIOSH staff Total number of major databases created by external researchers funded by NIOSH grants, For each database: Title, objective (in one to four sentences), and start and stop dates Partial vs complete sponsorship (if partial, who were cosponsors?) Study or surveillance-system design, study population, and sample size Primary âproductsâ of the database (such as number of peer-reviewed articles and reports) Complete copies of the âtop twoâ publications and/or findings, to date, from each database continued
260 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH TABLE 3 Continued Recommendations Total number of major recommendations For each: Complete citation (article, report, or conference where recommendation was made) Summary in one to four sentences Percentage of target audience that has adopted recommendation 1, 5, and 10 years later Up to three examples of implementation in the field Identifications of âtop fiveâ recommendations to date Tools, methods, or technologies (TMT) Total number of major TMT (includes training and education materials) For each: Title and objective of TMT (in one to four sentences) Complete citation (if applicable) Percentage of target audience that has used TMT 1, 5, and 10 years later Up to three examples of implementation in the field Identification of âtop fiveâ TMT to date Patents Total number of patents For each: Title and objective patent (in one to four sentences) Complete citation Percentage of target audience that has used product 1, 5, and 10 years later Up to three examples of implementation in the field Identification of âtop fiveâ patents to date Miscellaneous Any other important program outputs The EC should ask NIOSH to provide information on all relevant outputs for the specific program for the chosen time period. Questions to Guide the Evaluation Committee 1. What are the major outputs of the research program? 2. Did the research program produce outputs that addressed the high- priority areas? 3. To what extent did the program generate important new knowledge or technology? 4. Are there peer-reviewed publications that are widely cited and consid- ered to report âbreakthroughâ results?
A pp e n d i x A 261 5. Were outputs relevant to both genders, vulnerable populations and health disparities? 6. Were outputs relevant to health and safety problems of small businesses? 7. Are products user-friendly in terms of readability, simplicity, and design? 8. To what extent did the program help to build the internal or extramu- ral institutional knowledge base? 9. Did the research produce effective cross-agency, cross-institute, or internal-external collaborations? Assessment For this part of the assessment, the EC should provide a qualitative assessment discussing relevance, quality, and usefulness. A highly ranked program will be one with outputs that address needs in high-priority areas, contain new knowledge or technology that is effectively communicated, contribute to capacity-building both inside and outside NIOSH, and are relevant to the pertinent populations. The discussion should cover those aspects in sufficient detail to support the qualitative assessment of the outputs. III.B.7.â Review of Intermediate Outcomes (Box E in Figure 2) Intermediate outcomes, for the purposes of this evaluation, are related to the programâs association with behaviors and changes at individual, group, and orga- nizational levels in the workplace. An intermediate outcome reflects an assessment of worth by stakeholders outside NIOSH (such as managers in industrial firms) about NIOSH research or its products. Intermediate outcomes include the production of standards, or regulations based in whole or in part on NIOSH research (products adopted as public policy or as policy or guidelines by private organizations or industry); attendance at training and education programs sponsored by other organizations; use of publications by workers, industry, and occupational safety and health professionals in the field; and citations of NIOSH research by industrial and academic scientists. More difficult-to-collect intermediate outcomes that may be valid indicators of quality or utility include self-report measures by users and relevant nonusers of NIOSH outputs. These indicators include the extent to which key intermediaries find value in NIOSH databases for the repackaging of health and safety informa- tion, the extent to which NIOSH recommendations are in place and attended to
262 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH in workplaces, and employee or employer knowledge of and adherence to NIOSH recommended practices. A research program might be evaluated in terms of whether it is recognized as a national center of excellence, is one of the larger and best research programs in the country, is recognized only in terms of particular staff or a particular labora- tory, duplicates other, larger facilities, or is not unique or has little capability or capacity. Questions to Guide the Evaluation Committee 1. Has the program resulted in stakeholder training or education activi- ties that are being used in the workplace or in school or apprentice programs? If so, what is the response to what is being done, and how widespread is the response? 2. Has the program resulted in standards, regulations, public policy, or voluntary guidelines that have been transferred to or created by the workplace in response to NIOSH outputs? 3. Has the program resulted in new control technology or administrative control concepts that are feasible for use or have been adopted in the workplace to reduce risk factors? 4. Has the program resulted in new personal protective equipment that is feasible for use or has been adopted in the workplace to reduce risk factors or exposures? 5. Has the program contributed to changes in health care practices to improve recognition and management of occupational health conditions? 6. Has the program resulted in research partnerships with stakeholders leading to changes in the workplace? 7. To what extent did the programâs stakeholders find value in NIOSHâs products (as shown by document requests, web hits, conference at- tendance, and so on)? 8. Has the program resulted in changes in employer or worker practices associated with the reduction of risk factors? 9. Does the program or a subprogram provide unique staff or laboratory capability that is a necessary national resource? If so, is it adequate or does it need to be enhanced or reduced? 10. Has the program resulted in interventions that protect both genders, vulnerable workers or address the needs of small businesses? 11. To what extent did the program contribute to increased capacity at worksites to identify or respond to threats to safety and health?
A pp e n d i x A 263 Assessment Only a qualitative assessment of product development, usefulness, and impact is required at this point in the EC report. Some thought should be given to the relative value of intermediate outcomes, and the FC recommends applying the well- accepted hierarchy-of-controls model. The discussion could include comments on how widely products have been used or programs implemented. The qualitative discussion should be specific as to the various products developed by the program and the extent of their use by specific entities (industry, labor, government, and so on) for specific purposes. Whether the products have resulted in changes in the workplace or in the reduction of risk factors should be discussed. The recog- nition accorded to the program or the facilities by its peers (such as recognition as a âcenter of excellenceâ by national and international communities) should be considered in the assessment. A program to be highly ranked should have a high level of performance in most of the relevant questions in this section. Whether the impact was caused by NIOSH alone or in combination with external agents should also be considered in the evaluation. An aspect of the evaluation can be whether the impact would have probably occurred without NIOSHâs efforts. III.B.8.â Review of End Outcomes (Box F in Figure 2) End outcomes are defined by measures of health and safety and of impact on process and programs. The FC recognizes that a major challenge in assessing the causal relationship between NIOSH research and specific occupational health and safety outcomes is that NIOSH does not have direct responsibility or authority for implementing its research findings in the workplace. Furthermore, the benefits of NIOSH research program outputs can be realized, potential, or limited to knowl- edge gain. For example, negative studies contribute to the knowledge base and the generation of important new knowledge is a recognized form of outcome, in the absence of measurable impacts. Outcome impact depends on there being a âreceptorâ for research results, including regulatory agencies, consensus and professional organizations, and em- ployers. The ECs should consider questions related to the various stages that lead to outputs, such as 1. Did NIOSH research identify a gap in protection or a means of reduc- tion of risk? 2. Did NIOSH convey that information to potential users in a usable form? 3. Was the research applied? 4. Did the results work?
264 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH End outcomes, for purposes of this evaluation, are changes related to health, including decreases in injuries, illnesses, deaths, and decreases in exposures or risk factors resulting from the research in the specific program or subprogram. Quanti- tative data are preferable to qualitative, but qualitative analysis may be necessary. Sources of quantitative data include â¢ Bureau of Labor Statistics (BLS) data on fatal occupational injuries (Census of Fatal Occupational Injuries) and nonfatal injuries and ill- nesses (Annual Survey of Occupational Injury and Illnesses). â¢ NIOSH intramural surveillance systems, such as the National Elec- tronic Injury Surveillance System (NEISS), the coal worker x-ray surveillance program, and agricultural worker surveys conducted by NIOSH in collaboration with the US Department of Agriculture. â¢ State-based surveillance systems, such as the NIOSH-funded ABLES, and the SENSOR programs (for asthma, pesticides, silicosis, noise- induced hearing loss, dermatitis, and burns). â¢ Selected state workers-compensation programs. â¢ OSHA, which collects exposure data, in the Integrated Management Information System. The FC is unaware of surveillance mechanisms for many occupationally re- lated chronic illnesses such as cancers arising from long exposure to chemicals and other stressors. For many outcomes, incidence and prevalence are best evaluated by investigator-initiated research. The strengths and weaknesses of the various sources of outcome data should be recognized by the ECs. Quantitative accident, injury, illness, and employment data and databases are subject to error and bias and should be used by the ECs for drawing inferences only after critical evaluation and examination of whatever corroborating data are available. For example, it is widely recognized that occupa- tional illnesses are poorly documented in the BLS Survey of Occupational Injuries and Illnesses, which captures only incident cases among active workers. Most ill- nesses that may have a relationship to work are not exclusively so related, and it is difficult for health practitioners to diagnose work-relatedness; few are adequately trained to make this assessment. Many of these illnesses have long latency and do not appear until years after people have left the employment in question. Surveil- lance programs may systematically undercount some categories of workers, such as contingent workers. Challenges posed by inadequate or inaccurate measurement systems should not drive programs out of difficult areas of study, and the ECs will need to be aware of such a possibility. In particular, contingent and informal working arrangements that place workers at greatest risk are also those on which
A pp e n d i x A 265 surveillance information is almost totally lacking, so novel methods for measuring impact may be required. In addition to measures of illness and injury, levels of exposure to chemical and physical agents and to safety and ergonomic hazards can be useful. Exposure or probability of exposure can serve as an appropriate proxy for disease or injury when a well-described occupational exposure-health association exists. In such instances, decreased exposure can be accepted as evidence that the end outcome of reduced illness has been achieved. That is particularly necessary in cases (such as exposure to asbestos) in which latency between exposure and disease outcome (lung cancer) makes effective evaluation of the relevant end outcome infeasible. As an example of how exposure levels can serve as a proxy, the number of sites that exceed an OSHA Permissible Exposure Limit (PEL) or an American Confer- ence of Governmental Industrial Hygienists threshold limit value is a quantitative measure of improvement of occupational health awareness and reduction of risk. In addition to exposure level, the number of people exposed and the distribution of exposure levels are important. Those data are available from multiple databases and studies of exposure. Apart from air monitoring, such measures of exposure as biohazard controls, reduction in requirements for use of personal protective equipment, and reduction of ergonomic risks are important. Clearly, the commitment of industry, labor, and government to health and safety are critical external factors. Several measures of this commitment can be useful for the EC: monetary commitment of the groups, attitude, staffing, and surveys of relative level of importance. To the extent that the resources allocated to safety and health are limiting factors, the ECs should explicitly assess NIOSH performance in the context of constraints. Questions to Guide the Evaluation Committee 1. What are the amounts and qualities of end-outcomes data (such as injuries, illness, exposure and productivity affected by health)? 2. What is the temporal trend in those data? 3. Is there objective evidence of improvements in occupational safety or health? 4. To what degree has the NIOSH program or subprogram been respon- sible for improvements in occupational safety or health? 5. If there is no time trend in the data, how do findings compare with data from other comparable US groups or the corresponding populations in other countries? 6. Is there evidence that external factors have affected outcome measures?
266 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH 7. Has the program been responsible for outcomes outside the United States that have not been described in another category? Assessment For this part of the assessment, the EC should provide a qualitative assessment discussing the evidence of reductions in injuries and illnesses or their appropriate proxies (impacts). III.B.9.â Review of Other Outcomes There may be health and safety impacts not yet appreciated, and other benefi- cial social, economic, and environmental outputs, including potential NIOSH im- pacts outside the United States. Many NIOSH study results and training programs may be judged to be important, or there may be evidence of implementation of NIOSH recommendations, outside the United States. Questions to Guide the Evaluation Committee 1. Is the program likely to produce a favorable change that has not yet occurred or not been appreciated? 2. Has the program been responsible for other social, economic, security, or environmental outcomes? 3. Has the programâs work had an impact on occupational health and safety in other countries? Assessment Evaluation by the EC may consist of a discussion of other outcomes, includ- ing positive changes that have not yet occurred; other social, economic, security, or environmental outcomes; and the impact that NIOSH has had on international occupational safety and health. It might also consider the incorporation of in- ternational research results into the NIOSH program of knowledge transfer for industry sectors. III.B.10.â Summary Evaluation Ratings and Rationale An EC should use its expert judgment to rate the relevance and impact of the research program and its important subprograms by first summarizing its assess- ments of the subprograms and overall program according to the several items listed in Table 4. Table 4 is only a worksheet intended as an aid to the EC in its evalua-
A pp e n d i x A 160 HEARING LOSS RESEARCH AT N I O 267 SH TABLE 4 Evaluation Committee Worksheet to Assess Research Programs and Subprograms Please respond to each with âmajor or important,â âmoderate,â âlikely,â âlimited,â or ânone.â Background Context for Program Impact 1.1 Evidence of reduction of risk factors in the workplace (intermediate outcome) and evidence that external factors affected reduction 1.2 Evidence of reduction in workplace exposure, illness, or injuries (end outcome) and evidence that external factors affected reduction Subprogram Activity Addressing Charge 1 Category Program 1 â¦ â¦ n 1.3 Contributions of NIOSH research Research and transfer activities to changes in work-related practices Transfer 1.4 Contributions of NIOSH research Research and transfer activities to reductions in workplace exposure, Transfer illness, or injuries 1.5 Evidence of external factors Research preventing application of NIOSH research results Transfer 1.6 Contribution of NIOSH research Research to enhancement of capacity in government or other research Transfer institutions 1.7 Contributions of NIOSH research Research to productivity, security, or environmental quality (beneficial Transfer side effects) Addressing Charge 2 2.1 Relevance of current and recently Research completed research and transfer activities to future improvements Transfer in workplace safety and health 2.2 Progress in targeting research to Research areas of study most relevant to future improvements in Transfer occupational safety and health
268 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH tion. Its purpose is to encourage the EC to summarize its work in one place and to concentrate on the subprograms and the items that will contribute to the final impact and relevance scores. To set the context for this step in the evaluation of the impact of the research program in preparation to respond to charge 1, the EC will first need to consider the available evidence of changes in work-related risks and adverse effects and external factors related to the changes. That information should be organized as a prose response to items 1.1 and 1.2 in Table 4. Next, the EC should review the responses to the questions in Sections III.B.6 through III.B.8 and systematically rate the impact of the research program and its subprograms by responding to items 1.3-1.7 in Table 4. To complete the table, the EC response should use one of the following five terms: âmajor or important,â âmoderate,â âlikely,â âlimited,â or ânoneâ (since 1995). The EC should evaluate separately the impact of the research and the impact of transfer activities. High ratings on items 1.3-1.7 require the committeeâs judgment that the program has contributed to outcomes. For example, outcomes have occurred earlier than they would have or are better than they would have been in the absence of the research program, or outcomes would have occurred in the absence of external factors be- yond NIOSHâs control or ability to plan around. The EC should then assess the relevance of the research program and subpro- grams in preparation for addressing charge 2. The EC should review the responses to the questions in Sections III.B.2 through III.B.5 and rate the relevance of the research program and its subprograms by responding to items 2.1 and 2.2 in Table 4. The same five terms should be used (âmajor or important,â âmoderate,â âlikely,â âlimited,â or ânoneâ) to evaluate separately the relevance of the research and the relevance of the transfer activities. Transfer activities occur in two contexts: (1) NIOSH efforts to translate intellectual products into practice and (2) efforts by stakeholders to take advantage of NIOSH products. Final Program Ratings To provide the final assessment of the research program for charge 1 (impact) and charge 2 (relevance), the ECs will use their expert judgment, their responses to the questions in Table 4, and any other appropriate information to arrive at one overall rating for the impact of the research program and one for its relevance to the improvement of occupational safety and health. In light of substantial differ- ences among the types of research programs that will be reviewed and the chal- lenge to arrive at a summative evaluation of both impact and relevance, however, the FC chose not to attempt to construct a single algorithm to produce the two final ratings.
A pp e n d i x A 269 Having completed Table 4, the EC should undertake its final assessment of the impact and relevance of the program. Final program ratings will consist of the numerical scores and prose descriptions of why the scores were given. As ex- plained below, the ECs will summarize their responses to charges 1 and 2 by rating the relevance and impact of the NIOSH research program on five-point scales in which 1 is the lowest and 5 the highest rating. The FC has made an effort to es- tablish mutually exclusive rating categories in the five-point rating scale; when the basis of a rating fits more than one category, the highest applicable score should be assigned. ECs will need to consider the impact and relevance of both NIOSH completed research and research in progress. In general, the assessment of impact will consider research completed, and the assessment of relevance will include research in progress related to likely future improvements. When assessing the relevance of the program, the EC should keep in mind how well the program has considered the frequency and severity of the problems being addressed, whether appropriate attention has been directed to both genders, vulnerable populations or hard-to-reach workplaces, and whether the different needs of large and small businesses have been accounted for. The FC has some concern that the impact scoring system proposed below might be considered a promotion of the conventional occupational-health re- search paradigm that focuses on health-effect and technology research and not give much emphasis to socioeconomic and policy research and to surveillance and diffusion research (as opposed to activities) needed to effect change. Clearly, not all intermediate outcomes occur in the workplace. There are important outcomes much farther out on the causal chain that NIOSH can affect, and not all these can be defined as well-accepted intermediate outcomes. NIOSH, for example, has an important role to play in generating knowledge that may contribute to changing norms in the insurance industry, in health-care practice, in public-health practice, and in the community at large. The ECs may find that some of these issues need to be addressed and considered as important to influence the external factors that limit application of more traditional research findings. Given the rapidly changing nature of work and the workforce and some of the intractable problems in manu- facturing, mining, and some other fields, the ECs are encouraged to think beyond the traditional paradigm. Rating of Impact 5 = Research program has made a major contribution to worker health and safety on the basis of end outcomes or well-accepted intermediate outcomes.
270 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH 4 = Research program has made a moderate contribution on the basis of end outcomes or well-accepted intermediate outcomes; research program generated important new knowledge and is engaged in transfer activi- ties, but well-accepted intermediate outcomes or end outcomes have not been documented. 3 = Research program activities or outputs are going on and are likely to produce improvements in worker health and safety (with explanation of why not rated higher). 2 = Research program activities or outputs are going on and may result in new knowledge or technology, but only limited application is expected. 1 = Research activities and outputs are NOT likely to have any application. NA = Impact cannot be assessed; program not mature enough. Rating of Relevance 5 = Research is in highest-priority subject areas and highly relevant to im- provements in workplace protection; research results in, and NIOSH is engaged in, transfer activities at a significant level (highest rating). 4 = Research is in high-priority subject area and adequately connected to improvements in workplace protection; research results in, and NIOSH is engaged in, transfer activities. 3 = Research focuses on lesser priorities and is loosely or only indirectly connected to workplace protection; NIOSH is not significantly involved in transfer activities. 2 = Research program is not well integrated or well focused on priorities and is not clearly connected to workplace protection and inadequately connected to transfer activities. 1 = Research in the research program is an ad hoc collection of projects, is not integrated into a program, and is not likely to improve workplace safety or health. III.C.â Identifying Significant Emerging Research (Addressing Charge 3) Among the most challenging aspects of conducting research for the purpose of prevention of injury and illness is identifying new or emerging needs or trends and formulating an active research response that appropriately uses scarce resources in anticipation of those needs. Each EC should review the procedures that NIOSH has in place to identify needed research relevant to the NIOSH mission. Each EC should review the success that NIOSH has had in identifying and addressing research to emerging issues. The review should include examination
A pp e n d i x A 271 of leading indicators from appropriate federal agency sources, such as the Envi- ronmental Protection Agency, the Department of Labor, the National Institute of Standards and Technology, the National Institutes of Health, the Department of Defense, and the Department of Commerce. Those indicators should track new technologies, products, and processes and disease or injury trends. One source of inputs deserving particular attention is the NIOSH HHE re- ports. NIOSHâs HHE program is a separate legislatively mandated program that offers a potential mechanism to identify emerging research needs that could be incorporated as an input in each of the programs evaluated. The ECs should con- sider whether appropriate consideration has been given to findings from the HHE investigations as they are related to the research program under review. Some additional indicators might include NIOSH and the NIOSH-funded FACE, the AOEC reports, the US Chemical Safety Board investigations, SENSOR and other state-based surveillance programs, and others. In addition, appropriate federal advisory committees and other stakeholder groups should be consulted to provide qualitative information. The EC members should use their expert judgment both to evaluate what NIOSH has identified as emerging research targets (charge 2) and to respond to charge 3 by providing recommendations to NIOSH for additional research that NIOSH has not yet identified. An ECâs response to charge 3 will consist primarily of recommendations for research in subjects that the EC considers important and of the committeeâs rationale. Questions to Guide the Evaluation Committee 1. What information does NIOSH review to identify emerging research needs? a. What is the process for review? b. How often does the process take place? c. How are NIOSH staff scientists and NIOSH leadership engaged? d. What is the process for moving from ideas to formal planning and resource allocation? 2. How are stakeholders involved? a. What advisory or stakeholder groups are asked to identify emerg- ing research targets? b. How often are such groups consulted, and how are suggestions followed up? 3. What new research targets have been identified for future development in the program under evaluation? a. How were they identified?
272 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH b. Were there lessons learned that could help to identify other emerg- ing issues? c. Does the EC agree with the issues identified and selected as sig- nificant and with the NIOSH response, or were important issues overlooked? d. Is there evidence of unwise expenditure of resources on unimport- ant issues? IV.â EVALUATION COMMITTEE REPORT TEMPLATE The following outline flows from the FCâs review of the generalized logic model prepared by NIOSH, the request for information from NIOSH programs, and the assessment model described earlier in this report. I. Introduction: This section should be a brief descriptive summary of the history of the program (and subprograms) being evaluated, with respect to pre-NORA, NORA 1, and current and future plans of the research program presented by NIOSH. It presents the context for the research on safety and health; goals, objectives, and resources; groupings of subprograms; and any other significant or pertinent information. (A list of the NIOSH materials re- viewed should be provided in an appendix to the EC report.) II. Evaluation of programs and subprograms (charges 1 and 2): A. Evaluation summary (includes a brief summary of the evalua- tion with respect to impact and relevance, scores for impact and relevance, and summary statements addressing charges 1 and 2). B. Strategic goals and objectives: Describes assessment of the sub- programs and overall program for relevance. C. Review of inputs: Describes adequacy of inputs to achieve goals. D. Review of activities: Describes assessment of the relevance and quality of the activities. E. Review of research program outputs: Describes assessment of relevance, quality, and potential usefulness of the research program. F. Review of intermediate outcomes and causal impact: Describes assessment of the intermediate outcomes and the causal attribu-
A pp e n d i x A 273 tion to NIOSH; includes the likely impacts and recent outcomes in the assessment. G. Review of end outcomes: Describes the end outcomes related to health and safety and provides an assessment of the type and degree of causal attribution to NIOSH. H. Review of other outcomes: Discusses other health and safety impacts that have not yet occurred; other beneficial social, eco- nomic, and environmental outcomes; and international dimen- sions and outcomes. I. Summary of ratings and rationale (see Table 4). III. Identification of needed research (charge 2): The EC should assess the progress that the NIOSH program has made in targeting new research in the fields of occupational safety and health. There should be a discussion of the assessment process and results. IV. Emerging research areas (charge 3): The EC should assess whether the NIOSH program has identified signifi- cant emerging research areas that appear especially important in terms of their relevance to the mission of NIOSH. The EC should respond to NIOSHâs perspective and add its own recommendations. V. Recommendations for program improvement: On the basis of the review and evaluation of the program, the EC may provide recommendations for improving the relevance of the NIOSH research program to health and safety conditions in the workplace and the impact of the research program on health and safety in the workplace as related to the research program under review. Appendix A: List of the NIOSH and related materials collected in the process of the evaluation V.â FRAMEWORK COMMITTEE FINAL REPORT At the conclusion of all individual program reviews, the FC will prepare a final report summarizing the findings of all the evaluating committees and providing NIOSH with an overall evaluation. All program ratings will be summarized and might be plotted graphically or with a Web chart. The following is a proposed outline of the FCâs final report:
274 A g r i c u lt u r e , F o r e s t r y , and Fishing Research at NIOSH I. Summary of national needs identified by the research programs reviewed. A. On the basis of the best available evidence, place those needs in the context of the overall estimated potential work-related disease and injury burden. B. Discuss the choices made and alternatives that might be the focus of current or future attention. C. Comment on programs not selected by NIOSH for evaluation by the National Academies. II. Assessment of how well the program goals. A. Were matched to the research program needs. B. Were adjusted to new information and inputs as the field of inter- est changed or program results became available. III. Assessment of NIOSH overall performance in the research programs reviewed. A. Distribution of available inputs. B. Activities and outputs. C. Intermediate outcomes. D. Summary assessment of significant differences among the programs. E. International impact. F. Leveraging of the NIOSH research activity with respect to other public and private research programs. G. Assessment of relative importance of external factors in permitting or preventing intermediate or end outcomes; attention paid to accounting for and planning within the constraints of external fac- tors (not simply assigning lack of progress to external factors). IV. Overall assessment of NIOSH impact on progress in reducing occu- pational injury and illness. A. Breakthrough knowledge. B. International impact. C. Addressing disparities. D. Targeting residual risks and intractable risks. E. Coordinating NIOSH research activity with respect to other public and private research programs. F. Impact on occupational safety and health. V. Summary, Conclusions, and Recommendations.