Review of Disability and Rehabilitation Research NIDRR Grantmaking Processes and Products (2012) / Chapter Skim
Currently Skimming:
4 NIDRR's Peer Review Process
4 NIDRR's Peer Review Process
Pages 83-127
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 83... ...
The National Institute on Disability and Rehabilitation Research's (NIDRR's) peer review process encompasses recruiting and training reviewers, conducting the review, and approving the awards.
|
From page 84... ...
NIDRR is to provide training for peer reviewers as is deemed necessary and appropriate. Title 34 of the Code of Federal Regulations (Disability and Rehabilitation Research Projects and Centers Program, 2009)
|
From page 85... ...
. The competition manager arranges for the participation of additional NIDRR staff as necessary, recruits reviewers, confirms receipt of all applications, and performs a final screen of eligibility and responsiveness.
|
From page 86... ...
Additional knowledge of the field or the applicant is not to influence the review. Annex 4-1 at the end of this chapter provides more detail on the grant selection criteria, as well as an example of the selection criteria for a Disability and Rehabilitation Research Project-General (DRRP)
|
From page 87... ...
. Recruiting of Peer Reviewers NIDRR establishes peer review panels of five to seven members to review each submitted grant application (National Institute on Disability and Rehabilitation Research, 2006, 2009a, 2009b)
|
From page 88... ...
NIDRR staff also generally make time for one-onone consultation if it is requested. Peer Reviewer Orientation The competition manager conducts a competition-specific orientation session for all reviewers (National Institute on Disability and Rehabilitation Research, 2010b)
|
From page 89... ...
A maximum score does not require a written rationale, but reviewers are encouraged to include comments. As described by NIDRR management, the number of applications 6 Panel discussion procedures described here are a synthesis of information from written sources provided by NIDRR (National Institute on Disability and Rehabilitation Research, 2009a, 2009b)
|
From page 90... ...
NIDRR management stated that the site visit is considered a 7 Doris Werwie, personal communication, National Institute on Disability and Rehabilitation Research, April 14, 2011.
|
From page 91... ...
Applicants respond to the questions in writing prior to and during the visit. Prefunding Meeting Following peer review, NIDRR holds a prefunding meeting involving the NIDRR Director, the Deputy Director, the two division Directors, the agency's scientific advisor, the competition manager, and interested NIDRR staff to develop specific funding recommendations (National Institute on Disability and Rehabilitation Research, 2009a)
|
From page 92... ...
Finally, while all funded applications received an overall score of at least 77, the contractor observed a lack of consistency in the language used for the scoring criteria for each program mechanism and no consistency in the number of points assigned to each scoring criterion within a mechanism. NIDRR's peer review contractor surveys peer reviewers for feedback following every panel using the OSERS Panel Review Logistics Evaluation Form (Synergy Enterprises, Inc., 2010)
|
From page 93... ...
Combined preparation and participation time ranged from a low of an average of 15 hours to a high of an average of 60 hours. It should be noted that some peer reviewers both reported less time spent preparing and gave low ratings to time allowed for initial review, indicating they had less time to prepare than they wished.
|
From page 94... ...
(42%) SOURCE: Generated by the committee based on National Institute on Disability and Rehabilitation Research (2010a)
|
From page 95... ...
95 NIDRR'S PEER REVIEW PROCESS FY 2008 FY 2009 Awarded (% of reviewed) Awarded (% of reviewed)
|
From page 96... ...
RESULTS OF THE ASSESSMENT OF NIDRR'S PEER REVIEW PROCESS This section first describes observations of peer review panels by committee members and NRC staff. It then reports on perspectives of NIDRR staff and grantees on the peer review process, including challenges and suggestions for change.
|
From page 97... ...
Each panel was led by the competition manager/panel monitor, who ably guided the panel members through a thoughtful and consistent review following the peer review procedures outlined above. The competition manager was observed providing such help as reminding panel members they could not use other information outside of the proposals to rate the applications, should not deduct points from multiple criterion areas for the same flaw, should apply criteria consistently across applications, and needed to justify all scores adequately.
|
From page 98... ...
Quality and Consistency Most of the NIDRR staff interviewed participated in NIDRR's peer review process as competition managers and/or panel monitors. Respondents indicated that the process is very strong and that the hard-working nature of their colleagues contributed to the quality of the process.
|
From page 99... ...
Not because they didn't enjoy reading the applications but because it was too time consuming, too burdensome." Another interviewee suggested that peer reviewers are undercompensated, even compared with reviewers for other ED programs, and suggested that their honorarium be increased. Staff remarked that the agency is tackling the identified problems with the peer review process through a continuous quality improvement effort.
|
From page 100... ...
who served on NIDRR peer review panels during FY 20082009. NIDRR provided the reviewers' names and contact information, but not the competitions they reviewed.
|
From page 101... ...
These included support and facilitation of the review panel by NIDRR staff (78 percent) , integrity of the peer review process overall (76 percent)
|
From page 102... ...
(%) Support and facilitation of the review panel by 120 1 4 17 28 50 NIDRR staff Integrity of the peer review process overall 117 2 2 20 41 35 Thoroughness of the deliberation (i.e., grant 121 0 3 24 32 41 scoring and discussion)
|
From page 103... ...
A final survey question about NIDRR's peer review process was posed to the subset of peer reviewers (55 percent) who had experience with other federal agency peer review processes.
|
From page 104... ...
(%) Quality of the review process 63 6 21 36 21 16 Transparency of the review 61 8 13 46 17 16 process Quality of the proposals 62 2 19 47 16 16 reviewed Fairness of the review process 62 7 16 47 15 15 Reliability of the ratings 60 8 20 44 15 13 Expertise of the panel members 62 7 19 50 6 18 SOURCE: Generated by the committee based on data from the peer reviewer questionnaire.
|
From page 105... ...
Increase the amount of time to review applications prior to the panel meeting Eleven respondents expressed strong concern about receiving applications too close to the meeting date and advised NIDRR to send applications more than a few weeks before the meeting; two respondents suggested 1 month before and five respondents suggested 2 months before. Four respondents contended that there was a link among three of the six sources of peer reviewer role dissatisfaction; as articulated by one, "either give reviewers more time to review, or reduce the number of applications each person has to review, and increase the compensation rate." The quantitative data support peer reviewers' comments on dissatisfaction with the amount of time allowed to review applications prior to a meeting.
|
From page 106... ...
Provide reviewers with a choice of formats in which to review applications Eight respondents suggested that reviewers should be able to choose the format of the applications they review based on their preferences, such as printed copies, on CD, or through a password-protected website. Respondents noted that PDF files would allow them to use the search function, and Word files would allow them to embed comments through the track changes feature.
|
From page 107... ...
scoring process, (5) guidance and group facilitation provided by NIDRR staff, (6)
|
From page 108... ...
, the purposes of the priority to be funded (cited by three) , how the panel will be facilitated by NIDRR staff (cited by two)
|
From page 109... ...
Sixty-one percent of reviewers rated appropriateness of the evaluation criteria to applications under review as more than adequate to excellent, and 34 percent rated clarity of the criteria when applying them to applications as adequate. Improve the scoring process used to rate applications on the criteria Ten respondents suggested there is considerable variation in how reviewers apply the scoring system to evaluate applications.
|
From page 110... ...
. While these are reasonable suggestions, it must be noted that support and facilitation of the review panel by NIDRR staff was the element rated most favorably by peer reviewers.
|
From page 111... ...
Two respondents suggested limiting the number of discussion hours per day or scheduling more days but fewer hours, or possibly even holding panel meetings only during academic breaks (cited by one) as ways to improve quality and reduce reviewer fatigue.
|
From page 112... ...
rated the guidance they received from NIDRR in writing review comments as good to excellent. CONCLUSIONS AND RECOMMENDATIONS The sources relevant to peer review who were engaged in this evaluation, including NIDRR staff, grantees, and peer reviewers, consistently described NIDRR's peer review process as generally good, although still in need of some improvement.
|
From page 113... ...
NIDRR staff spend considerable time recruiting and screening potential reviewers. Competition managers regularly must manage potential conflicts of interest and rule out qualified reviewers.
|
From page 114... ...
Training enhancements should also take into account the different needs of inexperienced and experienced reviewers. While support and facilitation of the review panel by NIDRR staff was
|
From page 115... ...
However, it is the viewpoint of the committee that a more formal quality improvement initiative is needed to improve the consistency of managing the panel meetings. The guidance provided to peer reviewers by NIDRR in writing review comments was rated quite highly by peer reviewers, but several commented that the quality of the feedback actually provided to grantees was lacking in depth and specificity, and was inconsistent.
|
From page 116... ...
Finally, NIDRR may want to explore a blended model of in-person and teleconference meetings to reduce the burden imposed by teleconferences for some reviewers, as expressed in the survey. Use of Consumer Peer Reviewers To address its mission, NIDRR makes concerted efforts to include both scientists with disabilities and consumers without scientific expertise in the peer review process.
|
From page 117... ...
The Office of Congressionally Directed Medical Research Programs (CDMRP) , located in the Department of Defense, fully integrates consumers and scientists on peer review panels.
|
From page 118... ...
, 379-388. Disability and Rehabilitation Research Projects and Centers Program, 34 C.F.R.
|
From page 119... ...
With the exception of Part 359, governing SCIMS, where the points are prespecified, the distribution of points across the selected criteria is determined by NIDRR staff. All criteria are displayed in Table A4-1.
|
From page 120... ...
120 REVIEW OF DISABILITY AND REHABILITATION RESEARCH TABLE A4-1 Selection Criteria from Title 34, Code of Federal Regulations Title 34, Part 350 DBTAC DRRP KT Burn Title 34, Part 350 Importance of the problem x x x x Responsiveness to an absolute or competitive x x x x priority Design of research activities x x x x Design of development activities x x x x Design of demonstration activities x x x x Design of training activities x x x x Design of dissemination activities x x x x Design of utilization activities x x x x Design of technical assistance activities x x x x Plan of operation x x x x Collaboration x x x x Adequacy and reasonableness of the budget x x x x Plan of evaluation x x x x Project staff x x x x Adequacy and reasonableness of resources x x x x Title 34, Part 356 Quality and level of formal education Previous work experience Recommendations Quality of a research proposal The research hypothesis, methodology, and design Resources, equipment, institutional support
|
From page 121... ...
121 NIDRR'S PEER REVIEW PROCESS Part 356 Part 359 Part 75 TBI 21 FIP RRTC RERC ARRT Switzer SCI SBIR x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x continued
|
From page 122... ...
Title 34, Part 75 Need for project Significance Quality of the project design Quality of project services Quality of project personnel Adequacy of resources Quality of the management plan Quality of the project evaluation SOURCE: Generated by the committee based on the CFR, Title 34. of recommending criteria for a competition, NIDRR staff also recommend which subcriteria are relevant.
|
From page 123... ...
. In making a final selection, the Secretary considers the extent to which outstanding or superior applicants present a unique opportunity to effect a major advance in knowledge, address critical problems in innovative ways, present proposals that are consistent with the NIDRR's Long-Range Plan, build research capacity within the field, or complement and significantly increase the potential value of already planned research and related activities.
|
From page 124... ...
(1) The Secretary considers the responsiveness of the application to an absolute or competitive priority published in the Federal Register.
|
From page 125... ...
(2) In determining the quality of the plan of operation, the Secretary considers the following factor: (i)
|
From page 126... ...
(2) In determining the quality of the project staff, the Secretary considers the extent to which the applicant encourages applications for employment from persons who are members of groups that have traditionally been underrepresented based on race, color, national origin, gender, age, or disability (4 points)
|
From page 127... ...
(2) In determining the adequacy and accessibility of resources, the Secretary considers the following factors: (i)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.