Review of Disability and Rehabilitation Research NIDRR Grantmaking Processes and Products (2012) / Chapter Skim
Currently Skimming:
2 Evaluation Scope and Methods
2 Evaluation Scope and Methods
Pages 34-55
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 34... ...
Both the scope of the evaluation and potential limitations of its findings are discussed to clarify the extent to which the findings can be generalized and used by the National Institute on Disability and Rehabilitation Research (NIDRR) to enhance its priority-setting, peer review, and grant management processes.
|
From page 35... ...
However, these long-term outcomes were beyond the scope of the committee's evaluation. In summary, the scope of the evaluation encompassed key NIDRR processes of priority setting, peer review, and grant management (process evaluation)
|
From page 36... ...
of Peer review process and ranking Education Reduced Q5: INFORMING AND Legislative disparities GENERATING NEW framework Q3: PROJECT MANAGEMENT between people PROJECTS with disabilities NIDRR post award administration, project Public Evidence of movement of management, and reporting and general comments outputs along the R&D population in Grantee project management and reporting to pathway (e.g., theory to employment, Results of prior NIDRR measure, prototype testing to health and NIDRR market) functioning, assessments Evidence of outgrowths of GRANTEES participation first generation outputs (e.g., Standards and and community Implementation of R&D, capacity building, intervention found to be best practices living and knowledge translation grants effective with one population being adapted for another)
|
From page 37... ...
. Annex 2-1 at the end of this chapter shows the various quality criteria and dimensions used across these studies, as well as those compiled by an external advisory group convened by NIDRR in August 2008 to assist the agency in laying the groundwork for the current External Evaluation of NIDRR and its Grantees.
|
From page 38... ...
The process evaluation phase involved reviewing existing documentation and collecting testimonial data to examine how NIDRR, through its policies and procedures and in practice, develops its research and funding priorities, reviews and evaluates submitted proposals, makes decisions and awards grants based on these reviews, and manages grant-supported activities. The summative evaluation phase involved the use of expert panels to assess the quality of grant outputs.
|
From page 39... ...
, who examined 175 organizational studies, 55 percent is an acceptable response rate for a survey targeting executive directors. NIDRR peer reviewers The committee sent invitations to complete a survey to all individuals (a total of 156)
|
From page 40... ...
Analysis of Process Data The committee analyzed quantitative data from the online surveys of stakeholders and peer reviewers descriptively to examine frequencies and measures of central distribution. For process data gathered from NIDRR staff, grantees, stakeholder organizations, and peer reviewers (i.e., responses to open-ended questions that were based on individuals' opinions or perspectives)
|
From page 41... ...
Where data sets were larger (stakeholders = 72 respondents and peer reviewers = 121) , counts were used in reporting results of the qualitative analyses for greater transparency, but the committee acknowledges that in most cases, the number of specific observations for certain topics also is quite low.
|
From page 42... ...
These grants were drawn from nine of NIDRR's program mechanisms: Burn Model System, Traumatic Brain Injury Model System, Spinal Cord Injury Model System, Rehabilitation Research and Training Center, Rehabilitation Engineering Research Center, Disability and Rehabilitation Research Project, Field Initiated Project, Small Business Innovation Research II, and Switzer Fellowship. The primary focus of the committee's summative evaluation was on assessing the quality of research and development outputs produced by grantees.
|
From page 43... ...
– NIDRR External Advisory Group report • uestions for process evaluation interviews tailored for Q NIDRR personnel, Semistructured face-to NIDRR management and staff stakeholder face interviews • eb-based survey (form tailored for NIDRR W organizations stakeholders) Web-based survey (self-administered)
|
From page 44... ...
– Analysis of peer review tracking database (Synergy, 2008) • uestions for process evaluation interviews tailored for Q NIDRR personnel, Semistructured face-to NIDRR management and staff peer reviewers face interviews • eb-based survey (form tailored for peer reviewers)
|
From page 45... ...
• rantee outputs and supplemental questionnaire with G Grantees (PIs) Self-administered items classifying and describing the types of outputs, questionnaire features of the outputs that reflect the quality criteria, and future research and development as an outgrowth • uality of outputs: committee member rating sheet, Q Committee's peer Peer review of outputs based on criteria such as: review scores and other materials – Technical quality (APR, grantee – Advancement of knowledge or the field questionnaire)
|
From page 46... ...
However, NRC staff who conducted the interviews believe all NIDRR staff members were candid in their comments. To assess the quality of outputs for the summative evaluation, the committee used sound criteria that were based on the cumulative literature reviewed and its members' own research expertise in diverse areas of disability and rehabilitation research, medicine, engineering, and the social sciences, as well as their expertise in evaluation, economics, knowledge translation, and policy.
|
From page 47... ...
Fourth, with regard to the process evaluation, it is possible that the respondents choosing to respond to the online surveys of stakeholders and peer reviewers may have differed from those who declined to participate. However, the response rates were respectable on both surveys (80 percent on the peer reviewer survey and 55 percent on the stakeholder survey)
|
From page 48... ...
. Meeting report of the NIDRR External Evaluation Advisory Group.
|
From page 50... ...
• ovement along research and/or development pathway M Peer Recognition of Output • umber of peer-reviewed articles (e.g., using such N x x x x x measures as "journal impact factor" for assessing relative quality, or counting publications in high-quality journals/publishers) -- Number of original research articles x -- Number of review articles x • umber of research reports N x x x x • umber of conference papers published N x x x • umber of citations in peer-reviewed articles (e.g., N x x x x using such measures as "crown indicator," "highly cited publication counts," "H-index")
|
From page 51... ...
and evidence of their involvement • ppropriate targeting of audience A x Output Meets Acceptable Standards of Science and Technology • ppropriate methodology for answering research A x questions • utput is well defined and implemented with integrity O x • ell-documented conclusions, supported by the W x x literature • bility for the document to stand alone A x • eets human subjects protection requirements M x • eets ethical standards M x • pplies concepts of universal design and accessibility A x • esearch done on the need for and development of the R x instrument or device • easurement instruments tested and found to have M x acceptable type and level of validity • easurement instruments tested and found to have M x acceptable level of reliability 51 continued
|
From page 52... ...
• ilot testing conducted with consumers and adaptations P x made as indicated • heoretical underpinnings and evidence base (e.g., level T x of evidence per American Psychological Association, American Medical Association) used and well documented Output Has Potential to Improve Lives of People with Disabilities • alued by consumers V x • vidence of beneficial outcomes (i.e., How does the E x output improve abilities of people with disabilities to perform activities of their choice in the community and also expand society's capacity to provide full opportunities and accommodations?
|
From page 53... ...
S x • doptability A x • ffordability and repair cost (for devices) A x • ost/benefit C x • inancing available F x • ompatibility and durability C x • ustainability S x • afety issues considered S x Output Utility and Relevance • ddresses high-priority area A x x • ddresses NIDRR mission, International Classification A x of Function, Disability and Health (ICF)
|
From page 54... ...
• ignificance and magnitude of impact within and S x x x x x outside of field (i.e., Does output help build internal or extramural institutional knowledge? Does output transcend boundaries?
|
From page 55... ...
• valuation plan and consumer reviews E x • cope of training: topics covered, number of training S x events, hours of training, participant hours, participant characteristics • echnical assistance/consultation provided T x x • umber of sponsored conferences and workshops, with N x x x documented sponsorship, number and composition of attendees, products of the event, assessment of the event • issemination of research through databases D x x x x SOURCE: Generated by the committee based on cited sources.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.