Review of Disability and Rehabilitation Research NIDRR Grantmaking Processes and Products (2012) / Chapter Skim
Currently Skimming:
6 Summative Evaluation
6 Summative Evaluation
Pages 158-206
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 158... ...
The chapter answers this central question and also provides an assessment of the methods used by the committee to conduct the summative evaluation. The scope and methods used to conduct the External Evaluation of the National Institute on Disability and Rehabilitation Research (NIDRR)
|
From page 159... ...
Finally, the committee assessed the outputs through an expert review process. Development of Quality Criteria A key element of the summative evaluation was the response to NIDRR's request to develop criteria for assessing the quality of its grantees' outputs.
|
From page 160... ...
would represent on the quality scale and decided that the midpoint should be "meeting expectations for good quality." This midpoint anchor description was operationalized for assessing the technical quality of publications, which made up 70 percent of the outputs reviewed. For publications, a rating of 4 was generally assigned to journal articles that were published in peer-reviewed journals based on the fact that they had already been peer reviewed and met the scientific standards of their respective fields of research or development.
|
From page 161... ...
161 SUMMATIVE EVALUATION BOX 6-1 Examples of Quality Indicators Considered in Determining Output Scores Technical Quality • S trength of literature review and framing of issues • C ompetence of design, considering the research question and other parameters of the study • Q uality of measurement planning and description • A nalytic methods and interpretation; degree to which recommendations for change are drawn clearly from the analysis • D escription of feasibility, usability, accessibility, and consumer satisfaction testing Advancement of Knowledge or the Field • D egree to which a groundbreaking and innovative approach is presented • A pplication of a formal test of a hypothesis regarding a technique used widely in the field to improve practice • L evel of advancement and improvement of current classification systems • U sefulness of descriptive base of information about factors associated with a condition • N ovelty of ways of studying a condition that can be applied to the develop ment of new models, training, or research Likely or Demonstrated Impact • D egree to which the output is well cited or has promise to be (for newer articles) • P otential to improve the lives of persons with disabilities by increasing accessibility • P ossibly transformative clinical and policy implications • P otential for building capacity, lowering costs, commercialization, etc.
|
From page 162... ...
The second part of the questionnaire included a series of questions designed to elicit more in-depth description of an output when needed and to provide supplemental information on the output's technical quality, how it advanced knowledge or practice, its likely or demonstrated impact, and how it was disseminated. This type of information, needed for a comprehensive assessment of the output, would not always be apparent in reviewing the output in isolation.
|
From page 163... ...
. Included in that data set was extensive information on all of the outputs produced by 1 An original criterion on output usability was collapsed into the final technical quality criterion.
|
From page 164... ...
The committee went back to the most recent year in which at least two grants under TABLE 6-1 Number of NIDRR Grants Ending in 2007 to 2009, with Grants Included in Sampling Pool Highlighted Program Mechanism 2007 2008 2009 Burn Model System (BMS) 0 5 0 Traumatic Brain Injury Model System (TBIMS)
|
From page 165... ...
were randomly selected for review in the following way. To balance the desire for the sample of grants to represent the nine program mechanisms included in the pool, the committee stratified the sampling at the program-mechanism level as a proportion of all grants in the sampling pool.
|
From page 166... ...
Table 6-2 presents the number of grants included in the sample by program mechanism and the types of outputs that were reviewed. To place the outputs reviewed into the larger context of the outputs produced by grantees in the sampling pool of 111 grants, Table 6-2 also shows that the proportions of publications and other outputs (tools, technology, and informational products)
|
From page 167... ...
148 (11%) 1,393 Sampling Pool SOURCE: Generated by the committee based on data from the grantee questionnaire and NIDRR grantee database.
|
From page 168... ...
These statements were reviewed after the ratings had been completed to identify attributes that particularly characterized the varying levels of quality and were helpful in further exemplifying the dimensions of the criteria. Although the final scores used to report results of the output assessment were based on subgroups' consensus scores, the committee conducted an interrater reliability analysis of their initial independent ratings (i.e., raw scores before the subgroup discussions)
|
From page 169... ...
The ICC resulting from the analyses are shown for each of the 4 criteria in Table 6-3. On the technical quality criterion, 13 of the 15 grants had statistically significant ICCs greater than .75; on the impact criterion, 11 grants had ICCs in this acceptable range; on the advancement of knowledge or the field criterion, 10 grants had ICCs in this range; and on the dissemination criterion, 9 grants had ICCs in this range.
|
From page 170... ...
ICC = intraclass correlation coefficient. SOURCE: Generated by the committee based on data from the committee's interrater reliability analysis.
|
From page 171... ...
RESULTS OF THE EVALUATION OF GRANTEE OUTPUTS This section presents the results of the committee's quality ratings of the four types of grant outputs according to the four criteria described above. In reviewing publications, the committee also referred to well-known sources that rate journal impact factors and count the number of times published articles have been cited; these results are summarized for the publications reviewed.
|
From page 172... ...
Reviewers commented that questions on a survey did not ap pear to be well thought out, that weak findings were not fully explained in conclusions, that it was unclear whether adequate validation had actually been performed on a measure or technology product, or that it was unclear whether a new technology was based on scientific evidence. Outputs that were rated lower in the area of design and methods, included reports of studies with small sample sizes that were not representative or that used single-group design with self-reported measures, no controls, and a pretest 40.0 34.5 35.0 Percentage 30.0 22.5 25.0 19.7 20.0 15.0 10.6 9.9 10.0 5.0 1.4 1.4 0.0 1 2 3 4 5 6 7 Quality Scale FIGURE 6-1 Distribution of quality ratings for technical quality (N = 142)
|
From page 173... ...
Attributes of tools and technology products that were rated highly were strong design and an article on a technological innovation that won a prize for technical quality from the journal in which it was published. One grantee stood out as presenting three outputs in a cohesive manner that illustrated the technical quality of the research and development: the first output described the research base, the second described the software application, and the third assessed use of the protocol.
|
From page 174... ...
. SOURCE: Generated by the committee based on data from the committee's output review.
|
From page 175... ...
45.0 40.4 40.0 35.0 Percentage 30.0 25.0 18.4 17.7 20.0 14.2 15.0 7.8 10.0 1.4 5.0 0.0 0.0 1 2 3 4 5 6 7 Quality Scale FIGURE 6-3 Distribution of quality ratings for likely or demonstrated impact (N = 141)
|
From page 176... ...
Dissemination editable Figure 6-4, Output ratings on the dissemination criterion (see Figure 6-4) are somewhat different from those on the other criteria.
|
From page 177... ...
Ratings of Grant Performance on Outputs After reviewing all of the outputs of an individual grant, the committee considered and rated the grant's overall performance with regard to all of the outputs reviewed for that grant. In rating grant performance, the com mittee also considered the fact that the outputs reviewed had been identified by the grant's principal investigators as the "top" two outputs per project, best reflecting the grant's achievements.
|
From page 178... ...
The NIDRR APRs, which were taken into consideration, in some cases did a good job of providing this larger context, but in many cases reviewing the APR alone was not sufficient. (Issues related to rating outputs in the context of grants and program mechanisms are discussed further in the section on self-assessment of the committee's review methods later in the chapter.)
|
From page 179... ...
. Acknowledging these limitations, the committee used Scopus and the Web of Science to access information on the journal impact factors and number of citations for each article included in the review.
|
From page 180... ...
a (ISI) b Journal Name the Journal 0.16 2.25 18 Archives of Physical Medicine and Rehabilitation 0.12 1.44 12 Journal of Spinal Cord Medicine 0.11 1.56 5 Journal of Burn Care and Research 0.10 1.49 4 Disability and Rehabilitation 0.42 3.79 3 Health Affairs NA NA 2 International Journal of Telerehabilitation NA NA 2 Journal of Burn Care and Rehabilitation 0.17 2.78 2 Journal of Head Trauma Rehabilitation 0.36 3.43 2 Journal of Neurotrauma 0.13 1.71 2 Journal of Rehabilitation Research and Development 0.08 1.68 2 Rehabilitation Psychology 0.10 1.22 2 Topics in Stroke and Rehabilitation NA NA 1 AER-Journal of Research and Practice in Visual Impairment and Blindness 0.06 NA 1 Assistive Technology 0.12 1.75 1 Brain Injury 0.12 1.72 1 Burns 0.07 1.38 1 Developmental Neurorehabilitation 0.04 NA 1 Disability and Health Journal 0.03 NA 1 Generations 0.03 NA 1 Hearing Journal 0.14 1.78 1 IEEE Transactions on Biomedical Engineering 0.04 1.19 1 Interacting with Computers 0.16 2.03 1 International Journal of Geriatric Psychiatry 0.05 NA 1 Journal of Aging and Social Policy
|
From page 181... ...
. As context for the results of Table 6-4, Annex 6-1 at the end of this chapter includes another table that lists the journals in which the larger pool of 111 NIDRR grantees published (the original pool of NIDRR grants from which the 30 grants in the sample for this evaluation were drawn)
|
From page 182... ...
However, the small number of grants reviewed within each program mechanism, as well as the relatively small number of outputs reviewed other than publications, made these comparisons tentative. Therefore, these comparisons are not reported here.
|
From page 183... ...
Other limitations and possible biases were discussed above. Having extensively acknowledged the limitations in conducting the summative evaluation, the committee asserts that the system developed for assessing the quality of outputs worked reasonably well, especially for publications, which made up 70 percent of the outputs reviewed.
|
From page 184... ...
One way of setting the quality bar higher would be to begin to encourage grantees to use standardized reporting forms and checklists for reviewing the technical quality of their own work before subjecting it to external review. Many resources exist for reviewing different types of
|
From page 185... ...
, but SJR and ISI are widely used and accepted in the field, which facilitates comparison of journal outputs of NIDRR grants and those of other federal agencies and across the diverse fields of research and development funded by NIDRR. Several technology journals in the area of rehabilitation do have journal impact factors as presented in Table 6-4 and Annex 6-1 (e.g., Assistive Technology, Transactions on Biomedical Engineering, Telemedicine and E-health)
|
From page 186... ...
Individual members were generally confident in the review process and its potential replication. Aspects of the process in which the committee had the greatest confidence were the technical quality scores, the face validity of the consensus scores that were produced for outputs, and the appropriateness of a seven-point quality rating scale.
|
From page 187... ...
Although capacity building is a major thrust of NIDRR's center and training grants, assessment of training outputs, such as the number of trainees moving into research positions, was also beyond the scope of the committee's charge. NIDRR's program mechanisms vary substantially in both size and duration, with grant amounts varying from under $100,000 (fellowship grants)
|
From page 188... ...
Considering other evaluation objectives might offer NIDRR further opportunities to continuously assess and improve its performance and achieve its mission. Alternative designs would be needed to evaluate the quality of grants or to allow comparison across program mechanisms.
|
From page 189... ...
With regard to sampling methods, larger grant sample sizes that allowed for generalization and comparison across program mechanisms would be needed. An alternative would be to increase the grant sample size in a narrower area by focusing on grants working in specific research areas across different program mechanisms or on grants with shared objectives (e.g., product development, knowledge translation, capacity building)
|
From page 190... ...
Nonetheless, in implementing this method, the committee encountered challenges and issues related to the diversity of outputs, the timing of evaluations, sources of information, and reviewer expertise. Diversity of Outputs The quality rating system used for the summative evaluation worked well for publications in particular, which made up 70 percent of the outputs reviewed.
|
From page 191... ...
For the output categories of tools, technology, and informational products, grantees sometimes provided a publication that did not necessarily describe the output. In addition, some outputs were difficult to assess when no corroborating evidence was provided to support grantees' claims about technical quality, advancement of the field, impact, or dissemination efforts.
|
From page 192... ...
and should be field tested with grants under multiple program mechanisms and refined as needed. For example, the technical quality criterion includes the dimension of accessibility and usability.
|
From page 193... ...
For example, the dissemination of technology devices should be assessed by examining progress toward commercialization; grantees' partnerships with relevant stakeholders, including consumers and manufacturers; and the delivery of information through multiple media types and sources tailored to intended audiences for optimal reach and accessibility. Timing of evaluations The committee suggests that the timing of an output evaluation should vary by the output type.
|
From page 194... ...
The APRs did not report on all of the specific outputs reviewed by the committee. Some outputs may have been reported in earlier reporting periods or have been produced after the NIDRR grant ended.
|
From page 195... ...
Whether output is described in a publication output and if so, which one X X X Key findings or lessons learned X How output is contributing to outcome-oriented goal by solving a problem, X X X X closing an identified gap, or benefiting the target population Description of "other accomplishments and contributions" X X X X * Defined for grantees by NIDRR as "those that contributed most to achieving the outcome-oriented goals for the award by advancing knowledge; increasing capacity for research, training or knowledge translation; or facilitating changes in policy, practice, or system capacity." 195 SOURCE: Generated by the committee based on Research Triangle International (2009)
|
From page 196... ...
For the technical quality of outputs other than publications, Recommendation 6-4 provides examples of ways to operationalize dimensions of accessibility and usability, such as providing evidence of testing the psychometrics of measurement instruments; assessing the usability features of informational products; and documenting the results of research and development tests of technology products that relate to human factors, ergonomics, universal design, product reliability, and safety. The APR system currently asks for information on how outputs were validated, but data elements that relate to such testing might be further specified in the system.
|
From page 197... ...
If future evaluations of output quality are conducted, the methods developed by the committee should be implemented with refinements to strengthen the design, validity, and reliability of the process. Whereas assessing grantee outputs is valuable, the committee believes that even greater value would come from assessing outputs in the context of a more comprehensive grant-level and program mechanism-level evaluation, which could yield broader implications for the value of grants, their impact, and future directions for NIDRR.
|
From page 198... ...
Annual Performance Report: Web-based reporting system instruction manual. Research Triangle Park, NC: Research Triangle International.
|
From page 199... ...
199 SUMMATIVE EVALUATION ANNEX 6-1 JOURNAL IMPACT FACTORS OF PUBLISHED ARTICLES FROM GRANTS IN THE ORIGINAL POOL FROM WHICH THE SAMPLE OF 30 GRANTS WAS DRAWN (N = 111 GRANTS, 631 ARTICLES) TABLE A6-1 Number of Times Articles Were Published in SJRa ISIb This Journalc Journal Name 0.160 2.254 75 Archives of Physical Medicine and Rehabilitation 0.124 1.442 32 Journal of Spinal Cord Medicine 0.170 2.779 28 Journal of Head Trauma Rehabilitation 0.109 1.563 27 Journal of Burn Care & Research 0.097 1.489 24 Disability and Rehabilitation 0.099 1.224 17 Topics in Stroke Rehabilitation 0.125 1.762 16 American Journal of Physical Medicine & Rehabilitation 0.077 1.676 15 Rehabilitation Psychology 0.118 1.75 10 Brain Injury NA NA 10 Hearing Loss 0.099 1.592 9 NeuroRehabilitation 0.061 NA 8 Assistive Technology 0.120 1.718 7 Burns 0.125 1.708 7 Journal of Rehabilitation Research and Development 0.102 1.364 7 Physical Medicine and Rehabilitation Clinics of North America 0.031 0.484 7 Research and Practice for Persons with Severe Disabilities 0.14 NA 7 Trends in Amplification continued
|
From page 200... ...
200 REVIEW OF DISABILITY AND REHABILITATION RESEARCH TABLE A6-1 (Continued) Number of Times Articles Were Published in SJRa ISIb This Journalc Journal Name 0.083 0.969 6 American Journal of Hospice & Palliative Medicine NA NA 6 Assistive Technology Outcomes and Benefits 0.263 2.302 6 Muscle & Nerve 0.04 0.74 6 Rehabilitation Counseling Bulletin 0.132 1.534 6 Respiratory Care NA 0.71 5 Exceptionality 0.033 NA 5 Journal of Vocational Rehabilitation 0.374 4.23 5 Multiple Sclerosis 0.041 NA 5 Topics in Spinal Cord Injury Rehabilitation 0.099 2.075 4 Clinical Neuropsychologist 0.615 6.254 4 Critical Care Medicine 0.254 2.777 4 General Hospital Psychiatry 0.424 3.792 4 Health Affairs 0.123 1.805 4 Journal of Clinical and Experimental Neuropsychology 0.684 8.017 4 Neurology 0.37 3.203 4 Shock 0.731 5.756 4 Stroke 0.202 2.374 3 Annals of Biomedical Engineering 0.032 NA 3 Career Development for Exceptional Individuals 0.276 3.436 3 Developmental Disabilities Research Reviews 0.052 2.271 3 Exceptional Children NA NA 3 Exceptional Parent 0.137 1.782 3 IEEE Transactions on Biomedical Engineering 0.068 1.842 3 International Journal of Clinical and Experimental Hypnosis 0.04 NA 3 Journal of Disability Policy Studies 0.048 0.694 3 Journal of Early Intervention 0.115 1.596 3 Journal of Intellectual Disability Research 0.357 3.426 3 Journal of Neurotrauma 0.032 0.222 3 Journal of Rehabilitation 0.503 5.391 3 Pediatrics 0.354 3.974 3 Psychosomatic Medicine 0.19 2.51 3 Spine 0.057 1.215 3 Violence Against Women NA NA 2 ADVANCE for Directors in Rehabilitation 0.118 2.018 2 American Journal of Speech-Language Pathology 1.415 10.746 2 Annals of Neurology 0.151 2.304 2 Archives of Clinical Neuropsychology 0.778 7.108 2 Archives of Neurology 0.1 2.109 2 Body Image
|
From page 201... ...
0.038 0.6 2 Infants and Young Children NA NA 2 Inside MS NA NA 2 International Journal of Telerehabilitation NA NA 2 Journal of Burn Care and Rehabilitation 0.026 0.174 2 Journal of Developmental Disabilities NA NA 2 Journal of Mine Action 0.432 4.791 2 Journal of Neurology Neurosurgery and Psychiatry 0.079 1.644 2 Journal of the Acoustical Society of America NA NA 2 Journal of Visual Impairment & Blindness NA NA 2 Journal on Developmental Disabilities 1.977 21.659 2 Lancet Neurology 0.291 4.106 2 Medicine and Science in Sports and Exercise NA NA 2 MS in Focus 0.118 1.731 2 Neuropsychological Rehabilitation 0.547 5.355 2 Pain 0.189 2.645 2 Physical Therapy 0.4 3.134 2 Progress in Brain Research 0.052 0.634 2 Prosthetics and Orthotics International NA NA 2 Rehabilitation Education 0.04 0.561 2 Remedial and Special Education 0.161 1.826 2 Spinal Cord NA NA 2 TASH Connections 0.191 2.195 1 Academic Radiology NA NA 1 ADVANCE for Managers of Respiratory Care NA NA 1 AER-Journal of Research and Practice in Visual Impairment and Blindness 0.038 0.694 1 American Annals of the Deaf 0.462 5.115 1 American Journal of Medicine 1.024 5.224 1 American Journal of Pathology 0.072 NA 1 American Occupational Therapy Association 0.293 3.397 1 Amyotrophic Lateral Sclerosis 0.249 3.984 1 Annals of Behavioral Medicine continued
|
From page 202... ...
2.011 8.058 1 Human Molecular Genetics 0.074 2.828 1 IEEE Engineering in Medicine and Biology Magazine 0.134 2.182 1 IEEE Transactions on Neural Systems and Rehabilitation Engineering 0.124 3.176 1 IEEE Transactions on Power Electronics 0.048 1.327 1 Intellectual and Developmental Disabilities 0.044 1.192 1 Interacting with Computers NA NA 1 International Journal of Advances in Rheumatology 0.035 NA 1 International Journal of Disability, Development, and Education 0.160 2.029 1 International Journal of Geriatric Psychiatry 0.147 2.244 1 International Journal of Medical Informatics NA NA 1 International Journal of MS Care
|
From page 203... ...
203 SUMMATIVE EVALUATION TABLE A6-1 (Continued) Number of Times Articles Were Published in SJRa ISIb This Journalc Journal Name 0.097 1.055 1 International Journal of Psychiatry in Medicine 0.03 NA 1 International Journal of Web Engineering and Technology 0.039 1.15 1 Internet Research 0.03 0.351 1 Intervention in School and Clinic NA NA 1 Journal for Vocational Special Needs Education 0.053 NA 1 Journal of Aging and Social Policy NA NA 1 Journal of Applied Rehabilitation Counseling 0.226 3.169 1 Journal of Behavioral Nutrition and Physical Activity 0.19 3.044 1 Journal of Biomedical Materials Research Part A 0.136 1.415 1 Journal of Cardiopulmonary Rehabilitation and Prevention 0.088 1.612 1 Journal of Clinical Psychology 0.122 1.506 1 Journal of Clinical Psychology in Medical Settings 0.107 1.433 1 Journal of Communication Disorders 0.04 NA 1 Journal of Disability Policy 0.116 1.5 1 Journal of Health Communication 0.065 1.354 1 Journal of Interpersonal Violence 0.063 1.849 1 Journal of Marriage and the Family 0.353 2.628 1 Journal of Neural Engineering 0.112 1.805 1 Journal of Occupational Rehabilitation 0.279 2.976 1 Journal of Orthopaedic Research 0.401 4.851 1 Journal of Pain 0.25 2.64 1 Journal of Pain and Symptom Management 0.03 0.959 1 Journal of Policy and Practice in Intellectual Disabilities 0.057 1.943 1 Journal of Positive Behavior Interventions 0.065 NA 1 Journal of Positive Psychology 0.059 1.09 1 Journal of Primary Prevention 0.039 NA 1 Journal of Prosthetics and Orthotics 0.224 2.842 1 Journal of Psychosomatic Research 0.042 1.343 1 Journal of Special Education 0.397 3.913 1 Journal of The American Geriatrics Society 0.209 2.91 1 Journal of The International Neuropsychological Society 0.155 3.129 1 Journal of Trauma-Injury Infection and Critical Care NA NA 1 Journal of Usability Studies continued
|
From page 204... ...
204 REVIEW OF DISABILITY AND REHABILITATION RESEARCH TABLE A6-1 (Continued) Number of Times Articles Were Published in SJRa ISIb This Journalc Journal Name NA NA 1 Journal of Vocational Special Needs Education 0.03 0.3 1 Journal of Web Engineering NA NA 1 Kansas Public Policy Journal NA NA 1 L'Audition Revue D'Informations Techniques et Scientifiques NA NA 1 Learning Disabilities Quarterly 0.033 NA 1 Lecture Notes in Computer Science 0.398 3.183 1 Medical Care 0.335 2.195 1 Medical Care Research and Review NA NA 1 Mississippi Brain Injury Association Newsletter 0.039 NA 1 Missouri Medicine NA NA 1 Momentum NA NA 1 MS Exchange 0.619 5.932 1 Neuroimage 0.408 2.764 1 Neuromuscular Disorders 0.314 3.772 1 Neurorehabilitation Neural Repair NA NA 1 Novartis Foundation Symposium NA NA 1 O&P Business News 0.027 NA 1 OT Practice NA NA 1 Paraplegia News 0.226 2.672 1 Pediatric Critical Care Medicine NA NA 1 Perspectives on Neurophysiology and Neurogenic Speech and Language Disorders 0.029 NA 1 Physical & Occupational Therapy in Geriatrics 0.026 NA 1 Planning NA NA 1 Principal Leadership NA NA 1 Proceedings of IEEE Virtual Rehabilitation 0.08 1.376 1 Psychiatric Rehabilitation Journal 0.145 2.388 1 Psychiatric Services 0.128 2.589 1 Psychological Assessment 0.039 0.753 1 Psychology in the Schools 0.056 0.615 1 Rehabilitation Nursing NA NA 1 Rehabilitation Outlook 0.429 4.744 1 Seminars in Arthritis and Rheumatism 0.049 NA 1 Seminars in Hearing 0.062 1.048 1 Social Work 0.034 NA 1 Technology and Disability 0.081 1.297 1 Telemedicine and E-Health 0.027 NA 1 The ASHA Leader
|
From page 205... ...
NA NA 1 The Judge's Journal NA NA 1 The RT News, Newsletter of AER Division 11 0.036 NA 1 Universal Access in the Information Society NA NA 1 US-ISPO Highlights 0.1 1.287 1 Women's Health Issues 0.35 3.443 1 Wound Repair and Regeneration 0.026 NA 1 Young Exceptional Children Total Published Articles Reviewed 631 NOTE: NA = journal not tracked by Scopus or Web of Science in the 2010 databases. aSJR (Scopus)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.