Skip to main content

Currently Skimming:

8 Evaluating Outcomes of Interdisciplinary Research and Teaching
Pages 149-170

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 149...
... For example, in evaluating a federal program to reduce unemployment, policy makers are less concerned about the output: how many people participate in the program, and more so by the outcome: how many participants obtain employment as a consequence of participating in the program within a particular period. The same is true of research: there is less interest in the number of publications than in the impact of the publications in terms of their quality, relevance, and stature.
From page 150...
... First, a successful IDR program will have an impact on multiple fields or disciplines and produce results that feed back into and enhance disciplinary research. It will also create researchers and students with an expanded research vocabulary and abilities in more than one discipline and with an enhanced understanding of the interconnectedness inherent in complex problems.
From page 151...
... The examples cited here are intended to demonstrate approaches that may be useful. TOOLKIT BOX 8-1 Measures to Evaluate Interdisciplinary Work In a recent study by the Harvard Interdisciplinary Studies Project, Veronica Boix Mansilla and Howard Gardner looked at research and teaching practices at several interdisciplinary institutes and programs over the last 2 years.
From page 152...
... EVALUATING THE DIRECT AND INDIRECT IMPACTS OF IDR Many of the standard means for evaluating disciplinary research and teaching can also be applied to interdisciplinary research and teaching: the use of metrics, such as number of publications, citations of publications, and successful research-grant proposals; teaching evaluations by students; benchmarking with other programs (when comparable programs exist) ; and national or international awards for and recognition of researchers or teachers.
From page 153...
... Evaluating IDR productivity can also be complicated because, although in some situations IDR may take more time than disciplinary research, it may have a high degree of depth and importance of achievement. The contribution achieved by a research team may be more than the sum of the individual accomplishments.
From page 154...
... are systems-focused, interdisciplinary centers at universities all across the United States, each in close partnership with industry.a Primary goals of ERCs are to integrate engineering education and research, build competence in engineering practice, and produce engineering graduates with the depth and breadth of education needed for success in technological innovation and leader ship of interdisciplinary teams. NSF views ERCs as change agents for academic engineering programs and the engineering community at large.
From page 155...
... Criteria in each category change in the three review periods -- years 1-3, 4-6, and 7-10. For example: Years 1-3 Years 4-6 Years 7-10 Evaluation Category High Quality Low Quality High Quality High Quality Research Thrust team is Thrusts and Thrust team is Thrust team is program appropriately projects are appropriately appropriately cross- single- cross- cross disciplinary, disciplinary and disciplinary, disciplinary, projects are isolated from projects are projects are becoming one another, interdependent interdependent interdependent project results within the thrust within the thrust within the thrust have no role in and contributing and contributing and contributing other projects to other thrusts to other thrusts to other thrusts and thrusts Education & Cross- Little inter- Cross- Cross educational disciplinary dependence disciplinary disciplinary, outreach research culture between faculty research culture team culture for is developing, and students, has been students students work involving few developed, flourishes and in teams, ratio students in students work impacts other of graduate to teams, ratio of in teams, ratio parts of the undergraduate graduate to of graduate to university is at most 2:1 undergraduate undergraduate students is students is at over 2:1 most 2:1 continues eThird-Year Renewal Review Process and Review Criteria for Engineering Research Centers in the ERC Class of 2000.
From page 156...
... For example, developing the engineering technologies necessary to achieve space flight has led to advances in the computer control of engineering processes, which have resulted in improvements in the reliability of industrial products and processes. Information-Sharing Networks Researchers who divide their time between traditional disciplinary departments and interdisciplinary programs or centers often form "networks of practice"2 through which they share information that does not always appear in immediate or traditional forms, such as publications in academic journals (see Box 8-3)
From page 157...
... That indicates that, although center directors may act as the organizing force, it is graduate students and postdoctoral scholars that weave the web. aHybrid Vigor Institute home page http://www.hybridvigor.org/.
From page 158...
... That, in turn, can strengthen an institution's ability to attract outstanding graduate students, faculty, and postdoctoral scholars, as happened at the Joint Institute for Neutron Sciences at the University of Tennessee and Oak Ridge National Laboratory,6 the Keck Graduate Institute,7 and the Kavli Institute for Theoretical Physics at the University of California, Santa Barbara.8 Demonstrating the Value of Instrumentation IDR may demonstrate the value of a major tool or instrumentation that has multiple applications. For example, synchrotron radiation, which provides an ultrabright photon source, has had a major impact on many fields 4School of Natural Resources and Environment, Global Change Program home page http:/ /www.snre.umich.edu/faculty-staff-directory/list.php?
From page 159...
... It has also brought together multiple disciplines and groups of researchers, even though it was originally developed for the study of electronic and structural properties of materials. For example, in molecular biology, researchers use a synchrotron to obtain structures of proteins rapidly, and this enables pharmaceutical companies to develop new drugs (see Box 2-5)
From page 160...
... created the Integrative Graduate Education and Research Trainee (IGERT) program in 1997 in response to a grow ing recognition that graduate students in science and engineering needed to be better prepared for research that involves two or more disciplines.
From page 161...
... and Pionte, M "Content, Consciousness, and Colleagues: Emerging Themes from a Program Evaluation of Graduate Student Progress Toward Multidisciplinary Science." 42nd Annual Association of Institutional Research Forum.
From page 162...
... Indeed, the difficulty of developing effective review criteria is illustrated by a recent evaluation of NSF programs in IDR. Despite NSF's long-standing leadership, the evaluation urged the agency to "establish supplementary review criteria that will help to assess the quality of interdisciplinary effort in those programs where both single and multiple discipline proposals com
From page 163...
... . Addressing the complex strategic questions involved in identifying new directions and finding sufficient support can be aided by advisory committees that have the interdisciplinary expertise mentioned above.
From page 164...
... Finally, academic institutions have begun to implement interdisciplinary courses and minors, and even a few majors, but development of course and program evaluations is in its early stages. The beginning questions to ask in framing an evaluation mechanism might include the following: · Is interdisciplinary teaching attracting more of the general student population to science courses?
From page 165...
... COMPARATIVE EVALUATIONS AND RANKINGS OF RESEARCH INSTITUTIONS Comparative evaluations of research institutions, such as the National Academies' assessment of doctoral programs (see Box 5-1) and similar activities that rank university departments,11 should include the contributions of both interdisciplinary activities and single-discipline contributions.
From page 166...
... Given the inherent difficulty and expense of most interdisciplinary activities and the need to balance investments in research, it is essential to measure and maintain its value to the research enterprise. FINDING Reliable methods for prospective and retrospective evaluation of inter disciplinary research and education programs will require modification of the peer-review process to include researchers with interdisciplinary expertise in addition to researchers with expertise in the relevant disciplines.
From page 167...
... , the funding agencies required that centers include a core program on evaluation. TTURC evaluation researchers have developed the follow ing outcome metrics for measuring and evaluating science that bridges two or more disciplines:b · How well is the collaborative transdisciplinary work of the centers (includ ing training)
From page 168...
... · How well IDR enhances the training of students and the careers of researchers in ways that surpass the results expected from disciplinary research; these might include employment in a broader array of positions, more rapid progress in gaining tenure and other goals, and greater numbers of speaking invitations. · Whether the research leads to practical answers to societal questions; for example, an IDR effort to reduce hunger should produce some measurable progress toward that goal.
From page 169...
... · Review a proposed interdisciplinary program in climate change by using input not only from experts in climate change and related fields-such as oceanography, meteorology, atmospheric chemistry, and land use-but also experts in the constituent disciplines -- such as physics, chemistry, and statistics -- and nonresearchers for whom the research is relevant; the contributions of different disciplines might be submitted separately in written form, and these reports would be offered to a full review panel, including both disciplinary and interdisciplinary researchers, for final assessment. E-4: Comparative evaluations of research institutions, such as the Na tional Academies' assessment of doctoral programs and activities that rank university departments, should include the contributions of inter disciplinary activities that involve more than one department (even if it involves double-counting)
From page 170...
... The publication criteria might include the nature of the journal audiences for whom the work is published; citation analysis that reveals a broad interdisciplinary interest in the work being cited; "double counting" of publications, by which credit for a given paper is awarded to all coauthors; multiple-authorship patterns that would reveal the disciplinary backgrounds of coauthors; and others that are still being developed. · Include the facilitation of interdisciplinarity as part of the accreditation process.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.