Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
6 Evaluation by Self-Assessment Various elements of programs intended to enhance the integrity of institutional research were described in Chapter 4, and many of these elements have demonstrated at least a measure of success in some cir- cumstances and by some, often vague benchmarks. The committee con- cluded, however, that the principal mode for evaluation of the effective- ness of an integrated program should be based on self-assessment and peer review, particularly when undertaken in the context of institutional accreditation. The other elements of an effective programâperformance- based assessment, education, and attention to regulatory complianceâ are generally components of rigorous institutional self-assessment. Use of self-assessment as a principal tool directly extends, with greater specific- ity, the first two recommendations of the 1992 National Academy of Sci- ences (NAS) Panel on Scientific Responsibility and the Conduct of Re- search (NAS, 1992): â¢ Individual scientists in cooperation with officials of research insti- tutions should accept formal responsibility for ensuring the integrity of the research process. They should foster an environment, a reward sys- tem, and training processes that encourage responsible research practices. (p. 13) â¢ Scientists and research institutions should integrate into their cur- ricula educational programs that foster faculty and student awareness of concerns related to the integrity of the research process. (p. 13) 112
EVALUATION BY SELF-ASSESSMENT 113 The committee endorses the principle of self-assessment as an ante- cedent to formal appraisal of the performances of academic departments and individual faculty members. This chapter discusses in further detail the processes of self-assessment at the levels of both the institution and the research unit (at the level of the department, the research group, and the individual investigator), and it offers some initial steps that might be taken in the application of self-assessment to evaluation of the environ- ment for integrity in research. SELF-ASSESSMENT AND ACCREDITATION IN HIGHER EDUCATION State and federal governments mandate accreditation of institutions of higher education as a requirement for the recognition of the degrees they grant, but a process of peer review is used almost exclusively to grant accreditation. Many different accrediting bodies exist, and these are based either on geography or, for professional schools, on the degree granted. In virtually every case, the heart of the accreditation process is self-assessment (Borden and Owens, 2001; Ewell and Lisensky, 1988; Middle States Commission on Higher Education, 2000). Process of Self-Assessment in Higher Education Self-assessment begins with instructions from the accrediting body regarding the criteria for evaluation. These instructions generally provide a template for self-assessment that enables the institution to respond to a series of âmustâ and âshouldâ standards. The issues and questions posed are usually of a general nature so that institutions can present their solu- tions in different ways. These responses are then judged by external re- viewers and provide the basis of an institutionâs case for accreditation. Because institutions of higher education vary markedly in their histories, cultures, curricula, and human and physical resources, accreditation is not based on presumptions as to particular ârightâ answers. In fact, within very broad boundaries, institutional diversity is valued and encouraged. The process of self-assessment in institutions of higher education is lengthy, costly, and difficult (Ewell, 1991; Middle States Commission on Higher Education, 2000). In the process, institutions critically evaluate their strengths and weaknesses and consequently develop new ideas for self-improvement. Continuous quality improvement is the goal. The in- tent is to accomplish this by associating the process of self-assessment with anticipated improvements in the desired output (e.g., better-edu- cated students). Periodic reaccreditation provides a formal process for evaluation of the results. It cannot be accomplished by simple completion
114 INTEGRITY IN SCIENTIFIC RESEARCH of a mandated checklist, although attention to inputs and processes, as well as outcomes, is required. Determining the effectiveness of any program of continuous quality improvement is largely qualitative, although some measures can be quan- tified. For example, to evaluate an outcome of creating better-educated students, institutions may look to improvements in scores on examina- tions taken by their students interested in pursuing advanced degrees (e.g., the Graduate Records Examination and the Medical College Admis- sion Test), or they may monitor the postgraduation careers of their stu- dents to determine what kind of postgraduate programs or professional positions they enter and how well they were prepared (National Center for Higher Education Management Systems, 2001). Institutions that have recently completed an accreditation cycle often serve as models for other institutions as the latter prepare for a similar process, thereby encourag- ing a culture of quality improvement (Ewell and Lisensky, 1988). The self-assessment process is organized around a set of faculty com- mittees, each responsible for analysis and recommendations concerning an element or program of the institution considered important in deter- mining institutional effectiveness (Middle States Commission on Higher Education, 2000). Both the openness and the accuracy of the reports pro- duced by these committees are furthered by the inclusion of large num- bers of faculty in their membership. The reports are assimilated and inte- grated into a master document, the self-study report. A template provided by the accrediting body may guide both the format and the general con- tent of the report. This self-assessment is submitted to the accrediting body and is then provided to volunteer peer reviewers, who complete their evaluations with a visit to the institution. The reviewers generally interview members of the faculty, administration, and student body, and then prepare a detailed report that addresses perceived strengths, weak- nesses, and areas for desirable or necessary improvement. This report constitutes the basis of a recommendation to the accrediting body regard- ing the continuation of accreditation. The committee appreciates that in an open society like that of the United States, different people look at many measures of the quality of institutions of higher education and research (e.g., rankings of institu- tions according to the amount of support for research they receive from the National Institutes of Health, and U.S. News and World Report rankings of schools and educational programs). Nevertheless, governments have traditionally relied upon processes of accreditation to define those entities qualified to provide education at a particular level or in a specific field. Despite this reliance, however, the federal government has generally avoided specific involvement in accreditation processes, largely entrust- ing them to private-sector accrediting agencies, consistent with the
EVALUATION BY SELF-ASSESSMENT 115 nationâs tradition of limited government. The effectiveness of this ap- proach to rigorous institutional accreditation is reflected, at least in part, by the fact that the U.S. system of higher education is admired around the world, as well as by the publicâs continuing general acceptance of self- monitoring as a means of assessing professional quality. Promotion of Integrity in Research Promotion of integrity in the research environment is about institu- tional culture and behavior, as well as the professional performance of individuals. It is about the system in which research is done. Research trainees learn about the culture of science through experience, mentoring, and formal educational processes. Even the best institutional climate and programs for researchers and trainees, however, will not preclude either research misconduct or nonprofessional behavior. Still, an educational environment that makes clear what is expected of scientists and their teams, combined with systems and institutional behaviors and policies that encourage accurate and careful pursuit of scientific ends, can benefi- cially influence researchers and trainees. The committee has defined integrity in research more broadly than an absence of research misconduct. Integrity in research embraces the aspirational standards of scientific conduct rather than simply the avoid- ance of questionable practices. There is a role for education of students, faculty, and staff regarding not only professional behavior but also the common culture of science that as a whole promotes a research environ- ment of high integrity. It can focus on the joy of rightful discovery and recognition by oneâs peers. Such education should be viewed in its broad- est sense, however, occurring not only through formal instruction but also through the institutional atmosphere, policies, and guidelines, as well as through the quality of mentoring (King, 1999; Swazey et al., 1993). A Role for Institutional Accrediting Bodies The committee believes that assessment of the effectiveness of institu- tional efforts to ensure integrity in the research environment can best be accomplished by incorporating evaluations of integrity in research into existing accreditation processes for institutions of higher education. Ben- efits that flow from the systematic evaluation of institutional behavior and policies associated with a process of self-assessment and accredita- tion include the following (Ewell and Lisensky, 1988): â¢ highlighting a need for change without impairing institutional au- tonomy or uniqueness;
116 INTEGRITY IN SCIENTIFIC RESEARCH â¢ efficient coordination of data gathering; â¢ creation of a permanent capacity for analyzing institutional effec- tiveness; and â¢ changing of the attitudes of faculty and staff regarding institu- tional assessment from negative to positive. Although peer review of the self-assessment process used to ensure integrity in the research environment could be accomplished by establish- ing a new accrediting body for the purpose the committee believes this is not the preferred approach. A more attractive alternative is for research institutions to work with established accrediting bodies to incorporate research integrity into overall accreditation processes. The processes of established accrediting bodies should be more effective and more cost- efficient than those of a new entity, whose establishment would constitute one more administrative burden, and thus would encourage cynicism. Moreover, creating a separate body for assessment could easily commu- nicate an undesirable message that an environment that enhances integ- rity in research can readily be distinguished from one that promotes high- quality education and research more generally. The committee believes that the research mission should be considered as a whole. Thus, it seems reasonable that entities charged with accrediting the quality of education at institutions of higher learning that conduct scientific research might also be charged with reviewing the data from the institutionsâ self- assessment of their climate and procedures for promotion of integrity in research. The committee is aware, however, that adoption of this recom- mendation must begin with a substantial commitment from the institu- tions themselves. Accrediting bodies respond to priorities established by schools and universities in determining the issues to be addressed in the process of accreditation. Consequently, if institutional cultures are to be enhanced, then both the call for change and its implementation must come from research institutions. An important next step will be for uni- versities and university associations, working together, to acknowledge the importance of conducting research and education in research in an environment of high integrity. Certainly the strong governmental and public interest in research integrity will prove ample encouragement to support initiating the peer review process, Universities and research sponsors should urge accrediting bodies, especially those charged with accrediting education programs with a sub- stantial research mission, to include an evaluation of the environment for promotion of integrity in research in the overall processes of accredita- tion. This objective might first be accomplished through collaboration with specialized accrediting bodies for science-based professions, such as medicine and engineering. Although not always realized (Bayles, 1981), a
EVALUATION BY SELF-ASSESSMENT 117 hallmark of the concept of professionalism and an integral factor in the social contract of professions with society is a presumption of integrity and self-policing (ABIM, 2001; Camenisch, 1996). Schools of engineering have led in explicitly tying this responsibility to accreditation require- ments. Recently, the Accreditation Board for Engineering and Technol- ogy has established criteria which mandate that graduates demonstrate an understanding of professional and ethical responsibility (Engineering Accreditation Commission, 2000; Rabins, 1998). Additionally, research in schools of medicine is largely supported by the U.S. Department of Health and Human Services, in which the Office of Research Integrity (ORI) resides. Thus, if ORI were to support the development of pilot programs to evaluate this approach, the Liaison Committee on Medical Education, which is charged with accreditation of programs for medical education (LCME, 2001), and the Accreditation Board for Engineering and Technol- ogy might be effective places to begin. The heterogeneity of the accreditation process among major accredit- ing bodies is acknowledged. For example, the process of self-study lead- ing to accreditation of research universities may be less prescriptive than that for professional schools in regard to the specific areas mandated for evaluation. Universities are commonly allowed substantial leeway and choice in determining the elements examined in their self-assessment pro- cess. Accrediting bodies, however, respond to the demands of their client communities. Thus, if institutions of higher education regarded the integ- rity of research as an element essential for their accreditation, then it is likely that assessments of integrity in research would be incorporated into both the self-study and peer-review phases of accreditation. It should also be noted that self-assessment of an institutionâs envi- ronment for integrity in research does not depend inextricably upon for- mal ties to a process for institutional accreditation. The need for indepen- dent self-assessment has already been recognized by a number of prominent institutions absent a mandate (Center for Academic Integrity, 2001). Other mechanisms for peer review can be developed, but the com- mittee concludes that explicit and public processes for external peer re- view help to ensure credibility and public confidence. In institutions where accreditation is not available or cannot for some reason be incorpo- rated into institutional processes of accreditation, other approaches to the provision of external validation should be explored. This is particularly important in the large number of private research institutes and indus- trial research groups that offer formal programs of education in research at either the predoctoral or the postdoctoral level. By encouraging accrediting bodies to develop their own designs for review of research integrity, the committee anticipates improvements and a confluence of processes over time as these bodies learn from each other
118 INTEGRITY IN SCIENTIFIC RESEARCH and from feedback from the institutions. Although this will be an evolu- tionary process, it could begin with, and be facilitated by, such agencies as ORI. Implementation should take place as quickly as possible, given the increasing demands for accountability in the use of public funds. The committeeâs focus on the quality of an environment that pro- motes integrity in research within institutions of higher education is not to imply disinterest in such integrity in other contexts. However, virtually all investigators begin their research careers in a university setting; there- fore, the university research group can be considered the crucible for education in research. Additionally, as the principal recipients of public research funds, academic research groups have been the major focus of concern for integrity in research. Self-assessment as a first step in the accreditation of educational pro- grams is predicated on a judgment that the process can lead to important changes in the environment. Experience with the process in other facets of the institutional milieu supports this notion. Desirable changes have suc- cessfully been introduced in many universities, in such areas as personal harassment, teaching about sexuality in medical schools, treatment of minorities, addition of instruction in ambulatory care to medical educa- tion, and addition of instruction in ethics to law school and business school curricula. The impetus for these changes has come from within the university community and has occurred as a consequence of government initiatives. A more difficult challenge, but one that warrants a substantial research effort, will be to determine, as evidence of effectiveness, whether long-term changes in behavior have been achieved as a consequence of any interventions intended to promote integrity in research coupled with processes for self-evaluation (Davis et al., 1999; Parochka and Paprockas, 2001). A Role for Governments Governments have an appropriate interest in the effectiveness of pro- grams in self-assessment and accreditation, since they rely on such pro- grams in a number of ways. For example, state licensing boards for many professions require graduation from an accredited school as a condition of licensure. They can also influence curricula through their licensing powers. Similarly, only accredited schools of medicine are eligible to re- ceive federal education grants and to participate in federal loan programs. Additionally, federal agencies sometimes review and approve the accred- iting bodies themselves. Such a review, based on a set of federally deter- mined standards, becomes a condition for accepting the findings of these bodies in determining eligibility for the commitment of federal funds (Middle States Commission on Higher Education, 2000). Thus, the federal
EVALUATION BY SELF-ASSESSMENT 119 government may oversee accreditation efforts that are based on private- sector self-assessment and peer review, even though government does not participate directly in the review and accreditation process. These government policies, which restrict recognition of graduates and exten- sion of certain educational programs exclusively to accredited schools, are well accepted by the higher education community. The nongovern- ment coordinating agency for accreditation of postsecondary education is the Council on Higher Education Accreditation (CHEA, 2001). The U.S. Department of Education maintains a listing of recognized regional, na- tional, institutional, and specialized accrediting bodies (DoEd, 2001). Since the nationâs economy and national security, as well as the health of its people, are heavily dependent on continuing reliable research find- ings, the federal government and the public rightfully place a high prior- ity on integrity in research. The federal government deserves support in its call for more effective strategies to encourage changes in the environ- ment for the promotion of integrity in research and the evaluation of the outcomes of such changes. However, the committee judges that a direct role per se for the federal government in such programs that assess and accredit institutions for their integrity in research is neither necessary nor desirable. The committee believes instead that self-assessment coupled with peer review of the environment for the promotion of integrity in research is more likely to have a positive impact on those programs and cultures. Funding Funding is needed in several important areas: to support research aimed at developing new methods for fostering integrity in research (in- cluding research on assessing the effectiveness of such approaches), and to support the everyday operation of assessment programs already in place at numerous institutions. The federal government (through the Of- fice of Research Integrity and various research agencies) and private re- search foundations can play a role in supporting these efforts. One way for them to encourage this process would be to augment grant programs to provide support for research into the enhancement of integrity in re- search and assessment of the effectiveness of alternative approaches. In principle, the costs of conducting programs to gauge the integrity of research conducted under federal sponsorship could be covered by the reimbursement of facilities and administrative costs (so-called indirect costs) associated with federal research grants and contracts. However, administrative costs on grants and contracts to educational institutions (but not to other research entities) have been capped for more than a decade at 26 percent of direct costs (Goldman and Williams, 2000; OMB,
120 INTEGRITY IN SCIENTIFIC RESEARCH 2000). When coupled with the additional fact that most research-intensive universities at present have documented administrative costs in excess of 26 percent (personal communication, T. DeCrappeo, Council on Govern- ment Relations, March 18, 2002), the consequence is that universities alone bear any additional costs associated with the development or enhance- ment of programs that evaluate integrity in research. If further encour- agement of an environment for integrity in research is truly a priority for research sponsors, then the sponsors should work cooperatively with edu- cational institutions to share in the funding of such programs, particu- larly if the intent is to develop best practices rather than simply require minimal compliance with applicable regulations and policies. A Role for Professional and Scientific Societies Professional and scientific societies have a key role in developing, promoting, and inculcating codes of research ethics within their member- ships. The common culture of science within academic departments and professional and scientific societies provides the peer pressure that pro- motes professionalism. To date, however, the activities of these societies related to these efforts have been limited. Among their actions, professional and scientific societies should ex- amine the requirement for membership and standing relative to integrity. Are there organizational sanctions that apply to members who have been shown to engage in misconduct, for example? In addition, the organiza- tions should ensure that content relative to the responsible conduct of research is included at their annual meetings, in their journals and other publications, and in other organizational venues for communication. Well-crafted society codes of research ethics (ACS, 2001; ASBMB, 2001) could be used as guides in the development of both specific objec- tives and processes for the development of institutional self-assessment programs. Each scientific society should express its standards for the re- sponsible conduct of research and take steps to ensure that its members know these standards. It is important to note that major research univer- sities are homes to a large number of different disciplines. The issues should be regarded as equally important across the entire spectrum of general and discipline-specific scientific societies, including societies in- volved in the life and earth sciences; social sciences; chemical and physi- cal sciences; and mathematical, computational, and computer sciences. INDIVIDUAL SELF-ASSESSMENT Self-assessment for accreditation is an institutional process. It may involve global accreditation of the educational environment and programs
EVALUATION BY SELF-ASSESSMENT 121 or evaluation of particular institutionally imbedded structures, such as programs for evaluation of animal care and use or the protection of hu- man research subjects. A second useful level of self-evaluation is directed at the individual researcher or academic unit as an initial step in periodic appraisals of performance. Many institutions and departments have for- mal processes for assessment of faculty members and academic leaders (deans and chairs). Faculty members are usually evaluated annually. For deans and chairs the interval is usually longer, and the evaluation serves as a component of in-depth school or departmental reviews. The academic community accepts these processes on the basis of a presumption that they promote and reflect individual, departmental, and institutional excellence. Department chairs commonly regard annual self- evaluation of faculty members, followed by a formal discussion with the chair, as an important aspect of faculty mentoring. The committee con- curs, and it advocates the inclusion of questions in the self-assessment process that evaluate behaviors that promote integrity in research. Fac- ulty members should be asked to evaluate not only their own behaviors as researchers but also interpersonal relations and their research environ- ments. Similarly, as chairs and deans are formally evaluated, questions in self-assessment forms should address the extent to which their leadership within the institution promotes a culture of integrity in research. Re- sponses to such questions could then be incorporated more formally into institutional self-assessment as a component of the institutional accredita- tion process. There are few data regarding the effectiveness of individual self-as- sessment in altering behavior, particularly behavior that is deviant or lacking in integrity, and this is another area in which research is needed. However, individual self-assessment does provide a useful framework for a formalized process of evaluation by institutional superiors. Addi- tionally, such a process may have salutary effects, at least in terms of interpersonal relationships within a laboratory or department, particu- larly when it includes input from subordinates to the individual being evaluated. SUMMARY â¢ Evaluation of the institutional environment for the promotion of integrity in research should be based on processes of self-assessment and peer review. â¢ Evaluation of integrity in research should be incorporated into ex- isting processes for accreditation of educational and research institutions. Creation of a new entity specifically for accreditation of an environment
122 INTEGRITY IN SCIENTIFIC RESEARCH that promotes integrity in research is probably unnecessary; it would be costly and likely impose burdens disproportionate to the benefits. â¢ Assessment of integrity in research for research groups should be a component of regular performance appraisals for faculty and academic leadership. â¢ Effective self-assessment will require the development and valida- tion of evaluation instruments and measures. â¢ Federal research agencies and private foundations should support the development of programs to integrate self-assessment of the environ- ment for integrity in research into accreditation processes through inter- actions with and among stakeholders, and they should fund research into the effectiveness of such programs. â¢ Federal research sponsors should work with educational institu- tions to develop funding mechanisms to support programs devoted to promoting the responsible conduct of research. REFERENCES ABIM (American Board of Internal Medicine). 2001. Project Professionalism. [Online]. Avail- able: http://www.abim.org /pubs/default.htm [Accessed January 22, 2002]. ACS (American Chemical Society). 2001. Academic Professional Guidelines. [Online]. Avail- able: http://chemistry.org/portal/servlet/resources/org/chemistry/avercom/dis- play/ContentRetrievalServlet/ACS/ACSContent/careers/apg_Jan01_final.pdf [Ac- cessed November 5, 2001]. ASBMB (American Society for Biochemistry and Molecular Biology). 2001. ASBMB Code of Ethics. [Online]. Available: http://www.asbmb.org/ASBMB/ASBMBSiteII.nsf/ MenuHomePage/PublicAffairs [Accessed November 5, 2001]. Bayles MD. 1981. Professional Ethics. Belmont, CA: Wadsworth Publishing Co. Borden VMH, Owens JLZ. 2001. Measuring Quality: Choosing Among Surveys and Other As- sessments of College Quality. Washington, DC: American Council on Education. Camenisch PE. 1996. The moral foundations of scientific ethics and responsibility. Journal of Dental Research 75:825â831. Center for Academic Integrity. 2001. Academic Integrity Assessment Guide. [Online]. Avail- able: http://www.academicintegrity.org/assessGuide.asp [Accessed November 5, 2001]. CHEA (Council on Higher Education Accreditation). 2001. Good Practice Database. [Online]. Available: http://www.chea.org/good-practices/index.cfm [Accessed March 18, 2002]. Davis D, OâBrien MAT, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. 1999. Impact of formal continuing medical education. Journal of the American Medical Associa- tion 282:867â873. DoEd (U.S. Department of Education). 2001. Overview of Accreditation. [Online]. Available: http://www.ed.gov/offices/OPE/accreditation [Accessed March 18, 2002]. Engineering Accreditation Commission. 2000. Criteria for Accrediting Engineering Programs. [Online]. Available: http://www.abet.org/images/Criteria/2002-03EACCriteria.pdf [Accessed March 18, 2002 ]. Ewell P. 1991. Benefits and Costs of Assessment in Higher Education: A Framework for Choice Making. Boulder, CO: National Center for Higher Education Management Systems.
EVALUATION BY SELF-ASSESSMENT 123 Ewell PT, Lisensky RP. 1988. Assessing Institutional Effectiveness. Washington, DC: Consor- tium for the Advancement of Private Higher Education. Goldman CA, Williams T. 2000. Paying for University Research Facilities and Administration. Santa Monica, CA: RAND. King PM. 1999. Why Are College Administrators Reluctant to Teach Ethics? Synthesis: Law and Policy in Higher Education 10(4):756-757. Asheville, NC: College Administration Publications, Inc. LCME (Liaison Committee on Medical Education). 2001. Accreditation Procedures. [Online]. Available: http://www.lcme.org/procedur.htm [Accessed March 18, 2002]. Middle States Commission on Higher Education. 2000. Designs for Excellence: Handbook for Institutional Self-Study, 7th ed. Philadelphia, PA: Middle States Commission on Higher Education. Pp. 1â4, 13â21. NAS (National Academy of Sciences). 1992. Responsible Science: Ensuring the Integrity of the Research Process. Panel on Scientific Responsibility and the Conduct of Research. Wash- ington, DC: National Academy Press. National Center for Higher Education Management Systems. 2001. Comprehensive Alumni Assessment Survey (CAAS). [Online]. Available: http://www.nchems.org/Surveys/ caas.htm [Accessed November 5, 2001]. OMB (Office of Management and Budget). Circular A-21, Cost Principles for Educational Insti- tutions. 2000. [Online]. Available: http://www.whitehouse.gov/omb/circulars/a021/ a021.html [Accessed March 18, 2002]. Parochka J, Paprockas K. 2001. A continuing medical education lecture and workshop, physician behavior and barriers to change. Journal of Continuing Education in the Health Professions 21:110â116. Rabins MJ. 1998. Teaching engineering ethics to undergraduates: Why? what? how? Science and Engineering Ethics 4:291â302. Swazey JP, Anderson MS, Lewis KS. 1993. Ethical problems in academic research. American Scientist 84:542â553.