Individual participants offered suggestions for future actions to further the conversation around quality in undergraduate education.
IMPROVING DATA GATHERING, SHARING, AND COMMUNICATING
Jessica Howell (College Board) suggested that institutions should make better use of faculty in departments of economics, education, and sociology who are trained to work with administrative datasets and can determine the causal connections between universities’ efforts to improve quality and student outcomes. Participants in one small-group discussion believe that accrediting bodies require institutions to collect a lot of data that are not relevant to true measures of quality and advocated for influencing these bodies to require data that are more meaningful, in quality terms. Several participants, including Emily Slack (Education and Labor Committee, U.S. House of Representatives), suggested that further studies on how to encourage accreditors to focus more on student learning outcomes and less on input factors would be a valuable outcome of this workshop.
Individual participants discussed how institutions should communicate about educational quality to the general public and to policymakers. Participants in one small-group discussion highlighted what they determined to be the need for institutions to identify the key proficiencies that students acquire through their undergraduate experiences (both those that are general in nature and those specific to a discipline), ensure that these are clearly stated, and clearly connect them to the outcomes of an individual student. Scott Ralls (Northern Virginia Community College) stressed the importance of communicating in a focused way to each of the many different “publics.” He encouraged institutions not to assume that posting a Collegiate Learning Assessment (CLA) score on their website is contributing to either transparency or improved decision-making on the part of students, parents, or employers. Elsa Núñez (Eastern Connecticut University) believes that how an institution changes over time, adapting to the
realities of the student population and the external environment, is important to “creating a story the public can understand.”
Cliff Adelman (Institute for Higher Education Policy) suggested that the U.S. Department of Education Baccalaureate and Beyond Longitudinal Study1 be dramatically expanded by adding questions about students’ activity such as civic participation and cultural participation and by administering it to 500,000 people.
Several participants suggested that co-curricular transcripts and diploma supplements can communicate students’ experiences and abilities beyond the classroom. Andy MacCracken (National Campus Leadership Council) advocated for co-curricular transcripts as a way to bridge the divide perceived by students between what is valued in the classroom versus out of the classroom. This transcript would accompany the traditional transcript and list out-of-classroom experiences. One challenge, he noted, is in determining how the university would validate the student’s transcript as an indicator of success in the workplace. Adelman described how diploma supplements might work, noting similarities to efforts in Europe that focus on the institution rather than the student. A diploma supplement might include a description of the senior project (certified by the chief academic officer); a summary of how the student contributed to the university or its surrounding communities—local, regional, national, international (certified by the dean of students); a summary of the student’s foreign language skills; and a listing of completed courses.
PUBLISHING BEST PRACTICES AND GUIDELINES
Participants shared several ideas about publications to assist institutions in improving the quality of the undergraduate education. Jennifer Engle (Bill & Melinda Gates Foundation) suggested production of a report oriented toward higher education’s consumers (i.e., parents and students) to inform their decision-making.
Individual participants suggested that a report of best practices would help faculty, deans, and other administrators to leverage the abundant knowledge that already exists about high-quality undergraduate education, as has been done on other subjects.2 Marco Molinaro (University of California, Davis) suggested a
1 See https://nces.ed.gov/surveys/b&b/.
2 For example, National Academies of Sciences, Engineering, and Medicine. (2016). Promising Practices for Strengthening the Regional STEM Workforce Development Ecosystem. Washington, DC: The National Academies Press. doi:10.17226/21894; National Research Council. (2011). Promising Practices in Undergraduate Science, Technology, Engineering, and Mathematics Education: Summary of Two Workshops. Washington, DC: The National Academies Press. doi:10.17226/13099; and National Academies of Sciences, Engineering, and Medicine. (2016). Barriers and Opportunities for 2-Year and 4-Year STEM Degrees: Systemic Change to Support Students' Diverse Pathways. Washington, DC: The National Academies Press. doi:10.17226/21739.
periodical volume that could be reviewed regularly and possibly tied to the College Scorecard as long as the Scorecard is focused on student learning. He cited the example of the Academies’ series on disciplinary-based educational research, in which a first report outlined the evidence while a second report functioned as a practitioner’s guide.3
STRATEGIES FOR IMPROVING QUALITY (EXAMPLES)
Several participants described efforts by specific institutions to improve the quality of their undergraduate education. James Kvaal (the White House) described the positive results obtained by City University of New York’s (CUNY’s) Accelerated Study in Associate Programs (ASAP) through combining counseling, full-time enrollment, tuition waivers, and active remediation.4 Graduation rates of CUNY ASAP cohorts are approximately double those of non-ASAP comparison group students. Although not inexpensive, the program costs less per graduate than traditional higher education. Kvaal cited work on boosting success rates in developmental education as also very important. He described the cognitive tutor model that employs technology to ensure that students are progressing. The data generated by this approach allows the instructor to randomize methods for teaching different concepts, observe what works best for student learning, and build in continuous improvement.
Sally Johnstone (Western Governors University) outlined the approach used by Western Governors University, an institution described by Adelman as having a clear-cut focus and mission. Many of its students, almost 70 percent of whom are Pell-eligible, did not succeed in traditional models of higher education. To define the appropriate competencies or proficiencies and the right learning outcomes, Western Governors University engages employers, external academics well-known in their field, and instructional designers. The university employs a system for continuous measurement of learning outcomes online, looking at what learning resources students are using and not using, what is predictive of student success, and where students are having problems. Through these processes, she described how the university can identify successful pathways or clusters of pathways through course materials and uses that information to improve student success.
Núñez described the decade-long process of quality improvement at Eastern Connecticut University, Connecticut’s only public liberal arts university, and
3 National Research Council. (2012). Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering. Washington, DC: The National Academies Press. doi:10.17226/13362. and National Research Council. (2015). Reaching Students: What Research Says About Effective Instruction in Undergraduate Science and Engineering. Washington, DC: The National Academies Press. doi:10.17226/18687.
how she learned what was needed to support the university’s claims of quality. The faculty’s philosophy, as well as hers, is one of engaged learning—that students learn best when they can apply their classroom learning in internships for credit, co-ops, or undergraduate research or community-based projects. The incentive for the faculty is release time, which she acknowledged is costly.
Since its efforts directed at engaged learning, the university has improved on 70 percent of the National Survey of Student Engagement’s (NSSE’s) 42 measures. It scores higher than Research I institutions5 on measures such as engaging in internships and learning communities and conducting undergraduate research with the faculty. However, because NSSE is self-reported data, Eastern Connecticut University administers the Collegiate Learning Assessment (CLA) to groups of freshmen and seniors to collect externally validated evidence of quality. Núñez highlighted faculty participation in the Current VALUE Project: Multi-State Collaborative and SHEEO (the State Higher Education Executive Officers' association).6 A pilot project begun in Massachusetts and involving nine states, the program is currently housed at the Association of American Colleges and Universities (AAC&U). The project measures broad intellectual skills as well as integrated applied learning and evaluates actual artifacts of students’ academic course work. Faculty have embraced the project because its rubric measures the artifacts from their courses. The rubric is evaluated by faculty from out-of-state institutions, which lends strong support to her funding requests to the state legislature and the university’s board. She believes that this project is the first in the nation to use a nationally recognized rubric and an external review process to assess student’s actual products.
PLACING THE QUALITY DISCUSSION IN AN INTERNATIONAL CONTEXT
Ellen Hazelkorn (Dublin Institute of Technology), policy advisor to the Irish government, noted that the discussion about quality in the United States resembles the one that is happening internationally. She cited efforts to assess quality and rank institutions in the United Kingdom, Australia (myUni), and the European Union (U-Multirank). A commonality among these efforts, in her view, is an increasing focus on the quality of teaching. She sees an international trend to move beyond issues of improving quality to measuring and comparing performance and productivity, and linking this information to resource allocation. Institutions in other countries are often compared on specific objectives, generally national social objectives.
5 There are 115 Research I: Doctoral Universities–Highest research activity, as measured by research expenditures, number of research doctorates awarded, number of research-focused faculty, and other factors, as of the 2015 Classification update. More information is available at http://carnegieclassifications.iu.edu/.
David Dill (University of North Carolina) mentioned “quality enhancement,” a term used widely outside of the United States to describe a regulatory procedure for evaluating the quality assurance of college and universities, particularly in England, Hong Kong, and several Scandinavian countries. He noted that what distinguishes quality enhancement regulatory procedures from U.S. accreditation is the focus on the internal processes and governance by which colleges and universities ensure, measure, and design academic courses and maintain the quality of the academic standards.
Andrew Crews, an architect, discussed the global marketplace for academic credentials. In his work in higher education and accreditation for architecture he has seen a surge in a number of non-U.S. institutions seeking accreditation as a mark of quality outside the United States. Conversely, as the leader of human capital strategy for a global architecture and design firm, he has also observed that students and people wishing to enter his field are “recognizing that a non-U.S. credential is frequently faster and cheaper.” He therefore encouraged the group to keep the global marketplace for academic credentials in mind during discussions of quality in undergraduate education.
LEARNING FROM OTHER INDUSTRIES
Howell suggested that the discussion about quality in education could benefit from the literature from other industries such as in health care. Industries in which consumers make high stakes choices, such as choosing hospital and doctors, are also grappling with defining, measuring, and communicating quality to end users. She cited some researchers at the University of California, Davis, who are looking at measures of hospital quality and modeling it in the context of higher education.
Another participant referred to an interesting model for measuring quality in health care that grew out of the Institute of Medicine’s (IOM’s) National Roundtable on Health Care Quality, which was convened in 1995. That roundtable of representatives from industry, academia, government, and other stakeholder groups was tasked with identifying a set of health outcomes to be measured.7 The IOM group determined the most reliable hospital- and doctor-level measures of quality and then curated the methodological development of risk-adjusted measures. These measures are now utilized in the pay-for-success matrix behind some reimbursement policies and used in the Affordable Care Act.
During a luncheon keynote speech, Nigel Croft, chairman of the International Standards Organization’s (ISO) Committee for Quality Systems, described concepts of quality across fields as they have evolved and been applied around
7 A report issued by the National Roundtable on Health Care Quality is: Institute of Medicine. (1998). Statement on Quality of Care. Washington, DC: The National Academies Press. doi:10.17226/9439.
the world. The ISO’s formal definition of quality is “the degree to which a set of inherent characteristics of an object fulfills requirements.” He highlighted common challenges between industry’s and education’s needs to define and achieve quality.
Croft noted that such characteristics can have very different dimensions that “may or may not fulfill the needs and expectations of the different interested parties.” In addition, different products have different characteristics, and institutions of higher education have multiple relevant interested parties, from students to employers. A common challenge for industry and higher education is how to substantiate claims for the products and services offered.
As an example of addressing the quality needs of a broad set of interested parties, Croft described the work of the technical subcommittee responsible for the ISO 9000 quality management system standard.8 The subcommittee of representatives from government, education, and industry in 83 countries focused on finding high-level, principles-based commonalities. From those high-level principles, the subcommittee drilled down to examine quality in specific contexts from business-to-consumer situations to more abstract contexts such as health care in which the transaction is not linear.
He emphasized the importance of two contexts: (1) the organization, that is, “What are the external and the internal factors that allow that organization to achieve its objectives?” and (2) the interested parties. He observed an absence of students at the workshop and encouraged participants to account for their needs and expectations.
Croft told participants that ISO is developing a new standard, ISO 21001, on management systems in educational organizations and invited their participation in this effort.9 The discussion is currently at a high level and generic, involving all of the ISO member bodies. The goal is to create a common, high-level structure from which people working in different educational contexts can drill down.
CONTINUING TO DO MORE WITH LESS
Gabriela Weaver (University of Massachusetts at Amherst) reminded the audience that faculty, and institutions themselves, are being asked to do more with less. She described the “big gap between knowing what [quality] is and having the ability to provide it, when your institution is dealing with decreased levels of state funding, federal funding, and research money and increased levels of accountability on metrics that it hasn’t been held accountable for in the past.” “The question of quality,” she said, “has to intersect with the question of institutions having the ability to support the faculty to do those things.” Roy Swift (Workcred) cautioned against leaving smaller institutions out of the
8 See http://www.iso.org/iso/home/standards/management-standards/iso_9000.htm.
9 See http://www.iso.org/iso/catalogue_detail.htm?csnumber=66266.
quality discussion; institutions lacking abundant resources may need assistance in collecting, storing, and analyzing the data.
CONVENING A WIDE RANGE OF INTERESTED PARTIES TO DEVELOP A FRAMEWORK FOR QUALITY
Individual participants described a desire to bring the evidence to bear on the problem of quality in undergraduate education to help a wide range of stakeholders, including institutional actors, policymakers, and business leaders, to understand why certain steps must be taken. Participants in one small-group discussions suggested that the Academies could take a lead role in the quality discussion as convener of the process. In that capacity, Jillian Kinzie (National Survey of Student Engagement and National Institute for Learning Outcomes Assessment) urged that inclusive excellence be an integral part of the Academies’ efforts and that the Academies might bring together different audiences to address this issue more directly.
Other participants suggested that the Academies would serve the nation well by establishing a framework on which quality models can be developed and that the Academies are well positioned to serve the role as integrator and synthesizer of data. Hazelkorn asked, “Is there a core set of principles to guide the development of the framework for measuring quality?”
Some participants identified the driver of quality metrics to be one important component of the framework. Some institutions may require assistance in following the thread from their missions to the detailed data that must to be collected to assess whether they are fulfilling those missions.
Regarding the potential role of accrediting organizations in improving quality in undergraduate education, a participant suggested that the Academies could help to shape reform how accreditation is conducted.
This page intentionally left blank.