Quality in the Undergraduate Experience What Is It? How Is It Measured? Who Decides? Summary of a Workshop (2016) / Chapter Skim
Currently Skimming:
Appendix C: Defining and Measuring Institutional Quality in Higher Education
Appendix C: Defining and Measuring Institutional Quality in Higher Education
Pages 57-80
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 57... ...
1 Rising tuition prices and student debt levels have increased the public's concern over whether college investments are worth the money, and recent revelations of fraud and deceptive practices by large college chains have shed light on the reality that not all universities contribute positively to students' success, and fueled calls for greater accountability for institutions receiving public funds. It may seem paradoxical that skepticism over the quality and value of postsecondary education has intensified in recent years, as the college earnings premium has risen over the past several decades and is currently near record levels.
|
From page 58... ...
Colleges and the students they serve have myriad and diverse goals, and many of these are intangible and not readily subject to measurement or quantification. But this is changing: new administrative data is increasingly available through the efforts of federal and state governments, private data collections, and institutional consortia that link various student outcomes to the educational institutions they attended.
|
From page 59... ...
DEFINING QUALITY IN HIGHER EDUCATION In a 2001 report, Crossing the Quality Chasm, the Institute of Medicine (IOM) proposed a definition for health care quality that, suitably adapted, provides a useful starting point for defining quality in higher education: "The degree to which education services increase the likelihood of desired education outcomes." The heart of this formulation is that quality is defined in terms of the causal impact that exposure to some educational experience (e.g., attending college A, or studying engineering at college B)
|
From page 60... ...
It may be that graduation rates convey quality information indirectly, insofar as students more satisfied with the quality of their education may be more likely to complete it, but the existence of this link should not be taken for granted. Several other aspects of this definition deserve mention.
|
From page 61... ...
It is natural to view such a measure as a weighted average formalism, we can measure institution i's quality, π₯π₯ ππ, as of these J x K measures of institutional quality. Restating this with just a bit of π₯π₯ ππ = β ππ β ππ π€π€ ππ ππ πΏπΏ ππ ππ ππ where πΏπΏ ππ ππ ππ represents the causal impact of attending institution i on outcome j for students of type k, and π€π€ ππ ππ is the weight assigned to that outcome in constructing the overall measure of quality.
|
From page 62... ...
, and teacher quality measures in K-12 education take into account the prior achievement levels of the students they serve, quality measures in higher education need similar kinds of adjustments in order to isolate the causal impacts of institutions on students' outcomes.
|
From page 63... ...
The raw outcome differences can thus mislead prospective students about the quality differences between schools -- a more useful set of quality measures might be the predicted earnings of a student with average characteristics (e.g., family income, SAT scores, etc.) at each institution from a regression model.
|
From page 64... ...
Since such students are likely to have high graduation rates and labor market outcomes regardless of where they attend college, there is a tendency for more selective institutions to have better average student outcomes simply because they enroll more advantaged students. To disentangle this student "selection effect" from institutional quality effects, most studies in economics have relied on either multiple regression analysis (James, Alsalam, Conaty, and To, 1989; Loury and Garman, 1995; Brewer and Ehrenberg, 1996; Monks, 2000; Long, 2008)
|
From page 65... ...
Most of this work relies on aggregate institution-level outcomes, like graduation rates or average earnings, and adjusts those measures using aggregate institution-level student characteristics, such as median SAT scores, the fraction of students eligible for Pell grants, etc. In these studies, regression or matching techniques are used to estimate the predicted relationship between student and institution
|
From page 66... ...
To give a heuristic, if overly simplified, sense for how the technique works, Figure C-1 depicts the relationship between 6-year graduation rates for firsttime, full-time students and the percentage of students receiving Pell grants at 4year institutions based on IPEDS data. As might be expected, there is a strong negative correlation between the family income background of the student body and their graduation rates: for example, institutions where 80 percent of students receive Pell grants have graduation rates that are more than 35 percentage points lower on average than institutions where only 20 percent of students receive Pell grants.
|
From page 67... ...
. Estimates of college value-added based on these techniques yield dramatically different rankings of institutional quality when compared to raw graduation rates or average earnings.
|
From page 68... ...
A more important limitation is that it is likely that selection on the part of both students and college admissions offices leads to a correlation between institutional quality and aggregate student characteristics. To the extent this type of selection occurs, then the relationship between student characteristics such as the percentage receiving Pell grants and outcomes will reflect the impact of SES on outcomes, but also the higher quality of institutions with fewer poor students.
|
From page 69... ...
Finally, the Council of Economic Advisers (2015) used student-level information from students' FAFSA forms, including their parental education, family income, and the SAT scores of institutions to which they sent their FAFSA form to estimate regression-adjusted earnings measures for every degree-granting institution in the United States.
|
From page 70... ...
Cunha and Miller's results highlight the importance of accounting for differences in student characteristics when measuring institution quality. A student growing up near UT Pan-American choosing whether to go to school locally or to Texas A&M might make a very different college choice if he believed his earnings would be only 4 -- not 52 -- percent higher if he attended A&M.
|
From page 71... ...
Using these data she employs two complementary research designs aimed at eliminating the effect of two types of selection from college quality estimates. First, to isolate similar students who enroll at institutions with different selectivity, she identifies students "on the bubble" of admissions at each school by finding applicants to each school whose admission probability based on their SAT score is between 40 and 60 percent.
|
From page 72... ...
NEXT STEPS AND CHALLENGES As described above, only limited progress has been made in developing quality measures that have the desirable properties described above. Below, I comment on a set of next steps that seem necessary building blocks in the construction of better quality indicators and some challenges and considerations that will need to be addressed.
|
From page 73... ...
The number one reason to attend college offered by students in surveys is often related to improving their employment outcomes. There are several aspects of individuals' labor market outcomes that might be useful.
|
From page 74... ...
For some types of postsecondary training aimed at preparing individuals for a particular occupation it may be feasible to measure the work performance of alumni, and compare this performance across institutions. This is the logic, for example, behind recent proposed regulations on teacher preparation programs, which seek to tie eligibility for some federal funding to the student test scores of teachers from the program.
|
From page 75... ...
Many institutions conduct surveys of their students' satisfaction or engagement, and some of these measures have been shown to correlate with higher graduation rates. In their current incarnation, generally asked while students are enrolled or as part of an exit survey, these measures are probably most useful in helping assess what institutional practices are associated with quality (i.e., the institution's impact on outcomes)
|
From page 76... ...
This type of work could help identify, for example, whether individual-level data is critical, and what core sets of student characteristics need to be included in regression adjustment models for them to produce accurate quality measures. 16 Researchers also need to develop information about which institutional practices contribute most to quality to help inform improvement efforts.
|
From page 77... ...
One of the most pressing sources of demand for quality information about postsecondary institutions is to inform public policy efforts to ration public funds away from institutions with unsatisfactory outcomes. For example, the Higher Education Act proscribes participation in Title IV financial aid programs for institutions with high default rates on federal loans, and the Gainful Employment regulations do the same for vocational training programs whose graduates have high debt-to-earnings ratios.
|
From page 78... ...
This begs the question of what level should quality information be constructed to be most useful for informing college choice or other purposes. For example, some states have begun to report outcomes data, such as average earnings, for graduates separately by program of study.
|
From page 79... ...
. "Measuring the impact of teachers II: Teacher value-added and student outcomes in adulthood." American Economic Review 104(9)
|
From page 80... ...
. "The Promises and Pitfalls of Measuring Community College Quality." University of California, Davis mimeo.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.