Indicators for Monitoring Undergraduate STEM Education (2018) / Chapter Skim
Currently Skimming:
6 Existing Data Sources and Monitoring Systems
6 Existing Data Sources and Monitoring Systems
Pages 127-176
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 127... ...
The final section focuses directly on the committee's indicators, summarizing for each indicator current data sources, potential new data sources, and the research and data development that would be required to tap those potential sources for the purpose of ongoing monitoring of undergraduate STEM education. OVERVIEW Although many different postsecondary education data sources are available, they are limited in their ability to track students' progress into and through STEM programs and monitor the status of the committee's goals for undergraduate STEM: • Goal 1: Increase students' mastery of STEM concepts and skills by engaging them in evidence-based STEM educational practices and programs.
|
From page 128... ...
. TABLE 6-1 Major Sources of Data on Undergraduate STEM Education Coverage and Feasibility of Source Frequency Representativeness Disaggregation Federala IPEDS Annual Nationally Strong for race and representative; ethnicity, gender, institution mandatory so virtually type, and discipline; does 100% coverage not allow disaggregation for disability and Pell grant (socioeconomic)
|
From page 129... ...
Access may require intellectual property negotiations and fees. cHERI, Higher Education Research Institute.
|
From page 130... ...
The data are stripped of individual student identifiers, linked, and analyzed to help institutions and policy makers understand trends in student progression, completion, and outcomes. Some student unit record data systems also capture information on the demographic characteristics of students, grades, major areas of study, graduation rates, need based aid eligibility, and student earnings after graduation.
|
From page 131... ...
are often not structured to gather the information needed to understand the quality of undergraduate STEM education (National Academies of Sciences, Engineering, and Medicine, 2016)
|
From page 132... ...
The lack of nationally representative data on student trajectories through undergraduate STEM education results partly from policy decisions.
|
From page 133... ...
, IPEDS includes the universe of postsecondary institutions participating in federal student financial aid programs. In 2014, about 7,300 institutions complied with the mandate to respond, and an additional 200 institutions that did not participate in federal financial aid programs voluntarily provided data (National Center for Education Statistics, 2014)
|
From page 134... ...
Graduation Rates The graduation data cover the initial cohort of full-time, first-time, degree- and certificate-seeking undergraduate students at 2-year and 4-year institutions; the number of those students who complete their degrees or certificates within 150 percent of the normal time (i.e., 3 years or 6 years) ; and the number of those students who transferred to other institutions.
|
From page 135... ...
NCES designed the BPS survey to sample institutions and students within institutions. The agency used multiple methods to obtain responses, yielding strong response rates and nationally representative data.
|
From page 136... ...
In addition, more frequent data collection would allow the indicators to be updated annually, rather than only once every 3 years. The National Survey of Postsecondary Faculty Nationally representative data from 4-year institutions on undergraduate STEM instructors were formerly available from the NCES National Study of Postsecondary Faculty (Cataldi, Fahimi, and Bradburn, 2005)
|
From page 137... ...
Perhaps more importantly, the creation of this database has helped the Department of Education develop the technical capacity to obtain student unit record information from institutions and maintain that information in a central database. The department could potentially apply this technical capacity and expertise if the current legal ban on creation of a national student unit record database were overturned.
|
From page 138... ...
State higher education governing boards often manage these unit record data systems, using them to respond to questions from state policy makers that are not easily answered by other datasets. For example, these data systems can provide state-level information about the effect of policies (e.g., remedial and developmental education reforms, transfer policies)
|
From page 139... ...
Responding to two researchers' open letter about the quality of the data, the National Student Clearinghouse (2016a) posted the following statement on its Website: "The accuracy and completeness of the data can realistically be measured only by the institutions themselves." In a review of NSC data, Dynarski, Hemelt, and Hyman (2013)
|
From page 140... ...
In addition, the uneven reporting of all data elements poses a challenge to disaggregation of the data by demographic characteristics, such as racial and ethnic group, for the proposed indicators of equity, diversity, and inclusion. Higher Education Research Institute Surveys The Higher Education Research Institute (HERI)
|
From page 141... ...
Faculty Survey With the suspension of the NCES Survey of Postsecondary Faculty in 2004 (see above) , researchers and policy makers have increasingly relied on the HERI Faculty Survey.
|
From page 142... ...
However, the teaching practices it includes do not necessarily reflect the evidencebased STEM educational practices identified in the relevant research. In addition, it is based primarily on full-time faculty at nonprofit 4-year institutions and thus is not nationally representative of all types of faculty at both 2-year and 4-year institutions.
|
From page 143... ...
According to the NSSE Website, in 2016, student response rates within participating institutions ranged from 5 to 77 percent, with an average of 29 percent.5 In a recent study, Fosnacht and colleagues (2017) used data from NSSE administrations between 2010 and 2012 to simulate the effects of low response rates and low respondent counts.
|
From page 144... ...
MONITORING SYSTEMS The committee was not able to locate any existing systems that are designed specifically for monitoring the status and quality of undergraduate STEM education. However, it did identify existing monitoring systems that include elements relevant to undergraduate STEM, as discussed below.
|
From page 145... ...
For postsecondary education, the enrollment data include the number enrolled in STEM relative to other degrees; change over time in the number of undergraduate degrees conferred; demographic characteristics of students enrolled in STEM fields, including citizenship status; and the number of students enrolled in 2-year institutions by demographic characteristics. These statistics are tabulated from the IPEDS fall enrollment survey.
|
From page 146... ...
. Measures similar to these have been used by groups such as Complete College America and Achieving the Dream to gather institutional survey data and student unit record data from many public 2-year and 4-year
|
From page 147... ...
One of these measures overlaps to some degree with the committee's Indicator 3.1.1: Completion of foundational courses including developmental education courses to ensure STEM program readiness. Progress Metric 3: Success in Gateway (first-year)
|
From page 148... ...
Indicator 1.1.1: Use of Evidence-Based STEM Educational Practices in Course Development and Delivery Data Available and Potentially Available Currently, few data are available on the extent to which educators (namely, faculty members, graduate student instructors, adjunct instructors, or others) throughout the nation use evidence-based practices in course development and delivery.
|
From page 149... ...
TABLE 6-2 Research Needs and Potential Data Sources for the Proposed Indicators Needs for Research and Modification of Data Collection Instruments and/ Indicator Sourcea or Systems Needs for Coverage Improvements 1.1.1 Use of Evidence-Based HERI Faculty Survey Research to more clearly define Include 2-year institutions; more Educational Practices in Course evidence-based STEM educational systematic inclusion of graduate Development and Delivery practices; develop and add items teaching assistants NSOPF Research to more clearly define Expand number of institutions evidence-based STEM educational sampled to allow more granular practices; renew survey and develop disaggregation and add items Faculty Survey of Student Research to more clearly define Include more 2-year institutions Engagement evidence-based STEM educational and increase response rates at practices; develop and add items participating institutions 1.1.2 Use of Evidence-Based None Research to more clearly define STEM Educational Practices evidence-based STEM educational Outside the Classroom practices outside the classroom 1.2.1 Extent of Instructors' HERI Faculty Survey None Include 2-year institutions; more Involvement in Professional systematic inclusion of graduate Development student instructors NSOPF Renew faculty survey and review Expand number of institutions professional development items from sampled to allow more granular the 1988 department chairperson disaggregation survey for possible inclusion 1.2.2 Availability of Support or None Research to identify and clearly define Incentives for Evidence-Based key supports or incentives Course Development or Course Redesign continued 149
|
From page 150... ...
TABLE 6-2 Continued 150 Needs for Research and Modification of Data Collection Instruments and/ Indicator Sourcea or Systems Needs for Coverage Improvements 1.3.1 Use of Valid Measures of None Research to identify valid measures of Teaching Effectiveness teaching effectiveness 1.3.2 Consideration of None Research on how to measure Evidence-Based Teaching consideration of evidence-based in Personnel Decisions by teaching Departments and Institutions 2.1.1 Institutional Structures, None Research to identify and define key Policies, and Practices That structures, policies, and practices; Strengthen STEM Readiness for components could be added to IPEDS Entering and Enrolled Students on the basis of this research 2.1.2 Entrance to and BPS None Expand number of institutions and Persistence in STEM students sampled to allow more Educational Programs granular disaggregation HERI Freshman Survey and None Incorporate 2-year institutions in NSC Freshman Survey; increase coverage of students' academic programs in NSC data provided by institutions 2.1.3 Student Participation HERI First College Year/ Research to more clearly define Include public and 2-year in Evidence-Based STEM Senior Survey evidence-based educational practices; institutions and universities Educational Practices develop and add items NSSE Same as above Expand sample of 4-year institutions CCSSE Same as above Increase coverage of 2-year institutions
|
From page 151... ...
Diversity of STEM IPEDS Add departmental discipline None Graduate Student Instructors in Comparison with Diversity of HERI Faculty Survey None Include 2-year institutions; more STEM Graduate Students systematic inclusion of graduate teaching assistants 2.4.1. Students Pursuing STEM HERI Diverse Learning None Expand coverage of 2-year and Credentials Feel Included and Environments Survey 4-year institutions Supported in their Academic Programs and Departments continued 151
|
From page 152... ...
Institutional Practices None Are Culturally Responsive, Inclusive, and Consistent across the Institution 3.1.1 Completion of BPS 04/09 Research to more clearly define Expand number of institutions and Foundational Courses, including developmental and foundational students sampled to allow more Developmental Education courses granular disaggregation Courses, to Ensure STEM Program Readiness 3.2.1 Retention in STEM BPS 04/09 None Expand number of institutions and Programs, Course to Course students sampled to allow more and Year to Year granular disaggregation HERI Freshman Survey and None Incorporate 2-year institutions in NSC Freshman Survey; increase coverage of students' academic programs in NSC data provided by institutions 3.2.2 Transfers from 2-year NSC None Increase coverage of students' to 4-year STEM Programs in academic programs in data Comparison with Transfers to provided to NSC by institutions All 4-year Programs
|
From page 153... ...
NOTES: BPS, Beginning Postsecondary Students Longitudinal Study; CCSSE, Community College Survey of Student Engagement; HERI, Higher Education Research Institute; IPEDS, Integrated Postsecondary Educational Data System; NSC, National Student Clearinghouse; NSOPF, National Survey of Postsecondary Faculty; NSSE, National Survey of Student Engagement. aRefer to Table 6-1.
|
From page 154... ...
and the more recent Classroom Observation Protocol for Undergraduate STEM (Smith et al., 2013) require trained experts who analyze either videotapes or actual instruction using protocols that describe various teaching practices.
|
From page 155... ...
Thus, neither FSSE nor the HERI Faculty Survey provides sufficient information for this indicator from full-time community college instructors, as well as part-time instructors at 2- and 4-year colleges and universities. NSF requested funding for fiscal 2017 to re-institute the National Survey of Postsecondary Faculty in partnership with NCES (National Science Foundation, 2016, p.
|
From page 156... ...
Alternatively, the federal government could contract with one or more of the existing survey organizations to include new elements in their existing surveys and administer the revised surveys to nationally representative samples of instructors at 2-year and 4-year institutions. Indicator 1.1.2: Use of Evidence-Based Practices Outside the Classroom Data Available and Potentially Available No nationally representative data are currently available on the extent to which students in 2-year and 4-year institutions participate in evidencebased educational practices outside the classroom.
|
From page 157... ...
Data and Research Needs Data and research needs in the area of faculty surveys are discussed above, under indicator 3.1.1. Indicator 1.2.2: Availability of Support or Incentives for Evidence Based Course Development or Course Redesign Data Available and Potentially Available There are no national data currently available on the extent of support available to faculty for evidence-based course development or redesign.
|
From page 158... ...
or in a revived National Survey of Postsecondary Faculty. Indicator 1.3.1: Use of Valid Measures of Teaching Effectiveness Data Available and Potentially Available The committee was not able to find any existing national data on the extent to which 2-year and 4-year institutions use valid measures of teaching effectiveness to measure instructors' performance.
|
From page 159... ...
Indicator 2.1.1: Institutional Structures, Policies, and Practices That Strengthen STEM Readiness for Entering and Enrolled College Students Data Available and Potentially Available Currently, no existing data source provides information about the prevalence of programs and practices that can strengthen STEM readiness for U.S. college students.
|
From page 160... ...
Data and Research Needs The nationally representative longitudinal surveys administered by NCES face challenges with respect to frequency of data collection and coverage of data. Without more frequent data collection for new cohorts, policy makers may wait years for an updated indicator of the status of diverse students' entrance to and persistence in STEM academic programs.
|
From page 161... ...
, including the National Survey of Student Engagement, the Community College Survey of Student Engagement, and HERI's Your First College Year and College Senior Survey. Although comprehensive in their collection on the extent of students' engagement with those practices, the surveys do not focus on evidence-based STEM educational practices and programs as defined by the committee.
|
From page 162... ...
Alternatively, the federal government could contract with one or more of the existing survey organizations to include new elements in their existing surveys and administer the revised surveys to nationally representative groups of 2-year and 4-year institutions. Indicator 2.2.1: Diversity of STEM Degree and Certificate Earners in Comparison with Diversity of Degree and Certificate Earners in All Fields Data Available and Potentially Available The IPEDS Completion Survey can provide data to measure this indicator.
|
From page 163... ...
Although NSC data appear promising, they have significant limitations with respect to the availability of demographic characteristics; for example, they have limited coverage of students' gender, race and ethnicity, and Pell status.8 Thus, new data or changes in the data voluntarily submitted by institutions to NSC would be required before the data could be used to inform this indicator. Indicator 2.2.3: Time-to-Degree for Students in STEM Academic Programs Data Available and Potentially Available Only limited data are currently available for this indicator.
|
From page 164... ...
(As noted above, NCES has discontinued the National Study of Postsecondary Faculty.) As discussed above, the HERI Faculty Survey represents the most comprehensive and representative data source on postsecondary faculty, but those data have several limitations.
|
From page 165... ...
Several national surveys (e.g., the HERI Faculty Survey and the Faculty Survey of Student Engagement administered by Indiana University) contain measures of teaching practices, satisfaction, and, in the case of HERI's survey, sources of stress.
|
From page 166... ...
Similarly, there is no source with comprehensive data on the type of training and professional development available to faculty search committees, nor the extent to which search committees engage in this training or development. Data and Research Needs Additional research is needed to identify and establish common definitions of what instructional and educational practices qualify as being culturally responsive.
|
From page 167... ...
. A national student unit record data system could address the challenge that BPS is administered only periodically.
|
From page 168... ...
More frequent data collection for new cohorts is needed to provide an updated indicator of progress toward the objective of successful navigation into and through STEM programs of study. One alternative with different challenges would be data from HERI's Freshman Survey, matched with NSC's enrollment and completion data.
|
From page 169... ...
Indicator 3.3.1: Percentage of Students Who Attain STEM Credentials over Time, Disaggregated by Institution Type, Transfer Status, and Demographic Characteristics Data Available and Potentially Available The IPEDS Completion Survey provides comprehensive coverage of the credentials that students earn and the discipline or field of study associated with those credentials for all postsecondary institutions that receive Title IV funding.11 Data are collected at 100 percent, 150 percent, and 200 percent of expected completion time and are broken down by race and gender. The system provides information on the number and type of credentials (certificates, associate's degrees, and bachelor's degrees)
|
From page 170... ...
SUMMARY AND CONCLUSIONS The committee considered whether, and to what extent, various federal and proprietary data sources are representative of the national universe of 2-year and 4-year institutions and included data relevant to the goals and objectives in the conceptual framework and, more specifically, to the committee's proposed indicators. Focusing first on federal data sources and systems, the committee reviewed the IPEDS, which includes high-quality, current data related to the committee's goals and objectives, including detailed annual data on completion of degrees and certificates in different fields of study.
|
From page 171... ...
CONCLUSION 3 To monitor the status and quality of undergraduate STEM education, recurring longitudinal surveys of faculty and students are needed. The committee found that IPEDS and other federal data sources generally allow data to be disaggregated by students' race and ethnicity and gender.
|
From page 172... ...
. However, these sources would need enhanced coverage of institutions and students to be nationally representative, along with additional data elements on students' fields of study.
|
From page 173... ...
. Feasibility of a Student Unit Record System Within the Integrated Postsecondary Education Data System.
|
From page 174... ...
. Improving undergraduate STEM education: The ef ficacy of discipline-based professional development.
|
From page 175... ...
Slakey (Eds.) , Transforming Institutions: Undergraduate STEM Educa tion for the 21st Century.
|
From page 176... ...
. Wash ington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.