2

Rise Up to Meet 21st-Century Education Data Ecosystem Needs

The panel reimagines NCES as a leader in education statistics, evidence building, and data governance. The panel aspires for NCES to be in full control of how it meets its mission—operating strategically, anticipating environmental changes, and readily adapting to continue delivering high-value products and services. Finally, the panel envisions NCES as deeply engaged with stakeholders and a strong partner supporting evidence building within the Department of Education (ED).

MEET THE MISSION IN A CHANGING SOCIAL CONTEXT

Three dramatic demographic and social changes have affected NCES’s impact as a statistical agency. First, over the last few decades, the student population has grown more diverse in terms of race and ethnicity, sex and gender identity, disability status, and age. In parallel, the field of education has become more aware of the diversity of student needs and experiences, as well as the impact of institutions, instruction, and other factors on student outcomes and equity. The workforce and military are also experiencing rapidly changing training needs, as adults increasingly engage in continuing education or retraining. This has prompted practitioners, policy makers, and researchers to ask new questions, as advancing the field and understanding contemporary and emerging education issues require new types of information.

Second, recent decades have seen an explosion of data sources. Data production is no longer the sole province of a few large organizations. For example, more organizations and businesses are harnessing data collected

while doing business (e.g., administrative data, commercial data). States have developed administrative data systems with the support of NCES, in part through the Statewide Longitudinal Data Systems (SLDS) Grant Program. Other data sources are produced on the internet or electronically and, with new technologies and methods, can be transformed into analyzable data (e.g., “scraped” data and natural language processing of text documents). These data sources have provided unique insights, such as the use of data from charter school websites to study school choice and educational stratification (Haber, 2021). However, NCES has done little to grasp the opportunities created by this rapidly changing data and technology environment.

Third, contemporary policy makers, practitioners, and other stakeholders hunger for relevant and timely evidence to inform understanding and decision making. The momentum of recent years has been codified in the Foundations for Evidence-Based Policymaking Act of 2018 (commonly referred to as the Evidence Act), which establishes the commissioner of NCES as the chief statistical official of ED, who is expected to work closely with the chief evaluation officer and chief data officer to build evidence. Thus, NCES has many stakeholders, including Congress, the President, ED, and other federal agencies, along with state and local education agencies, school districts, policy makers, nonprofit organizations, academic researchers, and, of course, students and their families. While NCES may serve some stakeholders, such as academic researchers, adequately, the Center does not understand or serve all audiences equally well. This lack of broad stakeholder engagement intensifies the effects of the other two changes in the social context: the needs of the education system and the availability of non-NCES data sources to inform issues.

NCES’s relevance has declined simultaneously with these broad social changes. NCES has been seeking ways to stand out among all available data providers and to improve stakeholder engagement, with the long-term goal of making the Center’s “website and its data summaries the first stop for questions about the U.S. education system for audiences of all backgrounds and experiences” (NISS, 2021a, p. 6). Multiple presenters gave testimony on cutting-edge data innovations and education evidence-building projects that do not involve NCES, but where the Center could serve a constructive role (Appendix F). Further, NCES’s data products are often out of date upon release. For instance, data products released in late 2021 include 2-year-old provisional data on fiscal year 2019 revenues and expenditures for public school districts (NCES, 2021k), and a “First Look” product showing 3-year-old data on school year 2017–18 student financial aid (NCES, 2021a). Despite its declining relevance, over the last few decades NCES has advanced the field of education statistics with innovative data-collection approaches and rigorous statistical standards.

In studying the current situation, the panel’s core findings drive all its recommendations:

• NCES lacks an agency-level strategic plan or other systematic way to prioritize high-value products and services.
• NCES has not kept up with the changing and expanding needs of its stakeholders.
• NCES is neither leading nor seizing opportunities presented by the recent explosion of administrative data and other data sources that could provide new analytic insights.
• NCES is often left out of important discussions within ED and has a weak voice in important national conversations about education data, statistics, and evidence building.

These findings are broadly similar to those of previous review panels (NISS, 2016; NASEM, 1986). This similarity indicates that NCES needs transformative rather than marginal change to make true progress on ongoing issues. The conditions are better than ever to take action. The director of the Institute of Education Sciences (IES) requested this study, demonstrating an interest and commitment to support a new vision for NCES. The Evidence Act is a recent mandate that highlights and supports the role of statistical agencies in developing and facilitating access to data for evidence building. Particularly in the last 5 years, decision makers at every level of government have exhibited an increasing demand for useful data and statistics to help inform decisions. In fact, most states have created a chief data officer role to support this new emphasis on using data. With the availability of alternative data sources, NCES can stand out as a leader, particularly on data standards and quality. Although the conditions are ripe, transformative change always requires substantial, thoughtfully directed investment. The panel recommends that:

• NCES develop a strong strategic plan to make difficult tradeoff decisions (Recommendation 2-1);
• ED and IES support and empower NCES to set its own priorities (Recommendation 2-2);
• ED, IES, and NCES maximize the Center’s unique value for evidence building (Recommendation 2-3);
• NCES adapt to the changing world of education by increasing diversity and awareness of equity issues (Recommendation 2-4); and
• NCES expand data-acquisition strategies for new insights (Recommendations 2-5, 2-6).

These fundamental recommendations are discussed in further detail below (and additional recommendations are discussed in Chapters 3, 4, and 5). The panel recognizes that NCES is currently addressing some aspects of these recommendations and asks that NCES push further, to fully embody each recommendation. Since recent reports (e.g., NISS, 2016) may not have provided enough information to be actionable, the panel provides operational details to assist with implementation. The panel offers ideas and examples to illustrate implementation details, especially for projects conducted by other federal statistical agencies. The panel discusses its thoughts on strategic priorities, data products, services, and operations, yet understands that NCES needs to decide which options to adopt. Achieving these recommendations within NCES’s constraints will require planning, will, and discipline. If NCES achieves these fundamental recommendations, the panel is confident that the Center will make substantial progress towards the vision we offer and could take the helm as a meaningful leader in the nation’s education data ecosystem.

DEVELOP A STRONG STRATEGIC PLAN TO MAKE TOUGH DECISIONS

To meet its mission effectively, any government agency needs to understand its identity and core values, so that it can establish priorities to guide decision making and operations. By clarifying tradeoffs and greater goals, strategic planning helps agencies be proactive when making difficult decisions. The north stars of strategic foresight and planning are especially necessary for navigating today’s evolving environment, particularly when an agency’s mission changes. Not only has the social environment for education statistics changed, but NCES’s mission has also expanded in scope.

NCES’s core mission is to collect, analyze, and report education information and statistics that are high quality, objective, timely, and useful to key stakeholders, including “practitioners, researchers, policymakers, and the public.”¹ As a federal statistical agency, NCES has further roles and responsibilities to protect its data. Importantly, in January 2019, the Evidence Act substantially increased the roles and responsibilities of statistical agencies for evidence-building activities both within and outside the federal government, including assigning them a special role within their executive branch departments.² Despite this expanded mission, NCES does

___________________

¹ 20 U.S. Code § 9541(b). Available: https://www.law.cornell.edu/uscode/text/20/9541 [March 2022].

² 44 U.S. Code §§ 3581–3583 and 5 U.S. Code §§ 311–314. Available: https://www.law.cornell.edu/uscode/text/44/chapter-35/subchapter-III/part-D [March 2022], and https://www.law.cornell.edu/uscode/text/5/part-I/chapter-3/subchapter-II [March 2022].

not have an agency-level strategic plan to address this change in scope. In fact, to the panel’s knowledge, NCES has not had a strategic plan in recent decades.³

The panel does not imply that NCES is uninterested in innovation or in achieving greater mission impact. In fact, NCES is constantly innovating to increase its relevance and impact. NCES has been highly engaged with ED’s program offices and Office of the Chief Data Officer.⁴ In 2021, NCES fielded its timely School Pulse Panel, to understand the effects of the COVID-19 pandemic on education (NCES, 2021l). Further, NCES has engaged the National Institute of Statistical Sciences on dozens of advisory reports to improve specific programs, processes, or products (NISS, 2021b). However, these studies do not always connect to each other, resulting in a piecemeal approach to agency improvement.

NCES lacks an articulated prioritization plan for its products, services, and related improvements that is governed by a set of principles guiding the deployment of resources to achieve a vision. While the National Assessment of Educational Progress has conducted strategic planning, in recent decades NCES has lacked a routinized, holistic strategic-planning process covering the entire Center—its organization, operations, and programs.⁵ The lack of such a strategy makes it challenging to understand how new or old initiatives contribute to the Center’s mission and which products and activities to prioritize or discontinue. Thus, the panel’s strongest recommendation presents NCES with a method for making operational and tactical decisions resulting in an effective organization that can adapt to changing needs in education and changes in data used for evidence-based decision making.

RECOMMENDATION 2-1: To direct its future, NCES should develop and implement a bold strategic plan that incentivizes innovation and creative partnerships and that will produce relevant, timely, and reliable statistical products to assist education decision makers at every level of government. NCES should develop and begin implementation of the plan within 1 year of the release of this report.

Disciplined strategic planning will help NCES focus on high-value products and services so that it can achieve maximum effectiveness and mission impact. The panel recognizes the challenges of deprioritizing products and services, especially since every product has at least one stakeholder. NCES cannot be all things to all stakeholders and instead should focus on areas in which it can add the most value. Planning requires investment and

___________________

³ NCES response to question from the panel, p. 56.

⁴ NCES response to question from the panel, pp. 50–51.

⁵ NCES response to question from the panel, pp. 56–57.

implementation but can result in the efficient and effective use of limited resources. Other reports have also advised NCES to develop “strategic prioritization” (NISS, 2016, p. 11) or “a conceptual framework for organizing its program and for setting priorities in light of available resources” (NASEM, 1986, p. 27). Given that this recommendation has been repeated across decades, the panel cannot overstate the importance of NCES investing time to develop its own strategic plan for raising the Center’s overall effectiveness. With NCES’s new leadership, the Center has an exciting opportunity to control its future and discover new ways to create mission impact.

Such forethought is even more critical for a small agency with limited resources, many stakeholders, a disproportionately large administrative burden, and unfunded mandates. The Bureau of Transportation Statistics is an example of a small federal statistical agency with an up-to-date, short strategic plan (Bureau of Transportation Statistics, 2022; see Box 5-1) (see Appendix D for an overview of federal statistical agencies and their comparability to NCES). NCES can do the same.

Strategic planning is not a compliance exercise. To paraphrase President Eisenhower, plans may be worthless, but planning is essential. NCES can take this opportunity to self-assess, determine its identity, values, and goals, and decide how to move forward in the 21st century. The panel advises NCES to include many of the recommendations in this report. However, even if NCES disagrees with these recommendations, the key point is that NCES should perform strategic planning for the Center’s own sake. Once NCES institutionalizes its strategic intentions throughout the staff and implements its priorities with discipline, the Center will be proactive and nimble rather than being so “responsive to immediate demands”⁶ that its long-term progress and overall effectiveness are diminished.

Visioning, strategic planning, and implementation are time-consuming investments. It is difficult to set aside time for these processes amidst urgent production needs. Given that NCES is a small agency with vast contract resources, the panel suggests that NCES engage a consultant to assist with the most intensive strategic plan-related activities, such as holding leadership, staff, management (i.e., ED and IES), and stakeholder interviews; facilitating conversations; reviewing internal documents and data; and reviewing guidance on law, federal policies, and ED strategic plans. The consultant could integrate the information into a proposed plan and contribute ideas gleaned from experiences with other agencies. Engaging a consultant could substantially reduce staff time needed for strategic planning.

Whether or not a consultant is utilized, NCES would benefit from conversations about what the Center is, its strengths and weaknesses, what

___________________

⁶ NCES response to question from the panel, p. 56.

it wants to be, what its opportunities are, and how it wants to bring the most value to the education data ecosystem (e.g., state and local education agencies). The panel has a strong vision for NCES, but NCES will need to either adopt that vision or determine its own vision. Then NCES will need to examine the structures and resources that exist to support the vision, like the SLDS, and consider how to apply such forward-thinking models elsewhere. The panel recommends that NCES also utilize internal information to examine and understand the challenges that prevent it from fully achieving its vision. Given the marginal observable improvements in response to past review reports, the panel advises NCES to thoroughly investigate its operations and organizational structure, to find and fix the systemic and infrastructural issues. Finding a strategic path to overcome these obstacles can only occur through difficult and frank conversations, which can also increase employee engagement and empower employees to improve the organization.

After NCES determines its vision and the strategic plan begins to congeal, the NCES commissioner should engage IES and ED leadership to ensure buy-in and alignment of the strategic goals of ED and IES and to advocate for sufficient resources. The National Board for Education Sciences⁷ could provide external backing for NCES’s strategic priorities. Ensuring support from above is critical to the success of NCES’s strategic plan.

The strategic plan would benefit from an accompanying implementation plan that is executed with discipline. The implementation plan could include specific areas of prioritization and deprioritization, time lines, a plan for stakeholder engagement, steps for developing and studying new data sources, and an agency-level analytic agenda. The panel recommends that NCES invest effort and time into infusing the strategic and implementation plans throughout the Center. Organizational transformation requires repeated messaging, along with openness to feedback from all levels of staff. The panel suggests that NCES’s leadership team take the time to endorse the plan and engage the entire staff, to gain buy-in on its implementation. For the strategic plan to succeed, NCES staff at all levels must be aligned to that plan and use it to guide all decisions.

The strategic plan should be a comprehensive package describing priorities, gaps, and goals and explaining how new tools and technologies can be leveraged to build forward-looking operations and infrastructure to achieve those goals. It would also be useful for the plan to indicate the level of effort needed to manifest goals and objectives. Some programs and initiatives may need to be divided into stages. The panel recommends that NCES’s strategic plan address each of the seven areas considered in

___________________

⁷ See 20 U.S. Code § 9516(b) – National Board for Education Sciences, Duties. Available: https://www.law.cornell.edu/uscode/text/20/9516 [March 2022].

the sections that follow, where the first four are especially critical to the Center’s transformation.

Incentivize Innovation, Experimentation, and Continuous Learning

The panel advises that NCES’s strategic plan incentivize innovation, experimentation, and continuous learning throughout all facets of the Center. Innovation frequently results from challenges, and NCES often faces challenges that are largely beyond the Center’s control, such as changing presidential priorities. Experimentation (i.e., trying something new) is a key component of adaptability and resilience in the face of challenges. Moreover, continual improvement and innovation is a fundamental principle of federal statistical agencies (NASEM, 2021b). The panel suggests that NCES’s strategic plan address ways that the Center can encourage and incentivize an organizational culture of innovation and experimentation.

Throughout the strategic-planning process and beyond, NCES leadership and staff will need to actively, consciously, and explicitly inhabit the culture-of-innovation mindset to draft an actionable, thoughtful strategic plan. Without this mindset, NCES will be unable to thoroughly examine challenges, set priorities, or imagine new, effective solutions.

The strategic plan would benefit from practices that support a culture of innovation. For instance, a program or competition to improve processes, methods, and cost-effectiveness could encourage staff to improve their functions within the Center (NASEM, 2021b, Practice 3). In the panel’s opinion, not only should NCES hire staff with cutting-edge skills, but it should invest in the ongoing development, retention, and professional advancement of its staff, through activities such as presenting at professional conferences and engaging with research fellows (NASEM, 2021b, Practice 4). NCES’s seasoned and dedicated staff⁸ deserve to use and extend their knowledge and skills beyond just managing contracts. Moreover, retaining institutional knowledge is important for the long-term, efficient, and effective operation of any federal agency.⁹ NCES can develop an active research program that includes substantive analyses, as well as studies evaluating new methods, operations, and the fitness of alternative data sources (NASEM, 2021b, Practice 5). Finally, to accelerate the shift to a culture of innovation, the panel advises that NCES build its staff capabilities while forging strong partnerships with other components of IES, especially the new data sciences

___________________

⁸ About 83 percent of NCES’s staff have advanced degrees. Nearly half of the staff have earned doctorates (PhD or EdD) and another 34 percent have master’s degrees or are doctoral candidates (i.e., “all but dissertation”). NCES response to question from the panel, p. 46.

⁹ For the private-sector benefits of retaining in-house professional services, see Ding et al., (2020).

unit in the Office of the Director,¹⁰ innovative agencies, and other external experts, to identify best practices and pilot new, potentially transformative approaches (NASEM, 2021b, Practice 7). Additional ideas are provided in Box 2-1.

Expand NCES’s Presence in Education Evidence Building

The Evidence Act has expanded NCES’s mission in the absence of additional funding. In the panel’s opinion, NCES’s strategic plan should address how it wants to meet this expanded role and its relationship with other offices in the IES and ED, including its goals for data acquisition, sharing, provisioning data access for evidence-building purposes, and evidence

___________________

¹⁰ NCES document provided to the panel, “Relevant excerpts from the approved December 2021 IES reorganization justification.”

building itself. A recommendation and specific ideas for NCES, IES, and ED to advance evidence building are discussed later in this chapter.

Increase Diversity and Equity Awareness to Maintain Relevant Data Content

The panel recommends that NCES’s strategic plan address how the Center can proactively consider diversity, equity, inclusion, and accessibility (DEIA) (see Appendix A), to better reflect the experiences of the increasingly diverse student population and other contemporary education trends in its data designs, processing, and analyses. The plan should consider ways to embed DEIA awareness into other aspects of NCES’s work, including stakeholder engagement, hiring, retention, and work culture. A recommendation and specific ideas for leveraging DEIA awareness into stakeholder-relevant products and activities are discussed later in this chapter.

Explore Data Sources to Support Analytic Insights

The panel recommends that NCES’s strategic plan incorporate pathways and priorities for exploring new data sources, particularly administrative data, for use in statistical products and services. NCES’s Administrative Data Division is already exploring data sources including the Social Security Administration’s Master Earnings File and the Internal Revenue Service’s Education Tax Credit and Income Data, among others.¹¹ The strategic plan would benefit from prioritizing datasets and partnerships to pursue and including some intermediate goals, objectives, and milestones. Recommendations for exploring, evaluating, and incorporating alternative data sources into NCES’s statistical programs are discussed later in this chapter.

Prioritize Data Content, Services, and Activities

The panel recommends that NCES’s strategic plan describe its priorities, to establish the guiding principles and criteria for keeping, changing, and removing data content, services, or activities (NASEM, 2021b, Practice 6). Currently, NCES appears to add stakeholder content in the absence of overarching strategic principles. Sometimes a product can supersede the mission of an agency, particularly if it is the status quo, has been collected for years, and has an invested user base. While this “autopilot” mode may be good for efficiency, it is not necessarily good for maintaining the Center’s effectiveness. NCES is encouraged to review every collection and product routinely (e.g., every few years), to check that the content still fits

___________________

¹¹ NCES response to question from the panel, pp. 28–29.

the Center’s strategic goals, remains relevant to the social context, and continues to provide valid measures of the changing population.

NCES would benefit from eliminating content that is outdated, no longer useful, or that has lost value. If outdated content is federally mandated, NCES is encouraged to request Congress to change the statute. A recommendation and detailed suggestions on content priorities and guiding principles for prioritization are presented in Chapter 3 and Appendix C.

Expand Creative Partnerships and Engagement with Stakeholders

In the panel’s opinion, NCES’s strategic plan should include goals, objectives, and milestones for establishing creative partnerships and increasing engagement with stakeholders. Areas for NCES to consider include creating engagement feedback loops; expanding its role enabling data access, particularly for state and local education agencies; and improving dissemination, focusing on accessibility and usefulness. Chapter 4 discusses several recommendations and suggests pathways to expand NCES’s creative partnerships and engagement.

Align Operations and the Organization to Support NCES’s Mission and Vision

In the panel’s opinion, NCES’s strategic plan should address aligning its organization and operations to meet or make progress towards its new vision and strategic goals. Unlike the other areas addressed by the strategic plan, which NCES can begin to work on during the planning process, the organizational aspects are driven by the content of the plan and need to be considered later in strategic planning and during implementation planning. Achieving the vision and mission requires resources, effective operations, and a plan for maximizing use of resources. Determining the level of effort needed to meet specific strategic goals is a difficult process and NCES will need to make hard decisions, given the net loss of 33 percent of its statistical staff along with flatlined program appropriations since FY 2010, which have resulted in a loss of contract buying power when adjusted for cost inflation (see Figures D-1 and D-2). Based on these recent trends, the panel assumes no increase in resources for NCES’s statistical programs.

NCES can conduct its strategic planning to achieve its vision and strategic goals using the Center’s existing level of resources. The strategic plan should address how NCES attracts, develops, and retains staff with the skills needed to achieve the Center’s vision (Chapter 2, above). The strategic plan can address objectives to leverage external resources as a force multiplier (see Chapters 4 and 5) and can indicate where internal resources may be needed to leverage external resources. The Panel suggests that, during

planning, NCES consider the type of organizational structure (perhaps combined with a specific program-management model) that would best support progress towards its vision, given the Center’s existing resources. Chapter 5 discusses recommendations and ideas for transforming NCES’s current organizational alignment, staffing, and use of contractors and other resources to support the strategic plan.

If NCES is disciplined in its approach to strategic planning, prioritization, and implementation, its workload will decrease as lower-value activities, collections, and products are eliminated. Identifying new data sources or technologies that gain efficiencies and ending low-value and costly collections and contracts will free up program appropriations that NCES can strategically redirect to high-value uses.¹² However, all else equal, NCES’s vision could be achieved more quickly with a higher number of strategically deployed full-time equivalent employees (FTEs), as well as more program funding (see Box 2-2).

SUPPORT AND EMPOWER NCES TO SET ITS OWN PRIORITIES

While NCES is primarily responsible for its own strategic planning and priorities, IES and its subunits and other offices in ED need to fully support and collaborate with NCES for the Center to achieve its vision. Currently, NCES “does not have complete control over its priorities” and “has limited discretionary flexibility when setting priorities for its activities.”^13,14 This is due, in part, to the many constraints NCES operates under, along with its dependence on IES and ED for support services and resources. IES’s director and commissioners will need to determine the appropriate distribution of resources between NCES and other centers. In such discussions, NCES needs to be clear on its goals and its progress towards them, the value added from achieving the goals, and its resource needs. IES may also need to advocate for NCES to the secretary of education and to Congress. Overall, for NCES to successfully manifest its visioning, strategic planning, and implementation, ED and IES need to fully support their NCES colleagues.

___________________

¹² All staff are paid indirectly through an allocation of the ED’s Salaries and Expenses appropriation. The organization of staff into statistics and assessment functions does not align with program appropriations, which funds contracts. NCES response to question from the panel, pp. 3–10.

¹³ NCES response to question from the panel, p. 55.

¹⁴ “Each Commissioner, except the Commissioner for Education Statistics, shall carry out such Commissioner’s duties under this title under the supervision and subject to the approval of the Director [of IES].” 20 U.S. Code § 9517(d). Available: https://www.govinfo.gov/content/pkg/USCODE-2015-title20/pdf/USCODE-2015-title20-chap76-subchapI.pdf [March 2022].

RECOMMENDATION 2-2: The secretary of education, director of the Institute of Education Sciences, and NCES commissioner should collaborate to ensure that NCES is independent in developing, producing, and disseminating statistics.

Together, ED, IES, and NCES are should revisit and update internal policies and procedures to ensure that NCES can operate under the well-established principle and directives stating that federal statistical agencies must be independent from undue external influence in developing, producing, and disseminating statistics (NASEM, 2021b, Principle 4; U.S. OMB, 2008, 2014b). The Office of Management and Budget’s Statistical Policy Directive No. 1 outlines the fundamental responsibilities of federal statistical agencies to conduct “objective, credible statistics” (U.S. OMB, 2014b, p. 71610). Statistical Policy Directive No. 4 provides requirements on the release and dissemination of statistical products, by requiring federal statistical agencies to “adhere to data quality standards through equitable, policy-neutral, and timely release of information to the public [emphasis added]” (U.S. OMB, 2008, p. 12624).

The National Academies of Sciences, Engineering, and Medicine (NASEM, 2021b) describe multiple practices that support this principle of independence. It is particularly important for NCES to have the authority to make decisions on the scope, content, and frequency of its data and statistics; to select and promote professional staff based on skills and knowledge; and to “be able to meet with members of Congress, congressional staff, and the public to discuss the agency’s statistics, resources, and staffing levels” (NASEM, 2021b, Practice 2, p. 54). NCES should also have highly qualified staff¹⁵ to make decisions on data content based on scientific and professional considerations, and to gather input on data needs from stakeholders and ED policy and program officials (NASEM, 2021b, Practices 3, 4, 5, 9). The panel encourages ED, IES, and NCES to act as partners to ensure NCES operates with its full authority as a federal statistical agency, which includes serving its stakeholders in IES and across ED.

MAXIMIZE NCES’S UNIQUE VALUE FOR EVIDENCE BUILDING

It is an exciting time for evidence-based decision making, particularly in the federal government. The momentum towards data- and evidence-driven decision making was substantially boosted by the Evidence Act, enacted in January 2019, and was further enhanced by Presidential Memoranda and Executive Orders issued in 2021 (e.g., Biden, 2021; Executive Order 13994, 2021; Executive Order 14000, 2021). Laws and other mandates advance the development of, access to, and statistical use of data and evidence for strategic and operational decision making by governments at any level,

___________________

¹⁵ Qualifications needed include expertise in data analysis and science (defined broadly), sampling statistics, assessment development, survey methodology, and statistics. NCES response to question from the panel, p. 18.

researchers, and others.¹⁶ The Evidence Act and related mandates place statistical agencies at the heart of evidence-based decision making. Through these mandates, NCES has great opportunities to make an enormous impact on evidence building and to establish new avenues for retaining stakeholder relevance.

The Evidence Act, particularly in Title III, effectively expands NCES’s mission by giving statistical agencies new authorities, duties, roles, and relationships for evidence building. The Evidence Act directs all departments to make government data open by default and to share administrative and other data with statistical agencies, upon request, for developing evidence.¹⁷ The statistical agencies are, in turn, directed to expand secure access to data (i.e., restricted, acquired, linked, etc.) for evidence building, while protecting privacy.¹⁸ As leaders of the Interagency Council on Statistical Policy, the statistical agencies supported OMB in developing a Standard Application Process (SAP), by which federal “agencies, the Congressional Budget Office, state, local, and Tribal governments, researchers, and other individuals, as appropriate, may apply to access the data assets… for purposes of developing evidence.”¹⁹ The SAP highlights the mandate of statistical agencies to work with each other and with external stakeholders as data-access facilitators and governors.

The Evidence Act further establishes the NCES commissioner as the chief statistical official (SO) of ED, directed to work closely with other senior executive officers to advance ED’s development and use of scientifically rigorous evidence. Thus, the Evidence Act further bolsters the relationship between NCES and two of its most important stakeholders: IES and ED. The SO is mandated to work in partnership with the chief data officer (CDO), the chief evaluation officer (EO), and other chief executive officers to further evidence-based decision making based on high-quality data, statistics, and other evidence to inform ED’s “learning agenda,” or strategic agenda of analytic questions.

Because of the importance of federal statistical agencies in the movement towards data-driven decisions, the panel feels that NCES and the SO

___________________

¹⁶ 44 U.S. Code § 3583 – Application to access data assets for developing evidence. Available: https://www.law.cornell.edu/uscode/text/44/3583 [March 2022].

¹⁷ 44 U.S. Code § 3581 – Presumption of accessibility for statistical agencies and units. Available: https://www.law.cornell.edu/uscode/text/44/3581 [March 2022].

¹⁸ 44 U.S. Code § 3582 – Expanding secure access to CIPSEA data assets. Available: https://www.law.cornell.edu/uscode/text/44/3582 [March 2022].

¹⁹ 44 U.S. Code § 3583. Available: https://www.law.cornell.edu/uscode/text/44/3583 [March 2022]. See also U.S. OMB (2022) and “Standard Application Process (SAP) Policy.” Available: https://www.federalregister.gov/documents/2022/01/14/2022-00620/the-interagency-council-on-statistical-policys-recommendation-for-a-standard-application-process-sap and https://www.regulations.gov/document/OMB-2022-0001-0001 [March 2022].

can and should be central in ED’s implementation of these mandates. NCES provides technical assistance to many offices within ED, by:

• Managing ED’s EDFacts collection;
• Collecting the Civil Rights Data Collection for the Office for Civil Rights;
• Supporting the Office of Postsecondary Education and the Office of Career, Technical, and Adult Education with data updates or collections;
• Supporting effective implementation of the College Scorecard with ED’s Office of the Chief Data Officer in the Office of Planning, Evaluation and Policy Development, to ensure stronger alignment between the Scorecard and NCES tools that use Integrated Postsecondary Education Data System data;
• Serving on ED’s Data Governance Board Steering Committee; and
• Supporting ED in creating an ED Disclosure Review Board.²⁰

Given the new mandates, however, NCES’s support of ED offices goes well beyond technical assistance. NCES has significant new roles and responsibilities and, in the panel’s opinion, should be more proactive in leading ED on acquiring and managing data for evidence building in a privacy-protected manner. For instance, ED’s Data Strategy mentions IES only once, to note that IES provides “localized” access to restricted data (an NCES function), then dismisses the role of IES since a departmentwide solution is needed (U.S. ED, 2020, p. 15). Regarding the current division of labor for evidence building, ED’s EO notes that, when obtaining data for evidence building, he would expect the SO to facilitate access to data at a statistical agency, but would turn to the CDO to obtain data in general.²¹ ED’s EO notes that the first serious test of the Evidence Act’s presumption of accessibility of data for evidence building will arise when a roadblock is encountered when seeking external data, and how that challenge is overcome. This suggests that ED has not fully considered how to leverage NCES’s new mandates to acquire data and facilitate data access for evidence building.

In recent years, NCES has rarely interacted directly with the secretary of education’s office and, further, most contact pertains to the National Assessment of Educational Progress, not the statistical programs. In the past, direct, in-person communication with the secretary and the ED chief

___________________

²⁰ NCES response to question from the panel, p. 14.

²¹ Matthew Soldner, “Comments for the National Academies’ Panel on a Vision and Roadmap for Education Statistics,” presentation to the National Academies of Sciences, Engineering, and Medicine, August 2, 2021.

of staff occurred fairly often and was especially true during the release of high-profile reports such as the Nation’s Report Card. The NCES commissioner and associate commissioner conducted embargoed briefings for the secretary and senior ED staff in the secretary’s conference room or office.²² Given the strong momentum behind evidence-based decision making, coupled with NCES’s newly mandated role, now is the time for NCES to reassert itself.

RECOMMENDATION 2-3: The secretary of education, director of the Institute of Education Sciences, and NCES commissioner should immediately take actions to enable the NCES commissioner to most effectively fulfill the responsibilities of the statistical official delineated in the Foundations for Evidence-Based Policymaking Act of 2018 and to support evidence-building needs across the Department of Education.

Evidence-based decision making is the purview of the entire ED, and NCES is mandated to serve a particular role. Implementation of the Evidence Act is relatively nascent²³ and this moment presents an opportunity for the secretary of education, the director of IES, and the commissioner of NCES to establish a vision and value proposition for NCES’s role in ED’s evidence-building activities. Together, the three entities are encouraged to determine how to best maximize the unique value NCES brings as a producer of credible and relevant evidence, a recognized leader in data standards, and a data-access facilitator.

The secretary, director, and commissioner should update all related policies, divisions of responsibilities, processes, and practices to empower the NCES commissioner and leadership team to most effectively carry out the SO’s duties. Further, NCES needs to both affirm its authority as central to ED data and evidence conversations and back up that authority by making meaningful contributions in partnership. In the past, NCES has successfully positioned itself as an essential voice in ED conversations informing education policy.²⁴ NCES is fully capable of returning to this stature if its entire leadership team aligns with a visionary strategy for engagement on evidence building.

NCES can ensure its standing by contributing its expertise as an important information source as decisions are being made. Combined, the Evidence Act and the Information Quality Act (U.S. OMB, 2019b, 2004, 2019a) direct government agencies to make decisions based on high-quality

___________________

²² NCES response to question from the panel, p. 51.

²³ At the time of writing, OMB had not issued guidance for the implementation of Titles II and III of the Evidence Act.

²⁴ NCES response to question from the panel, p. 51.

evidence. To enable this, the SO needs to collaborate with the CDO and EO in ED-wide data governance, overseeing the use of data in ED’s evidence building, developing learning agendas, and identifying data needed to inform learning-agenda questions. NCES adds value to the partnership in many areas, including semantic data-standards expertise, privacy-protection practices, rigorous methods for developing data and evidence, determinations of data relevance and fitness for use or purpose, the authority to ask for and receive government data, and the duty and expertise to provide data access to a broad range of stakeholders for evidence-building purposes. The CDO emphasizes ED’s data, data governance, inventories, formats, and open data. The EO has purview over the analytic educational and operational questions needed to advance understanding in a strategic way (U.S. OMB, 2019b). The SO connects the CDO and the EO by turning data into high-quality information fit to inform policy and decision making.

The Evidence Act presents evidence goals to be fulfilled and encourages departments to use new techniques, share and analyze appropriate data, and form new partnerships. NCES and IES can fulfill this need, which can be filled by no one else in ED. The panel urges the centers within IES, which currently appear siloed, to come together to build stronger partnerships, grasp these enormous opportunities, and lead ED’s evidence building and research.

The National Center for Education Research awards grants for education research (IES, 2021a). The National Center for Education Evaluation and Regional Assistance, whose commissioner is ED’s EO, leads ED’s learning agenda (strategy) and related evidence building and evaluation. NCES collects data, acquires²⁵ secondary data, provides access to restricted data, evaluates fitness for use, and has rigorous quality standards for its statistics and analyses. By working together strategically, the centers could explore the important ideas and current questions in education, determine how to rigorously test research hypotheses, and decide how to operationalize new learnings in education data and statistics, with each center contributing its unique expertise. Further, the centers could collaboratively decide the priority datasets NCES should acquire and link (e.g., long-term outcomes on experimental data), determine which data could be made public, and provide restricted data access to external researchers as a force multiplier for answering learning-agenda questions (see Chapter 4).

Similarly, NCES and the other IES centers could collaborate with the CDO to determine data needs, data standards, open-data formats, and product-release timing that would be of high value to ED and its stakeholders. NCES needs to constantly fuel evidence building by identifying

___________________

²⁵ In this report, data collection means primary data collection, whereas data acquisition includes acquiring secondary data in addition to primary data.

opportunities to use various data sources, establishing standards and techniques for increasing data quality, ensuring data are fit for purpose, and protecting privacy. NCES’s leadership is essential for the effectiveness of ED’s Data Governance Board and ED Data Strategy (led by ED’s CDO), as well as ED’s Disclosure Review Board (led by ED’s chief privacy officer or senior agency official for privacy). The Evidence Act has immense potential for advancing education data and analytics. This innovative era presents great opportunity for NCES to increase its relevance and expand its impact. The Evidence Act is best implemented with full and equal partnerships between the SO, EO, CDO, and other key officials in ED. It will take vision, strategic planning, and collaboration between NCES, IES, and ED to fully leverage the evidence-building power of the SO role.

ADAPT TO THE CHANGING WORLD OF EDUCATION: INCREASE DIVERSITY AND AWARENESS OF EQUITY ISSUES

NCES’s mission includes relevance and usefulness to key stakeholders, which means that the Center’s data and statistics must address contemporary issues in education. In recent decades, the nation has become more diverse in terms of race/ethnicity, sex, gender identity, students with disabilities, and sexual orientation. For instance, children (ages 5–17) shifted from 75 percent white and 9 percent Hispanic in 1980 to 50 percent white and 26 percent Hispanic in 2020 (Figure 2-1) (NCES, 2021e, Table 101.20). The diversity in public school systems is more pronounced, projected to be 44 percent white and 28 percent Hispanic by 2029 (NCES, 2021e, Table 203.50). Broadly similar trends are seen in higher education, with increasing racial and ethnic diversity and more women than men attending university or higher (NCES, 2021e, Table 306.10).

Disability status is another important trend among students. The number of students ages 3–21 who received services under the Individuals with Disabilities Education Act nearly doubled, from 3.7 million in the 1976–77 school year to 7.3 million in the 2019–20 school year (NCES, 2021e, Table 204.30). The types of disabilities children face have also changed. In 1976–77, the top three conditions were speech or language impairment, intellectual disability, and specific learning disability, accounting for 83 percent of all disabilities (Figure 2-2). By 2019–20, the conditions diversified as more became known and diagnosed. For example, autism now makes up 11 percent of student disabilities. As these conditions, and how they affect learning, become better understood, the data on these students, their needs, and their teachers will also need to adapt.

Other education trends are broader. For instance, more students are attending college and beyond than in the past, nearly doubling from 11 million in 1976 to 20 million in 2019, with a peak of 21 million in 2010

(NCES, 2021e, Table 306.10). More students from economically and racially disadvantaged groups are aspiring to higher education, but have lower enrollment and college completion rates than their peers (Backes et al., 2015; NCES, 2021e, Tables 306.10, 326.10, 326.15; NCES, 2022a). The number of adults obtaining education has increased as well, and this pattern is closely tied to equity issues (Grodsky et al., 2021). The numbers of institution types, sites, and modes of instruction have increased, with home schooling, internet elementary and secondary schools, online

colleges, and training provided by noncolleges now existing alongside public, private, and charter schools, many of which also offer online learning. Program and curriculum content have changed, as have instructional styles. Schools, teachers, policy makers, and others want to understand how these trends factor into student outcomes, and data are needed for this purpose.

There are many more dimensions of population diversity that have become nationally significant in recent decades (e.g., rural and urban status, sexual orientation and gender identity, immigrant students, Native American students). The aspects of education that are policy relevant will continue to change. Data are needed to understand these changes and NCES currently measures some of these dimensions. However, data collections designed in the past are no longer adequate to understand today’s diverse students and educational contexts, so understanding the current education situation requires an adjusted approach to data collection and products. We have seen no evidence that the rate of change in the social context is lessening. Thus, it is likely that the education context will continue to evolve, requiring NCES to continue to revisit its approach to remain effective. The challenge here, for any statistical agency, is that providing relevant and useful information on current education conditions requires addressing deeply embedded equity issues throughout the data life cycle, such as underlying assumptions in questionnaire design or imputation algorithms. Addressing such issues will require the sustained involvement of relevant communities, to decide which questions need to be informed by data collections, which data to collect, how to collect those data, how to organize and present data, and how to make meaning from data. While this process obviously requires engagement with the stakeholders that represent the diversity in American education, the process is likely to be facilitated by staff and contractors who also represent that diversity (Executive Order 14035, 2021). Thus, we suggest that NCES address diversity in the composition of internal and contractor staff, assess the inclusiveness of the Center’s internal cultures, and ensure data products accurately reflect contemporary communities as society evolves.

RECOMMENDATION 2-4: NCES should proactively embed diversity, equity, inclusion, and accessibility in all areas of its work and organization, to adapt and serve contemporary communities of the changing world of education.

The panel takes a strong view that becoming adaptable and staying relevant to stakeholders relies on NCES addressing DEIA issues (see Appendix A). A key principle for federal statistical agencies is to provide “objective, accurate, and timely information that is relevant to important public policy issues” (NASEM, 2021b, p. 27). To maintain relevance, statistical agencies should:

• Regularly evaluate their data collections to ensure relevance (NASEM, 2021b, Practice 6);
• Have staff analyze the agencies’ own data (NASEM, 2021b, Practice 5); and
• Work with data users and other stakeholders to keep abreast of evolving data needs (NASEM, 2021b, Practice 9).

DEIA issues and disparities have long been relevant topics in education policy, with the groups of interest changing over time. Grappling with DEIA issues is critical for NCES’s relevance and ability to adhere to these practices by:

• Ensuring validity of survey instruments and measurements, not only for measures of demographic characteristics, segregation indices, or other direct measures of population diversity and equity, but also for any concept or measure that may be interpreted differently over time and context by respondents with varying lived experiences (Box 2-3);
• Ensuring that data processing and analyses accurately reflect the diverse perspectives and lived experiences of today’s students and education workforce rather than reiterating historical and systemic assumptions, emphases, and absences in the data; and
• Deepening engagement with a broadly diverse group of stakeholders to understand their needs for data content, data products, and statistical information, to include equity (Box 2-3).

NCES has made notable efforts in some aspects of DEIA. For instance, the Center produces data and analyses on groups that have been historically disadvantaged, such as reports disaggregating by race and ethnicity (NCES, 2018, 2020, 2021j). Not only is this type of information an important aspect of measuring diversity and equity, but all federal agencies

have also been directed to disaggregate data by groups that have been “historically underserved, marginalized, and adversely affected by persistent poverty and inequality” to advance equity (Executive Order 13985, 2021, p. 7009). However, the panel believes that proactively embedding DEIA considerations involves much more than disaggregating analyses by groups, such as race and ethnicity or disability status. To adequately measure and address DEIA issues in its data, NCES needs to embed DEIA considerations throughout the Center’s workforce, not only via staffing decisions, but by creating a culture of proactive thinking about DEIA issues throughout its data designs, data acquisitions, standards, analyses, stakeholder engagement, relationship building, partnerships, and contracts.

The panel recommends infusing DEIA thinking throughout the data life cycle, starting with data design (e.g., standards setting, questionnaire design, construction of measures and indicators) to ensure validity and measure diverse populations as we now understand them. Creating new categories to measure important, socially recognized characteristics and issues, such as NCES’s new collections on student acceptance groups based on gender identity and sexual orientation, is a critical part of addressing

changing populations, but it is still insufficient (Hansen, 2019). At a minimum, NCES should revisit questionnaire language to ensure it is bias-free (American Psychological Association, 2021). To go deeper, accurate measurement of demographic characteristics and life outcomes is founded on understanding people, lived experiences, and social issues. Survey instruments and questions reflect those who write them and need to be reevaluated and revised if they are no longer appropriate for understanding today’s students (NASEM, 2021b, Practice 6). NCES can better understand students by thinking more deeply about and being exposed to the ever-evolving diversity of lived experiences and how those experiences relate to student outcomes in education and other areas of life. Such understanding is a critical first step in the data life cycle and will help NCES to stay relevant as the social context evolves. From this improved understanding follows the need to continually ensure validity and accuracy of data and statistical activities in a changing world.

In the panel’s judgment, NCES should adjust its approach to questionnaire and study design to understand diverse student perspectives and to decide which information is relevant to that understanding (Box 2-3). Importantly, NCES needs to involve members of these diverse communities in the initial stages of data design, using qualitative methods such as focus groups and cognitive interviewing. This process begins with members learning how they see themselves, how their lived experiences may lead to interpretations of data collections that are different than those assumed or intended by researchers, and which life outcomes (e.g., earnings, health, social connectedness) are meaningful to them. Survey methodologists may not be intentionally biased, yet they often reflect unique lived experiences; even survey methodologists from diverse backgrounds may not fully understand all dimensions of diversity. Thus, it is crucial to learn from diverse communities to understand and measure concepts and social issues in ways that resonate with them. Survey questions and instruments need to be redesigned and cognitively tested to ensure the validity of the measures.²⁶

NCES should further examine, from an ethical standpoint, the dynamic between researchers and survey participants—especially participants in historically disadvantaged communities—to consider how the data-collection interaction can be beneficial to respondents and their communities. For example, every time a teacher completes a survey, an opportunity is created for NCES to support that teacher and demonstrate the Center’s commitment to advancing equity. When designing data collection operations,

___________________

²⁶ For examples of testing measurement of sexual orientation and gender identity, see NASEM (2022) and Ellis et al. (2018). For race and ethnicity concept measurement, see studies at https://www.census.gov/about/our-research/race-ethnicity.html [March 2022]. For an example of testing concepts other than dimensions of diversity, see de la Cruz (2011) and Jensen (2013).

NCES is encouraged to seek out members of diverse communities to help survey operations experts understand how to approach various populations and provide benefits to survey respondents (NASEM, 2018; NISS, 2020). This approach of reciprocity often improves survey response rates as well (Box 2-3). Engaging members of the public at this stage of design requires expertise in qualitative methods. NCES can collaborate with other centers in IES or can contract out this work, but permanent NCES staff need to be closely involved to assure that knowledge is retained and to develop a culture of DEIA awareness that can be applied to other areas of NCES’s work.

Instilling DEIA thinking into analytics includes not only responsible interpretation of statistics and analyses, but also consideration of data processing and other structural features underlying the analysis. For instance, imputation methods during data processing or recodes for an analysis can result in statistically biased estimates. Care should be taken to use ethical and empathetic methods for imputing data, particularly for studies on equity for historically disadvantaged groups (Brown et al., 2021). Machine learning and predictive algorithms typically reinforce outcomes based on assumptions, emphases, and absences from historical social systems. Critical assessment of analytic questions is also important—rephrasing a research question can help a study to be more equitable and inclusive. For example, asking “Are school systems ready to educate local 5-year-olds?” instead of “Are 5-year-old children ready to attend school?” shifts the burden of readiness from individuals and families to schools.²⁷ In terms of product content and presentation, the panel believes that NCES should ensure analysts use inclusive language in all products, and that the use of colors and icons, and the ordering of information in data visualizations, supports inclusivity.²⁸ The panel suggests that NCES also think deeply about DEIA considerations in terms of data-dissemination formats (particularly for people with disabilities), as well as the general usability of all its products, tools, and services. The panel presents specific ideas and suggestions for data content, engagement, and dissemination in Chapters 3 and 4.

NCES can also benefit from infusing DEIA throughout the organization: its staffing decisions, its culture, and its partners. By engaging with diverse stakeholders (e.g., schools with diverse student populations) and relationship building (e.g., advisory forums, scholars of color), NCES can better propel change in the education data ecosystem. From its stakeholders, NCES can learn what types of data it should be collecting and can gain an understanding of stakeholders’ needs for data, data products, and

___________________

²⁷ Gabriela Katz, “StriveTogether Cradle to Career Network,” presentation to the National Academies of Sciences, Engineering, and Medicine, August 6, 2021.

²⁸ See examples: https://content-guide.18f.gov/our-style/inclusive-language/, https://nasaa-arts.org/nasaa_research/inclusive-language-guide/, and Schwabish and Feng (2021) [March 2022].

statistical information. This will help NCES to deliver useful and relevant products to its stakeholders. In granting licenses for data access, NCES can solicit projects from scholars of color. This would both support DEIA values and expose NCES staff to new insights and perspectives on diverse populations.

NCES should instill a culture of DEIA throughout its staff, rather than burdening only staff from diverse backgrounds. NCES leadership should demonstrate DEIA values by taking actions, such as those suggested above, to grapple with DEIA issues. NCES leadership should consider diversity in hiring and staffing decisions, and the Center is encouraged to develop an inclusive work environment to attract and retain staff from diverse communities. NCES can also put DEIA on individuals’ performance plans, particularly those of managers, to support an environment that provides a sense of belonging to all staff. Supportive actions can include seeking opportunities to ally with staff, calling on those who are having difficulty speaking up,²⁹ and promoting a DEIA perspective in all aspects of NCES’s work. A work environment that proactively seeks to understand diversity and equity issues supports a better understanding of study populations, higher productivity (Carr et al., 2019), and greater in-house innovation (Rock and Grant, 2016; Rock et al., 2016). These benefits reinforce the importance of staff obtaining and retaining knowledge from engaging with diverse communities during data design. Further, since many of NCES’s data collections are conducted by contractors, it is essential that DEIA issues are integrated into its contracts. Finally, to maximize impact, NCES can leverage its SLDS grants to encourage thoughtfulness around DEIA issues at the state and local levels.

Embedding DEIA considerations throughout its work, relationships, and organization will help NCES to anticipate evolving needs. NCES is currently performing some of this work, however, the panel recommends that the Center critically consider all aspects of its programs, to determine how to infuse DEIA thinking and actions that will advance NCES as an equitable government agency. Following its strategic plan, NCES can then determine when and how to change its products and services to maintain the Center’s usefulness and relevance.

EXPAND DATA-ACQUISITION STRATEGIES FOR NEW INSIGHTS

The education data ecosystem envisioned by the panel will incorporate data from three categories: probability sample surveys (traditionally used by many federal statistical agencies, including NCES), administrative

___________________

²⁹ For examples, see: https://adurolife.com/blog/employee-well-being/how-to-create-a-senseof-belonging-in-the-workplace/ [March 2022].

records data (which have always been important to NCES and are growing in importance within the government data ecosystem³⁰), and new and emerging data sources. The number of new data sources is huge, diverse, and growing, and includes commercial data available for purchase from private vendors (e.g., household panel data, scanner data from businesses); transactional data (e.g., credit card purchases, job openings); social media and networking data; archival data and video recordings (e.g., NCES’s Trends in International Mathematics and Science Study data); and internet documents, webpages, and videos that can be harvested, “scraped,” or processed with natural language processing to create an analytic database.

Each of the three data categories has pros and cons. Sample surveys provide representative population estimates with quantifiable accuracy and can deliver a rich variety of information. However, they are expensive, and it is becoming more difficult to gain the cooperation of respondents. Administrative records data often have little incremental cost, since they were previously collected for another purpose, and they may contain complete records from a selected population of interest, for example, participants in a specific program. However, administrative data can have systematic biases in coverage. Available data may not contain the precise measurements of interest or may not be fit for use (having been collected for another purpose), may have variable quality, and may be difficult to access. Often, statistical agencies must invest in data cleaning and standardization, which adds to the cost. New and emerging data sources are highly diverse. Many potential sources have the advantages of timeliness, granularity, ease of access, and low cost. Disadvantages include lack of representation, lack of quantifiable accuracy and, in some cases, a lack of mature methodological analysis tools (see Appendix B for details on data sources).

Statistical agencies have been experimenting with the use of administrative records and alternative data sources and have found that, when taken together and carefully integrated, the three sources—surveys, administrative records, and new and emerging sources—can significantly enhance information content, timeliness, and granularity of information systems. Likewise, new data sources may enrich education research by providing data that are unavailable via surveys or administrative data. These emerging data sources present enormous opportunities for NCES to advance evidence building.

RECOMMENDATION 2-5: To improve its efficiency, timeliness, and relevance, NCES should continually explore alternative data sources

___________________

³⁰ There has been a strong push in recent years for federal agencies to use administrative data for statistical purposes, including building evidence for decision making and evaluation of government programs, as evidenced by OMB guidance (U.S. OMB, 2014a) and the passage of the Evidence Act.

for potential use in data and statistical products, conduct studies on the quality of these sources and their fitness for use, and expand responsible access to data from multiple sources and linkage tools. Testing and adoption of new data-science methods for harnessing alternative data should be done in collaboration with other federal statistical agencies, as well as with other components of the Institute of Education Sciences that are actively exploring ways to strengthen the impact of these techniques.

The panel strongly advises that data source exploration and expansion be a part of NCES’s strategic plan. The panel understands that it takes time to study the quality and usefulness of alternative data sources. It takes additional planning and time to understand the potential benefits and costs, integrate alternative data sources into existing operations and products, conduct bridge studies (such as those successfully performed by the National Assessment of Educational Progress program) to evaluate shifting measures and ensure continuity, and assess the privacy implications. Ideally, acquiring more data sources will grow NCES’s community of users, as demand for responsible access to blended data increases. NCES should use parts of its strategic plan (e.g., those covering data content priorities and engagement) to guide a phased plan to prioritize development of specific data sources and use of their content.

Many other federal, state, and local agencies, along with other organizations, are taking advantage of data linkage and blending to conduct innovative analytics and to create experimental products that may eventually become operational. With strategic alignment and engagement, NCES can do the same. NCES can also create tools to combine its data more easily with users’ data sources. For example, NCES already produces public school catchment geographic areas and could enhance their use by incorporating new data and geographic linkage keys. Chapter 4 provides detailed recommendations and suggestions. NCES can conduct ongoing data development to operate more efficiently, cost-effectively, and with higher impact.

RECOMMENDATION 2-6: For primary collections, NCES should modernize standard language on consent and planned usage, to permit secure secondary uses that enable high-quality follow-up studies, such as through privacy-protected linkages with other data sources.

In anticipation of data linkage, NCES should reconsider the consent language and planned usage for all primary collections, to support ongoing uses for statistical activities. An extreme example is the National Education Longitudinal Study of 1988, in which participants were expressly promised

they would not receive additional follow-up contacts from NCES.³¹ This prohibits NCES from recontacting respondents to request consent for uses other than the originally stated purposes. Modernized consent language could ask respondents to allow secondary uses and linkages, could ask for recontact for future requests, or could establish other levels of permission, depending on the respondent population. The panel suggests that NCES look to federal statistical agencies for best practices when drafting statements regarding privacy of survey respondents’ data and its potential uses. NCES is also encouraged to work with potential respondents to learn how to design appropriate consent language.

For data linkage, the panel recommends that NCES ask ED’s Office of General Counsel for a modernized interpretation of what is possible within existing laws regarding personally identifiable information (PII) and the holding of a national-level population dataset.³² For instance, NCES could ask whether it can hold a state or other subnational administrative dataset, with or without PII, or a national sample of administrative data from the states. The panel urges NCES to anticipate demand for data linkage by directing its contractors, who collect and hold PII from NCES’s primary collections, to deposit PII linkage keys with partner federal statistical agencies such as the U.S. Census Bureau. Examples of federal partnerships are discussed in Chapter 4.

___________________

³¹ NCES response to question from the panel, p. 33.

³² For details, see 20 U.S. Code § 9572 – Prohibitions, and 20 U.S. Code § 1015c – Database of student information prohibited. Available: https://www.law.cornell.edu/uscode/text/20/9572 and https://www.law.cornell.edu/uscode/text/20/1015c [March 2022].