4

Expand Engagement and Dissemination for Greater Mission Impact

As part of the charge given to the National Academies of Sciences, Engineering, and Medicine, the panel was asked to recommend a future portfolio of activities, including desirable changes to data governance and statistical programs, and ways to increase the impact of education statistics produced by NCES. As the nation’s principal statistical agency focused on education and related topics, NCES interacts with stakeholders ranging from Congress and the White House to students and their families. The panel considers NCES’s stakeholders to include:

• The Department of Education (ED) including the Institute of Education Sciences (IES);
• State education agencies (SEAs);
• Local education agencies (LEAs);
• Education practitioners, such as teachers and education administrators;
• Education consumers, such as students, parents, families, and employers;
• Congress and the President;
• Other federal agencies, especially the principal statistical agencies;
• Other data-holding agencies and organizations;
• State and local policy makers and the policy community;
• The civil rights-monitoring community;
• Advisory boards and professional associations; and
• Researchers and academia.

These stakeholders have varying needs, but common themes include access to data; data products that are useful, timely, and user friendly; feedback from users on data needs; data governance and standards setting; and technical assistance and guidance. While NCES cannot be all things to all stakeholders, it can strategically focus on high-value activities that result in multiple beneficial outcomes. NCES also produces population-level statistics on education for the nation, and the panel suggests that this core role continue. This chapter addresses methods by which NCES could increase its impact through engagement with stakeholders; incorporating user feedback into products and services; expanding its role in data governance and facilitating data access; enabling SEAs to build data-linkage infrastructure; and promoting and delivering products that are accessible, actionable, and timely.

In October 2021, the NCES commissioner introduced a draft stakeholder map (Figure 4-1), which provides a useful perspective of NCES’s stakeholders along two dimensions: criticality to mission and level of engagement.¹ The panel finds this approach useful, though one might argue about the placement of some of the stakeholders within the map. One issue is whether the map is meant to be descriptive of current stakeholders or to reflect preferred stakeholder relationships. Another issue is that the role of a stakeholder depends on whether that stakeholder is viewed as a consumer of NCES products or as a contributor to those products. For example, other federal statistical agencies do not currently play a large role with respect to NCES’s mission but partnering with such agencies could create valuable linkages with other types of education-relevant data. Data scientists could potentially be valuable in providing alternative types of data and alternative sources that may limit the need to conduct surveys. It is interesting that the map lists families but not students (who are perhaps meant to be incorporated within families in some cases). The list does not include employers, who are stakeholders both in terms of wanting qualified employees and more specifically in terms of their interest in vocational education. The panel recommends that these issues be addressed in NCES’s strategic-planning process.

CREATE ENGAGEMENT FEEDBACK LOOPS TO ENSURE RELEVANCE OF PRODUCTS AND SERVICES

Engage Broadly for Continuing Feedback

It is challenging for NCES to fully understand the needs of its many stakeholders. NCES has created and participated in many forums, in which it

___________________

¹ Peggy Carr, NCES, “Partnership, Innovation, and Equity: A Vision for NCES Now and into the Future,” presentation at the Association of Public Data Users, October 21, 2021.

both provides information and obtains feedback.² NCES’s efforts are particularly strong with regard to the National Assessment of Educational Progress (NAEP).³ Additionally, NCES participates in other groups not specifically devoted to NAEP. The National Forum on Education Statistics includes about 100 members, half of whom represent states, with the remainder including representatives from federal offices involved with education data, representatives from education agencies in extra-state jurisdictions, and national organizations with an interest in elementary and secondary education data (NCES, 2021h). These members are presumed to represent the needs and concerns of all 50 states, the District of Columbia, the U.S. territories, and over 13,000 school districts. The National Postsecondary Education Cooperative is charged with developing a research and development agenda focusing primarily on the Integrated Postsecondary Education Data System (IPEDS). NCES works with multiple panels, including a technical review panel for each sample survey data collection, panels coordinated through the National Institute of Statistical Sciences (NISS), a standing panel of K–12 principals to share ideas about new data collections and products, and the NCES Teachers Panel. Additionally, NCES is a member of the Children’s Forum, works with several interagency working groups focused on content needs across the federal statistical system, and a group of private-school stakeholders, also including representatives from the ED’s Office of Non-Public Education. NCES has collected information from stakeholders for specialized purposes: NAEP collected information from stakeholders, resulting in a Future of NAEP plan of action, and the Ed Tech Equity Framework was developed using stakeholder meetings and panels. Although the Advisory Council on Education Statistics no longer exists⁴ and there is no group representing the needs of consumers, such as students, their families, employers, and practitioners, NCES has many avenues for engagement. The panel applauds NCES for its efforts to reach out and listen to stakeholders and encourages NCES to make full use of these resources for improvements to products and services.

___________________

² NCES response to question from the panel, pp. 24–25.

³ NAEP has the National Assessment Governing Board to set policy, meets with the Council of Chief State School Officers and Council of the Great City Schools on the strategic development of NAEP, funds NAEP state coordinators embedded in state education departments to support NAEP, leads a contracted service center that provides a forum for interaction between NCES and others, has the NAEP Design and Analysis Committee to provide psychometric and large-scale assessment technical support to NCES, has the NAEP Validity Studies Panel to provide input relating to the validity of NAEP, and has standing committees of subject-matter experts for NAEP-assessed topics.

⁴ The passage of the Education Sciences Reform Act disbanded the Advisory Council on Education Statistics (established under 20 U.S. Code § 9006) and formed the National Board for Education Sciences (20 U.S. Code § 9516). Available: https://www.govinfo.gov/content/pkg/USCODE-1996-title20/pdf/USCODE-1996-title20-chap71-sec9006.pdf and https://www.law.cornell.edu/uscode/text/20/9516 [March 2022].

NCES conducts some broad, grassroots, technical-assistance efforts that are not well promoted. The Center has a blog that is updated at least monthly, with content generated by a mix of staff and contractors, but the reach of the blog is unclear. The main communication vehicle for sending notifications (via email) when new data or reports are published is IES’s Newsflash Subscription Service, which sends information for IES and all of IES’s centers.⁵ NCES’s website provides a form for submitting questions and comments and also provides resources for help with IPEDS, restricted-use data licenses, and general inquiries, but these resources may be more relevant to providing technical assistance as opposed to obtaining feedback.⁶ NCES also participates in the American Consumer Satisfaction Index Survey.⁷ When using such tools, it is important to obtain actionable information that results in improvements to the Center.

RECOMMENDATION 4-1: NCES should deepen and broaden its engagement with current and potential data users, to gather continuing feedback about their needs and ways that NCES can meet those needs more effectively. This feedback will help NCES shape its efforts to develop and disseminate standards, provide technical assistance, and strengthen its user community.

NCES can leverage existing vehicles to expand its engagement and mission impact in many ways. For example, NCES currently offers Distance Learning Dataset Training (DLDT) (NCES, 2021f) which, while targeted particularly towards new users, provides self-paced, dataset-specific modules to deliver information about studies conducted and data acquired by NCES. DLDT engages its user community via Facebook (3.1K followers⁸), Twitter (24.7K followers⁹), and NCES’s blog (number of followers unknown). In 2020, NCES, in collaboration with contractors, created a new user community of graduate students as part of the Coleridge Initiative, by running a trial program to allow students remote access to restricted-use survey data (Schneider, 2021). Another possibility for increasing awareness and use of NCES data might be to coordinate with teachers of applied statistics in education, providing a repository of replication datasets for papers and possibly adding lesson plans.

The panel believes that NCES has a solid foundation for providing technical assistance and supporting new users, but the Center can go further

___________________

⁵ NCES response to question from the panel, pp. 37–38.

⁶ See: https://nces.ed.gov/help/webmail/ [March 2022]; https://nces.ed.gov/ipeds/contact-us [March 2022]; https://nces.ed.gov/pubsearch/licenses.asp [March 2022].

⁷ NCES response to question from the panel, p. 26.

⁸https://www.facebook.com/EdNCES [March 2022].

⁹https://twitter.com/EdNCES [March 2022].

to consider the impact of the DLDT program and to solicit and incorporate feedback on this program. What do DLDT users think about the program? Are DLDT members representative of NCES usership? Can DLDT usership be broadened? Can successful components of the DLDT model be expanded and less successful components improved? Does NCES employ other mechanisms to engage users? How does NCES collect feedback to understand user needs and how does the Center address such feedback within its strategic priorities? For example, one-on-one stakeholder discussions may reveal that a stakeholder’s specific needs are best met outside of NCES, but that NCES may still play an important role in data facilitation. One good example for soliciting and incorporating feedback is the U.S. Department of Agriculture’s data-user reviews of the National Household Food Acquisition and Purchase Survey (NASEM, 2020; Wilde and Ismail, 2018).

Many statistical agencies develop ongoing relationships with their user communities. For instance, the U.S. Census Bureau’s American Community Survey (ACS) represents a best practice.¹⁰ The ACS Office solicits feedback and questions from all users via email and phone, and it partners with the Population Reference Bureau, which hosts the ACS Data Users Group—a community that shares information, materials, and events.¹¹ Agencies comparable in size to NCES have developed methods for receiving broad user feedback and/or communicating directly with users via topical webinars or emails announcing product releases, for instance. NCES may consider creating a data-user advisory group to direct the Center’s outreach efforts.¹² NCES can also extend its use of technical review panels, such as its Technical Review Panel for IPEDS, which obtains input on IPEDS-related project plans and products and fosters communications with potential data users (RTI International, 2021).

Broad engagement is important for identifying and monitoring current and emerging issues, which drive product and service effectiveness. Ideally, broad engagement will develop into a user community that supports knowledge sharing and spurs innovation among users and NCES. Such a synergy has many positives. When users help each other, the need for NCES to provide technical assistance is lessened. When NCES receives feedback and input, it can more readily adapt to changing needs for education data and services. Finally, as NCES engages a broad community of users, its key role in the education data ecosystem will become prominent.

___________________

¹⁰ See https://acsdatacommunity.prb.org/ [March 2022].

¹¹ See https://acsdatacommunity.prb.org/ [January 2022].

¹² For additional ideas for user engagement, see Recommendation 6.7 in NASEM (2021c).

Engage for Data Acquisition and Best Practices

In addition to ED and IES, other federal, state, nonprofit, and commercial organizations that hold and steward data are also critical stakeholders for NCES. If NCES were to partner strategically with other data-holding organizations, it could share best practices, expand its data holdings, advance evidence building, spur innovation, and create broad mission impact.

RECOMMENDATION 4-2: NCES should actively collaborate with other data-holding federal agencies and organizations to develop useful products and processes, including those that utilize data from alternative sources, to provide timely, policy-relevant insights.

NCES is connected with other federal statistical agencies through the Interagency Council on Statistical Policy, but it could more fully leverage those relationships to advance education statistics and insights. For instance, NCES has partnered with the U.S. Census Bureau to produce the Education Demographic and Geographic Estimates (EDGE) program but has not linked person-level data to the U.S. Census Bureau’s vast holdings (U.S. Census Bureau, 2021b,e). There are only two NCES datasets available for use (and potential linkage) in the secure Federal Statistical Research Data Centers run by the U.S. Census Bureau, both fielded by the U.S. Census Bureau on NCES’s behalf: the Current Population Survey School Enrollment Supplement and the National Crime Victimization Survey School Crime Supplement.¹³ NCES could enhance the value of its data by submitting and enabling more data for linkage and analysis, to gain new insights into education.

The U.S. Census Bureau has multiple joint projects with other federal, state, local, and nonprofit organizations to link data creatively to understand crosscutting issues in demography and social science (U.S. Census Bureau, 2021c).¹⁴ Three such projects involve education data that are not from NCES:

• The Post-Secondary Employment Outcomes project with the University of Texas System, seeking to understand students’ earnings, employment, and other outcomes by major field of study, which has now expanded to 11 states (U.S. Census Bureau, 2021d);¹⁵

___________________

¹³ According to U.S. Census Bureau response to question from the panel, as of December 21, 2021, there was one proposal to use these datasets, and no active or completed projects.

¹⁴ See also U.S. Census Bureau document provided to the panel, “Evidence Building Projects.”

¹⁵ See also David Troutman, “University of Texas System Partnership with the U.S. Census Bureau,” presentation to the National Academies of Sciences, Engineering, and Medicine, July 9, 2021.

• The Using Multiple Discontinuities to Estimate Broad Effects of Public Need-Based Aid for College project, using state data on financial aid to evaluate the impacts of the Wisconsin Grant on college enrollment, employment, wages, migration, and receipt of social benefits programs;¹⁶ and
• The Outcomes Under the Post-9/11 GI Bill project, studying the return on investment of the post-9/11 GI Bill compared to other GI Bill and veteran benefit programs, using data from the National Student Clearinghouse (a nonprofit data-holding organization) and partnering with the Department of Veterans Affairs.¹⁷

Multiple federal agencies partner with the U.S. Census Bureau on other joint projects using data linkage or making linked data available. For example, the Economic Research Service has Research Innovation and Development Grants in Economics to develop innovative research on food- and nutrition-assistance issues, and the Department of Housing and Urban Development has deposited Moving to Opportunity and Family Options Study experimental data at the U.S. Census Bureau so that researchers can study long-term outcomes of housing interventions. NCES can increase the value of its data by making more data available to researchers via the U.S. Census Bureau’s infrastructure.

The U.S. Census Bureau is not the only agency engaging with other organizations to support data linkage for evidence building and innovative research. The Social Security Administration’s statistical agency, the Office of Research, Evaluation and Statistics, runs an extramural research program through its Office of Data Development (Social Security Administration, 2021). The Treasury Department’s statistical agency, Statistics of Income, runs its own joint statistical research program (Internal Revenue Service, 2021). The National Center for Health Statistics (NCHS) has a research data center and multiple partnerships using linked data (Centers for Disease Control and Prevention, 2021). The Administration for Children and Families is not a statistical agency, yet it provides access to the National Directory of New Hires data, has multiple research projects involving linked data, and is seeking more opportunities to use and link administrative data to better understand its programs.¹⁸

Many states and nonprofit organizations also see the benefits of partnering and sharing data. The National Student Clearinghouse, the Manufacturing Institute, Child Trends, and Data Quality Campaign are

___________________

¹⁶ U.S. Census Bureau document provided to the panel, “Evidence Building Projects.”

¹⁷ U.S. Census Bureau document provided to the panel, “Evidence Building Projects.”

¹⁸ See, for example: https://www.acf.hhs.gov/opre/project/child-maltreatment-incidence-data-linkages-cmi-data-linkages-2017-2022 [March 2022] and Holman et al. (2020).

all organizations integrating data to understand access to education and student outcomes outside of schooling. The Western Interstate Commission for Higher Education is a multistate partnership for longitudinal data exchange, involving 15 western states and the U.S. Pacific Territories and Freely Associated States (Western Interstate Commission for Higher Education, 2020). Similarly, the Midwest Collaborative, supported by the Coleridge Initiative and the National Association of State Workforce Agencies, aims to exchange data across state lines (Midwest Collaborative, 2020). Due in part to NCES leadership and investment in the Statewide Longitudinal Data Systems (SLDS) Grant Program, Massachusetts integrates data across its education agencies and other government offices to develop longitudinal data on, for example, student and teacher characteristics, student education outcomes, and employment outcomes. All these agencies have partnered with other data-holding agencies to be more useful and valuable to stakeholders.

NCES has many opportunities to support innovative evidence building by combining data with other data-holding agencies. There is vast potential for increased understanding of students, teachers, and schools, as well as other actors and facets of education. Expanding the number of engagements and potential data sources does not mean that NCES itself must produce new product lines from those data—this can be achieved with a relatively high value-to-effort ratio. There are other benefits to partnering in addition to increasing mission impact, such as sharing best practices, providing mutual support, and spurring innovation.

As evidenced by the recent advisory report Setting Priorities for Federal Data Access to Expand the Context for Education Data (NISS, 2021c), NCES is seeking to increase the value of its data. The report provides many ideas for federal partnerships to increase the value of NCES’s data. NCES is well positioned for increasing its partnerships, as it is already connected to the heads of other statistical agencies and departmental statistical officials through the Interagency Council on Statistical Policy and the Federal Committee on Statistical Methodology. In the panel’s opinion, NCES can and should expand its network to support creative evidence building and increase its mission impact.

Receive and Use the Help of an External Review Body

In addition to engaging broadly, the panel recommends that NCES routinely receive recommendations from experts and address those recommendations. The Advisory Council on Education Statistics used to provide

recommendations to NCES.¹⁹ However, the Council was disbanded after the 2002 Education Sciences Reform Act²⁰ legislation, which moved NCES under the umbrella of IES and established the broader-scoped National Board for Education Sciences (IES, 2021b). While this board still exists, it has been inactive in recent years and is more predominantly focused on other IES centers than was the previous advisory committee, which focused solely on NCES. The panel believes that NCES could benefit greatly from a review body focused solely on reinforcing NCES’s strategic goals and helping the Center to meet them.

RECOMMENDATION 4-3: NCES should explore and establish creative models for a nimble, ongoing consulting body, supplemented by a pool of ad hoc consultants, to help NCES innovate and be accountable for progress on strategic goals.

To keep the consulting body nimble and flexible, the panel recommends that the body not be subject to Federal Advisory Committee Act (FACA) regulations. The body might contain a set of regular members with knowledge of the full scope of NCES’s activities, to provide strategic advice and accountability, along with additional and varying participants (e.g., NISS, 2021b) depending on the particular expertise necessary at a given time. The consulting body would have moral but not statutory authority with regard to NCES, and might also at times provide backing when NCES faces difficult decisions. The goal is a continuing relationship between NCES and the consulting body, in which the consulting body provides advice, helps with strategic planning, addresses special needs as they arise, and supports NCES’s decision making. The consulting body would also serve as a source of innovative ideas, helping to prevent insularity and providing motivation and support for continual improvement. Examples of the potential work of the consulting body follow, along with examples of similar kinds of activities, some from FACA committees.

When used properly, an external consulting body is a valuable ongoing resource for NCES, particularly as the Center adapts to maintain relevance (NASEM, 2021b, Principle 1). The Scientific and Public Affairs Advisory Committee within the American Statistical Association (ASA) could serve this role, as could a special committee created in coordination with the ASA specifically for this purpose. The Council of Professional Associations on Federal Statistics is another potential source of a review body.

___________________

¹⁹ See: https://www.govinfo.gov/app/details/USCODE-1996-title20/USCODE-1996-title20chap71-sec9006/summary; repealed 2002: https://www.law.cornell.edu/uscode/text/20/9001 [March 2022].

²⁰ See: https://www.law.cornell.edu/uscode/text/20/chapter-76/subchapter-I [March 2022].

A consulting body is an additional source of stakeholder input and can provide advice, support, validation, and accountability to help NCES stay on target and implement its strategic goals. With enough information (e.g., costs, users, uses, level of effort, beneficiaries) the consulting body can assist NCES with decisions, including prioritizing and deprioritizing data collections. Importantly, the review body can validate NCES’s decisions and serve as a key ally, providing reinforcement to NCES in the wake of difficult or controversial decisions.

Many federal agencies have formal advisory committees, most, but not all, of which fall under FACA regulations. Importantly, such boards are only as useful as the agency chooses to make them. Successful agency/advisory committee relationships are characterized by true interaction, discussion, and mutual feedback. The Federal Economic Statistics Advisory Committee is one example. This committee advises the Bureau of Economic Analysis (BEA), the U.S. Census Bureau, and the Bureau of Labor Statistics (BLS), and handles “statistical methodology and other technical matters related to the collection, tabulation, and analysis of federal economic statistics.”²¹ BEA also works well with its BEA Advisory Committee, which focuses on “the development and improvement of BEA’s national, regional, industry, and international economic accounts, especially in areas of new and rapidly growing economic activities arising from innovative and advancing technologies.”²² In the panel’s opinion, the BEA Advisory Committee has been instrumental in supporting BEA’s exploration of alternative data sources for measuring new and emerging economic topics.

The NCHS Board of Scientific Counselors, although subject to FACA, is a particularly good example of a successful advisory committee. The Board of Scientific Counselors advises NCHS on “goals and objectives, strategies, and priorities,” “statistical and epidemiological research and activities that focus on various health issues” and “about opportunities for NCHS programs to examine and employ new approaches to monitoring and evaluating key public health, health policy, and welfare policy changes.”²³ The scope of NCHS’s board could be the appropriate scope for an NCES review board.

A more deeply engaging and helpful model is that of the Energy Information Administration (EIA), whose Committee on Energy Statistics advisory board is run via the ASA.²⁴ The structure of the Committee on Energy Statistics’ work is unusual because members engage with EIA staff directly in the

___________________

²¹ See: https://apps.bea.gov/fesac/ [March 2022].

²² See: https://www.bea.gov/beaacm [March 2022].

²³ See: https://www.cdc.gov/nchs/about/bsc.htm [March 2022].

²⁴ See: https://www.eia.gov/about/stakeholders.php; and https://ww2.amstat.org/committees/commdetails.cfm?txtComm=CCNARS03 [March 2022].

early phases of projects. This means that EIA staff must be open with information and ideas before they typically feel comfortable sharing with external stakeholders. It can be challenging to find staff who are willing to work with outside academics and experts in this intimate way. EIA has found that once staff agree to participate in this joint project, the rewards are multifold. For example, review board members collaborate with staff as they build and develop ideas for projects, so that the members’ advice has greater impact on the success of the results. Staff members develop in-house expertise and institutional knowledge. Moreover, staff and members typically become very engaged. Using this model, review board members often report great satisfaction from helping EIA—more than they would if presented with a project in the late stages. In fact, review board members, not EIA staff, often present projects at committee meetings.

EIA’s review board, a non-FACA committee, is a good model for NCES. NCHS’s board may be a good model for leveraging a standard FACA advisory committee. Regardless of the model chosen, NCES’s programs, staff development, culture of innovation, and relationship building would greatly benefit from a nimble consulting body that could help the Center make decisions and meet its goals.

EXPAND NCES’S ROLE ENABLING DATA ACCESS TO SERVE AND ENGAGE STAKEHOLDERS

Facilitate Data Access and Use

The panel finds that NCES focuses heavily on its primary data collections and products to achieve mission impact. However, as discussed above, NCES can expand the value of existing data products by providing key services that have high return on investment of effort. Further, the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act)²⁵ effectively expanded NCES’s mission to include data governance and facilitation for evidence-building purposes. The panel strongly encourages NCES to expand its role as a data facilitator in the education data ecosystem.

CONCLUSION 4-1: NCES can expand its impact by providing leadership and expertise to facilitate responsible data use and access. NCES can help organizations develop capacity to integrate and analyze education data and other data, to produce actionable analyses.

NCES can maximize its mission impact by expanding its role in data governance (i.e., in helping create the processes and metrics to aid the use of

___________________

²⁵ See: https://www.congress.gov/bill/115th-congress/house-bill/4174/text [March 2022].

information). NCES already performs aspects of data governance, in terms of setting standards for data, statistics, and privacy for state data providers, and as part of ED’s Data Governance Board. The Common Education Data Standards program is critical in setting standards for state data systems and works closely with agency governance leads throughout the nation. NCES also issues licenses for use of restricted data. NCES can expand its governance role by assisting states and other organizations in data linkage.

NCES can streamline data linkage, prepare and curate data for ease of linkage, develop data-sharing agreement and proposal templates, and simplify data-access processes. NCES can also guide state and local data providers in data infrastructure (e.g., by vetting data-sharing projects and analysts), while simultaneously considering secure-access environments, data content, and output products (the “Five Safes” [Desai et al., 2016; Wikipedia, 2021b]). NCES can also refer to the Advisory Committee on Data for Evidence Building’s recommendations for establishing a national secure data service (Advisory Committee on Data for Evidence Building, 2021) as another valuable source for best practices in data facilitation and infrastructure.

Expanding NCES’s data governance role will require internal resources, but the value of facilitating responsible data access with strong governance will provide high return on NCES’s investment. By increasing data access, more external analysts can participate in evidence building for decision making. Moreover, by setting standards for data and governance processes, NCES can better manage data quality and transparency in data design and provenance—two of NCES’s strengths. Eventually, NCES can create a virtuous cycle, as its initial investments lead to a stronger data-governance and facilitation role in the broader education data ecosystem.

Leverage NCES’s Strengths to Support State and Local Education Agencies

As noted throughout this study, NCES can invest internal resources to activate external resources as a force multiplier, for broader mission impact. To maximize use of external resources, NCES needs to align these resources to its strategic plan. The panel finds that NCES is underutilizing its ability to engage the states in achieving the Center’s strategic goals.

RECOMMENDATION 4-4: NCES should strengthen state capacity to link data across systems, adopt shared data standards, and provide actionable information to state and local education agencies to help improve student learning outcomes. NCES can leverage its Statewide Longitudinal Data Systems Grant Program to achieve this goal.

Like most federal statistical agencies, NCES influences its data collections and secondary data acquisitions, including its state and local data providers, by setting standards and governance processes. Unlike many of its sister agencies, NCES has substantial financial influence via its SLDS grants. As part of implementing its strategic plan, the panel suggests that NCES be thoughtful about the goals and outcomes of SLDS awards, to align others’ resources to the Center’s goals. NCES can then circulate these intentions to the states and award those proposals that meet the criteria.

The panel finds that certain strategic activities may be particularly high value for broadening NCES’s mission impact. In the panel’s opinion, NCES should support SEAs in observing shared data standards and governance processes. NCES should also support infrastructure investments for states and state consortiums that want to integrate data for evidence building. This includes prioritizing states’ evidence-building and data-linkage capacity and identifying high-value linkages with data systems across other social domains (e.g., justice, health, well-being), both across states and with other government agencies.

Importantly, the panel finds that LEAs often lack resources such as data analysts and managers. LEAs need data access and technical assistance. To address these gaps, NCES could award SLDS grants for states to share data with LEAs, facilitate LEAs’ access to data for evidence building, and provide technical assistance. SEAs could also offer analytic assistance, using state and local data to help agencies diagnose and prioritize issues faced in their schools, measure implementation and impact of programs or policy changes, and perform predictive modeling of the results of such changes. SLDS grants could facilitate collaborations among regions or groups of LEAs and could also generate products and tools useful to LEAs and SEAs. This would be a large task given the lack of a national curriculum or common standards, but NCES could be a catalyst for creating standards. Such tools might be offered to other states and localities, to scale their impact. In other words, NCES could capitalize on the existing SLDS infrastructure to maximize its reach and impact.

Another way to mobilize states to conduct high-value activities is to fund a data and statistics liaison at each state, possibly through SLDS grants, who would help to coordinate between state personnel and NCES. This general model already exists at NCES, as NCES-funded state-level coordinators for NAEP, and a review of the NAEP state coordinator role might help to create NCES state coordinators. Ideally, a state liaison for data and statistics would focus on the building and maintenance of data systems, data analysis, data collection, and reporting.²⁶ The liaison could execute data-governance procedures in terms of standards, data-access

___________________

²⁶ Carrie Conaway, testimony provided to the panel, “NCES-funded position in SEAs.”

processes and security, building and testing high-quality data linkages, and collecting longitudinal data.²⁷ The liaison could also produce estimates and other analyses for local areas, produce federally required reports, and support new NCES data collections, such as quick-response surveys.²⁸ A structure of liaisons might also facilitate cross-state collaboration. While NCES is advised to establish such liaison positions as soon as possible, the activities of the liaisons need to align with NCES’s strategic plan, to increase the Center’s effectiveness with relatively minor investments of internal resources.

Partner with External Researchers and Analysts for Evidence Building

As described in the Evidence Act:

• Statistical agencies are presumed to be able to access any data asset held by any executive agency for the purpose of developing evidence, with limitations.²⁹
• Statistical agencies must expand access to their data for the purposes of developing evidence, while also protecting the data from inappropriate access and use.³⁰
• Statistical agencies must establish an agency-level process adhering to the Standard Application Process (established by the director of the Office of Management and Budget) so that “agencies, the Congressional Budget Office, state, local, and Tribal governments, researchers, and other individuals, as appropriate, may apply to access” data for evidence building.³¹

To support NCES’s expanded role in data governance and evidence building, NCES can leverage its existing data-licensing program to further expand access to data while also directing and participating in research and analysis.

RECOMMENDATION 4-5: NCES, in collaboration with the Institute of Education Sciences, should establish a joint statistical research program that includes matching internal staff with highly qualified

___________________

²⁷ Ibid.

²⁸ Ibid.

²⁹ 44 U.S. Code § 3581 – Presumption of accessibility for statistical agencies and units. Available: https://www.law.cornell.edu/uscode/text/44/3581 [March 2022].

³⁰ 44 U.S. Code § 3582 – Expanding secure access to CIPSEA data assets. Available: https://www.law.cornell.edu/uscode/text/44/3582 [March 2022].

³¹ 44 U.S. Code § 3583 – Application to access data assets for developing evidence. Available: https://www.law.cornell.edu/uscode/text/44/3583 [March 2022].

external researchers, statisticians, and data scientists to develop new data analyses, tools, and publications.

ED will make faster progress in its evidence-building efforts by both broadening the community of researchers and policy makers who can access data for analytics and by signaling important topics, such as those for which ED especially wants data. For that purpose, we recommend that NCES establish a joint statistical research program (JSRP) for external researchers and fellows. By establishing a JSRP aspect to its data-licensing program, NCES can provide data access to external analysts, while also collaborating with those analysts to learn about the Center’s datasets. A JSRP can be used to improve methods, evaluate data quality and fitness for purpose, study trends, and understand questions and emerging issues. External analysts can contribute innovative ideas, benefiting NCES staff and supporting a learning-centered environment. Ultimately, a JSRP builds research, analysis, and evidence on important topics in education. Examples of other statistical agencies’ JSRPs and similar programs have been discussed, including those of Statistics of Income; the Office of Research, Evaluation, and Statistics; the Economic Research Service; NCHS; and the U.S. Census Bureau (for examples of criteria for access, see U.S. Census Bureau, 2021a). Another example is the BLS’s collaborative Senior Research Fellow Program, conducted in conjunction with the ASA and funded by a grant from the National Science Foundation (Bureau of Labor Statistics, 2021).

In addition to joint research, the panel feels that NCES can and should expand and modernize its data-licensing program to further increase responsible data access for evidence building. Data licensing is often somewhat ad hoc, in that researchers request licenses to pursue research with little relationship to NCES’s priorities. Such requests pose little burden on NCES other than license administration, and NCES has an interest in maintaining such licensing to promote the Center as a source of quality data. At the same time, NCES could promote specific types of data-access applications, in which the highest-priority topics and questions for evidence building (e.g., America’s Datahub Consortium, 2021; U.S. Environmental Protection Agency, 2022) have been jointly determined by the commissioner of NCES (ED’s statistical official) and the commissioner of the National Center for Education Evaluation and Regional Assistance (NCEE) (ED’s evaluation officer). By doing so, the evidence developed will build on previous research to advance knowledge on a subject, rather than resulting in an expanded but disconnected set of analyses. NCES could further encourage such research through a secondary analysis grant program, such

as its former program to promote the use of NAEP,³² or other IES grant program³³ with criteria designed to encourage the desired research.

Both broadening the size of the research community using licensed data and increasing the diversity of such users could benefit NCES. Current licensees tend to be R1 elite universities, but increasing diversity could generate new perspectives and unique uses of data. Similarly, NCES might consider expanded licensing for graduate students, who might use the data for their dissertations. Generally NCES does not allow graduate students to obtain licenses except through a sponsoring professor but, during the COVID-19 pandemic, NCES ran a trial program to give graduate students online remote access to data through a program called the Coleridge Initiative. This and related approaches might be explored further. Graduate students tend to be more diverse than faculty, so increasing their access could increase the diversity of NCES’s data users and add new perspectives.

To fully leverage a strategic, extramural research program, the panel suggests that NCES establish requirements for external researchers to share what they learn with NCES and NCEE, so that external research projects will still support NCES’s learning and innovation. Additionally, external projects can feed back into IES’s information systems and contribute to broad dissemination tools, such as NCEE’s What Works Clearinghouse, or other dissemination venues, such as NCES or NCEE webinars.³⁴

The goal of increasing data licensing is not only to increase the number of individuals working with licensed data, but also to create a community of users who share research results and best practices. Such a community could provide internal support to data users while increasing the visibility of NCES’s data and facilitating participation of new users.

NCES could also expand its analytic reach by partnering with other agencies that have similar extramural research programs. In these partnerships, the panel recommends that NCES set the strategic analytic agenda and require feedback on the data and analyses performed. Such collaborations could aid NCES both by leveraging external federal resources and best practices to expand the Center’s extramural research program, and by leveraging external analytic resources to expand education evidence building.

___________________

³² See IES’s NAEP Secondary Analysis Grants Program awards from 2002 to 2007: https://ies.ed.gov/funding/grantsearch/index.asp?mode=1&sort=1&order=1&searchvals=&SearchType=or&checktitle=on&checkaffiliation=on&checkprincipal=on&checkquestion=on&checkprogram=on&checkawardnumber=on&slctAffiliation=0&slctPrincipal=0&slctYear=0&slctProgram= 40&slctGoal=0&slctCenter=0&FundType=1 [March 2022].

³³ See: https://ies.ed.gov/funding/overview.asp [March 2022].

³⁴ See: https://ies.ed.gov/ncee/wwc/ [March 2022].

IMPROVE DISSEMINATION, FOCUSING ON ACCESSIBILITY AND USEFULNESS

Ensure Accessibility and Usability of NCES’s Products and Tools

Multiple studies have assessed NCES’s products, tools, and website for usability and recognition of NCES’s role (NISS, 2021a; NISS, 2021c; NISS, 2020; NISS, 2016). The panel finds this area in need of further improvement. The panel applauds NCES’s efforts to make complex data available to the public via tools such as the College Navigator (NCES, 2021c). However, in recent years, people have come to expect higher standards in terms of user accessibility and internet search capabilities. For example, compare NCES’s College Navigator to ED’s College Scorecard (U.S. ED, 2021a), both of which serve a similar audience: students and their families. The latter has a simple layout and is easy to read and use, with toggles or tabs to control search options.

Improving data accessibility requires thoughtful review and modernization of key resources directed at general audiences. To support equity and inclusion, government agencies need to be transparent and accessible to the populations they serve. In terms of mission impact, NCES would benefit from applying accessible, human-centered design to its website, products, and tools. Sometimes, rather than creating new tools itself, NCES might make use of others’ efforts by certifying systems that meet high standards, if this is allowed within NCES’s mandate. IES’s What Works Clearinghouse does something similar in assessing the quality of evaluations.

CONCLUSION 4-2: NCES can improve the accessibility and usability of its products, tools, website, and other dissemination platforms to allow a broader range of audiences to benefit from its products.

NCES can also extend its impact by ensuring that its website, statistical products, and tools are accessible to diverse audiences, particularly those that lack data and analysis resources, such as many state and local school districts and consumers (e.g., students, parents, employers). To reach these audiences, NCES needs to communicate the existence of its products while also providing products and tools in user-friendly, inclusive formats on an intuitive website that allows users to find information easily.

Accessibility means more than applying 508 compliance (U.S. General Services Administration, 2021a,b) so that people with disabilities can perceive and understand web content. Today, accessibility includes usability, which means applying human-centered design, so that products are easy for everyone to use and understand (University of Washington, 2021; Henry et al., 2016). Language can factor into accessibility, in terms of both

inclusivity for non-English speakers and a reading level that supports ease of use. The panel suggests engaging experts to design accessible products and a website that accounts for the ways people, including those with disabilities, currently search for and digest online information. The goal is to build and broaden NCES’s general user base while also making data discovery and access easier for even its more tech-savvy users.

Web usability can be difficult to address because, for people who are knowledgeable, everything may seem easy to find. For novices who are less familiar with NCES and its products, however, finding desired material can be much more difficult. Specifically, NCES’s website is organized around surveys and programs, while a novice user might want to search by topic area. To understand the experience of new users, NCES could both talk to users about their needs and assess their processes when asked to find specific information on the Center’s website.³⁵ Eye-tracking technology can reveal the foci of users’ attention as they search for information. Examples of well-designed websites, at varying levels of complexity, could guide NCES’s website usability efforts.³⁶

Developing more application programming interfaces (APIs) (i.e., software that provides a way of submitting requests for data and obtaining results, as when checking the weather from one’s phone) could also help users identify and access data. For example, the Urban Institute’s Education Data Portal project provides most of NCES’s administrative data in API format for ready access and adds further value by harmonizing variables for easy comparison across datasets and time.³⁷ College Navigator could possibly be run through an API, or topical searches could be addressed through an API. NCES might sponsor a hackathon or competition to promote the development of APIs, which, at the same time, could further promote NCES.

Create Actionable Products and Tools for Local and State Agencies

The content of products and tools is also important for building a broad user base. Individual stakeholders have unique needs in terms of content, quality of information, and timeliness—what is relevant and useful for one group may not be for others. NCES can build its user base, both by creating useful tools and by making well-documented, reliable data available to developers who can customize the data for the unique purposes of stakeholders.

___________________

³⁵ For additional ideas for engaging users on accessibility and usability, see Recommendation 6.7 in NASEM (2021c).

³⁶ See: The Opportunity Atlas at https://www.opportunityatlas.org/; the Stanford Education Data Archive at https://edopportunity.org/; and ED’s Common Education Data Standards at https://ceds.ed.gov/Default.aspx [March 2022].

³⁷ See: https://educationdata.urban.org/documentation/ [March 2022].

Practitioners within LEAs want to understand their school districts in context, for example, by comparing themselves to the rest of the nation or to schools with similar demographics. These practitioners want to understand how similar or different their LEAs are and what actions can be taken to improve their schools and student outcomes. However, many practitioners within LEAs, and to a lesser extent SEAs, often lack data-analysis skills and tools for even the initial step of selecting comparable districts or schools. This means that pure data summaries, without explanatory text, have limited usefulness. Even short briefs containing descriptive statistics and light explanatory text may not suffice to help practitioners understand their LEAs or SEAs in comparison to others, let alone help them to take steps to improve. Providing tools to easily select and examine comparable districts or schools could be very helpful.

Timeliness, in which information is available when it needs to be used in decision making, is another dimension of usefulness. As with accessibility, the definition of timeliness has shifted. In this new era of fast and available (but questionable-quality) data, the careful preparation, collection, processing, and dissemination of high-quality survey data and analyses can seem slow, unresponsive, and irrelevant. There are conflicting priorities and incentives for data that adhere to rigorous methods, standards, and quality compared to data that are timely and relevant. For practitioners and policy makers, the tradeoff often favors timeliness and relevance, even if information is approximated or of moderate quality.

RECOMMENDATION 4-6: NCES should release data and data products that are useful, actionable, and timely for local and state education agencies and other stakeholders. To increase timeliness, NCES, in collaboration with the Institute of Education Sciences, should review and revise its internal and external quality assurance processes.

To assist state and local school districts that have few resources for data analysis, NCES needs to deliver products that help LEAs improve their districts’ schools and student outcomes. Products such as NCES’s Public School District Finance Peer Search (NCES, 2021i) need to be extended to other topics and could be improved by the addition of analytic features, tools, or templates that provide statistical testing for samples.³⁸ Products that include links to curated resources to assist LEAs with researching interventions can directly address NCES’s usefulness and actionability.

The NISS reports (2021a,c) provide additional examples of products and services that could increase the usefulness of NCES’s data for LEAs,

___________________

³⁸ See, for example, U.S. Census Bureau’s Statistical Testing Tool. Available: https://www.census.gov/programs-surveys/acs/guidance/statistical-testing-tool.html [March 2022].

such as localized summary statistics and/or model estimates for “schools like ours.” Expanding on these ideas, the panel sees value in a tool that would allow districts to input key student demographics (e.g., exam or graduation statistics) to produce estimated, localized, score-distribution percentiles to contextualize data. This would require NCES (or its contractors) to model state data with those key demographics. For infographics, briefs, and practitioner-oriented products, NCES could partner with NCEE to provide resources or search terms from the What Works Clearinghouse that may be relevant to the topic at hand.³⁹ Such collaboration has the added benefit of cross-promoting IES products and tools to boost both centers. Table 4-1 provides ideas for improving current products and services for various stakeholders’ needs.

Regarding timeliness, the panel recommends that NCES consider three things: stakeholders, operations, and review processes. NCES is encouraged to engage with practitioners, policy makers, and other stakeholders on their needs, as expectations for timeliness vary by stakeholder and have changed over time. The panel urges NCES to revisit the timeliness of each of its existing products, to determine if the degree of timeliness suits the product’s target audience.

To further improve timeliness, NCES could assess its data collection and processing procedures, which could include exploring acceptable quality tradeoffs to decrease lag time. The panel understands the challenges inherent in balancing high response rates, completeness, quality-assurance processes, and timely dissemination. NCES should consider whether producing quick, crude estimates (appropriately caveated) could sometimes prove useful and informative. These estimates could be followed by refined estimates providing necessary, high-quality data. At times, NCES produces multiple versions of its data (as with the 2015–2016 National Teacher and Principal Survey),⁴⁰ and this could be done more often—estimates can often be revised with minimal ill effects. When BEA produces the Gross Domestic Product, a leading economic indicator, it releases this indicator and then later revises it. Similarly, BLS produces employment statistics every month that are frequently revised after publication. The COVID-19 pandemic reinforced NCES’s ability to perform fast, high-quality work, such as the School Pulse Panel (NCES, 2021l). The panel urges NCES to assess its collections to determine which purposes can tolerate, with transparency, a slightly reduced level of data quality with the potential for revision.

Another way to improve timeliness involves staggered data releases, which prioritize the release of high-priority content that is most useful when timely, followed by more detailed information as it becomes available. By

___________________

³⁹ See: https://ies.ed.gov/ncee/wwc/ [March 2022].

⁴⁰ NCES response to question from the panel, p. 58.

TABLE 4-1 Current NCES Products and Services, Stakeholders, and Known and/or Anticipated Needs

using this approach, NCES would not be trading off data quality, but rather strategically prioritizing releases for greater product impact. NCES could even use staggered or first-release data to its benefit. For example, states usually release a first-look dataset that is left open to edit by districts. The state of Ohio, like most states with unit record systems, regularly collects both K–12 and higher education data from LEAs and institutions of higher education but allows for data corrections to ensure accuracy and data quality. NCES may consider scenarios in which it is advantageous to involve SEAs, LEAs, and other data providers in data-quality processes, iterating between first releases and revisions.

Additionally, NCES, in collaboration with IES, is advised to conduct a top-to-bottom review of both NCES’s and IES Standards Review Office’s⁴¹ quality-assurance processes, to revise the review procedure for products requiring more timeliness. NCES is encouraged to consider ways to embed transparency about quality tradeoffs (see Appendix E for detailed information on NCES and IES review processes). NCES has a more extensive review process than any other center in IES or many other statistical agencies (IES Standards and Review Office, 2006).⁴² In principle, both NCES and IES would benefit from fewer layers of review, to issue timely, relevant, rigorous analyses and products that inform decision making.

Together, NCES and IES should consider the roles and relevance of each reviewer, to determine whether each is necessary and provides meaningful feedback, or whether certain reviewers could be limited to particular aspects of review. For instance, the commissioner could mainly conduct sensitivity reviews instead of full reviews.⁴³ As another example, after a product has been approved by a branch chief and an associate commissioner, it undergoes methodological review by four to five people—a senior mathematical statistician leads the review, a research associate and two research assistants usually conduct technical review, and the chief statistician signs off (IES Standards and Review Office, 2006). In comparison, methodological reviews for the U.S. Census Bureau’s Social, Economic, and Housing Statistics Division products are conducted by one mathematical statistician in parallel with the branch chief’s review.

Products need various levels of reviewer expertise. For instance, a conference presentation or working paper with the author’s disclaimer may not need the same level of review as an official report with a press release. Experimental products that are transparent about their quality and methods may not need a high level of review. Twenty-one percent of NCES’s products underwent external review in the last 3 years.⁴⁴ Considering that the median time to issue an IES disposition memo is 1 week for internal reviews and 6 weeks for external reviews (Table E-3), IES (and thus NCES) may want to consider when and if external reviewers are needed (see exemptions

___________________

⁴¹ The Education Sciences Reform Act of 2002 (20 U.S. Code § 9501 et seq.) requires IES to have a peer review process. See: https://ies.ed.gov/director/sro/, and https://ies.ed.gov/director/sro/ppt/Scientific_Peer_Review.pptx [March 2022].

⁴² For example, BEA released updated guidelines in 2019 that reduced the number of individuals required to approve most products. Also, the U.S. Census Bureau’s Social, Economic and Housing Statistics Division considers it acceptable to use peer reviewers outside of the author’s branch but internal to the agency.

⁴³ Currently, “many products also receive a full review at the commissioner’s level.” NCES response to question from the panel, p. 35.

⁴⁴ External review is conducted by experts outside of IES from “across a wide range of substantive and methodological fields.” IES document provided to the panel, “Peer Review of IES Reports,” p. 1.

from peer review in U.S. OMB [2004]). In some statistical agencies, such as the U.S. Census Bureau, peer reviewers can come from other units within the agency.

In the panel’s opinion, NCES and IES would benefit from a careful consideration of ways to match the review process to the product, focusing on the desired outcomes of review (e.g., timely, relevant, high quality). Tradeoffs to be considered include not only the quality and timeliness of the product, but also effects on staff. Long reviews frequently depress staff morale, inhibit a culture of innovation, and prevent or delay engagement with stakeholders.

This page intentionally left blank.