Practice 6

Strong Internal and External Evaluation Processes for an Agency’s Statistical Programs

STATISTICAL agencies should have processes in place to support regular evaluations of their major statistical programs and their overall portfolio of programs. Reviews of major data collection programs and their components should consider how to produce relevant, accurate, and timely data in the most cost-effective manner possible. Reviews of an agency’s portfolio should consider ways to reduce duplication, fill gaps, and adjust priorities so that the overall portfolio is as relevant as possible to the information needs of policy makers and the public (e.g., see NASEM, 2018c, 2020a and NRC, 2009a). Such evaluations should include internal reviews by staff and external reviews by independent groups.

Agencies should seek outside reviews not only of specific statistical programs but also of program priorities and quality practices across their entire portfolio. They should also consider ways to improve program cost-effectiveness by combining data from multiple sources, particularly because fewer people and organizations are responding to surveys than in the past. It is increasingly urgent to determine whether there are alternative data sources to surveys that also offer better quality (e.g., see NASEM, 2017b, 2017c, 2017d, 2018a, and Practice 5).

Statistical agencies that fully follow practices related to an active research program (Practice 5), openness (Practice 10), dissemination of statistical data products (Practice 9), and commitment to quality and professional standards (Practice 3) will likely be in a good position to make continuous assessments of and improvements in the relevance, quality, and efficiency of their data collection systems. Yet even the

best-functioning agencies will benefit from an explicit program of internal and independent external evaluations to obtain fresh perspectives.

Evaluating Quality, Relevance, Efficiency

Evaluation of data quality for any kind of data collection program begins with regular monitoring of quality indicators that are readily available to users. Agencies should use broad quality frameworks (see Practice 3 and Appendix C) and assess the costs and benefits of using alternative data sources (see Practice 5 and NASEM, 2017b, 2017d). These evaluations should be undertaken periodically and the results made public (see Practice 10 and NRC, 2007b).

When it is disruptive to implement improvements on a continuing basis, a common practice is to bundle changes to implement several at the same time. For example, classifications such as the North American Industry Classification System (NAICS) are updated every 5 years, and agencies may implement other changes at the same time as this. Agencies should ensure that the intervals between innovations do not become so long that data collection programs deteriorate in quality, relevance, and efficiency. Regular, well designed program evaluations, with adequate budget support, are key to ensuring that data collection programs do not deteriorate. Having a set schedule for research and development efforts will enable data collection managers to ensure that the quality and usefulness of their data are maintained and help prevent locking in less optimal procedures.

As part of ongoing evaluation, the relevance of an agency’s data collection programs and products needs to be continually assessed. The question of relevance is whether the agency is “doing the right thing,” in contrast to whether the agency is “doing things right.” Relevance should be assessed not only for particular programs or closely related sets of programs, but also for an agency’s complete portfolio in order to make the best choices among program priorities given the available resources (see Practice 1).

Engaging and consulting with stakeholders—through such means as regular meetings, workshops, conferences, and other activities—is important to ensuring relevance (see Practice 9). Including other federal statistical colleagues in this communication, both as users and as collaborators, can be valuable (see Practice 7).

Statistical agencies commonly find it difficult to discontinue a particular data series, even when it has largely outlived its usefulness relative to other series, because of objections by users who have become accustomed to it. In the face of limited resources, however, discontinuing a series is preferable to across-the-board cuts in all programs, which would reduce the accuracy and usefulness of both the more relevant and less relevant data series. Regular internal and external reviews and a documented priority-setting process or framework can help an agency not only in reassessing its priorities but also in developing the justification and support for changes to its portfolio.

Finally, statistical agencies should review their statistical programs for efficiency and cost-effectiveness.²⁴ Federal statistics as a public good represent a legitimate draw on public resources, and statistical agencies in turn are properly called on to analyze the costs of their programs on a continuing basis to ensure the best return possible on tax dollars. For this purpose, statistical agencies should develop complete, informative models for evaluating the costs of current procedures and of possible alternatives and follow best practice in the design of statistical production processes.²⁵

Types of Reviews

Regular statistical program reviews should include a mixture of internal and external evaluation. Agency staff should set goals and timetables for internal evaluations that involve informed staff outside the program under review. Independent external evaluations should also be conducted on a regular basis. The frequency of these external evaluations should depend on the importance of the data, how quickly the phenomena being measured change, and how quickly respondent behavior and data collection technology may adversely affect a program change.

External reviews can take many forms. They may include recommendations from advisory committees that meet at regular intervals (typically, every 6 months). However, advisory committees should never be the sole

___________________

²⁴ “Efficiency” is generally defined as an ability to avoid waste (of materials, energy, money, time) in producing a specified output. “Cost-effectiveness” connotes a broader, comparative look at inputs and outputs to assess the most advantageous combination. (“Cost-benefit” analysis attempts to add monetary values to outputs.) In the context of federal statistical programs, cost-effectiveness analysis would assess the costs of conducting a program for different combinations of desired characteristics, such as improved accuracy or timeliness and reduced burden on respondents.

²⁵ See Generic Statistical Business Process Model of the United Nations Economic Commission for Europe in Appendix C

source of outside review, because the members of such committees rarely have the opportunity to become deeply familiar with agency programs. External reviews can also take the form of a special committee or panel established by a relevant professional association, such as the American Statistical Association, or by some other recognized group, such as the National Institute of Statistical Sciences (also see NRC, 2009a).