National Academies Press: OpenBook

Principles and Practices for a Federal Statistical Agency: Seventh Edition (2021)

Chapter: Practice 6: Strong Internal and External Evaluation Processes for an Agency's Statistical Programs

« Previous: Practice 5: An Active Research Program
Suggested Citation:"Practice 6: Strong Internal and External Evaluation Processes for an Agency's Statistical Programs." National Academies of Sciences, Engineering, and Medicine. 2021. Principles and Practices for a Federal Statistical Agency: Seventh Edition. Washington, DC: The National Academies Press. doi: 10.17226/25885.
×

Practice 6

Strong Internal and External Evaluation Processes for an Agency’s Statistical Programs

STATISTICAL agencies should have processes in place to support regular evaluations of their major statistical programs and their overall portfolio of programs. Reviews of major data collection programs and their components should consider how to produce relevant, accurate, and timely data in the most cost-effective manner possible. Reviews of an agency’s portfolio should consider ways to reduce duplication, fill gaps, and adjust priorities so that the overall portfolio is as relevant as possible to the information needs of policy makers and the public (e.g., see NASEM, 2018c, 2020a and NRC, 2009a). Such evaluations should include internal reviews by staff and external reviews by independent groups.

Agencies should seek outside reviews not only of specific statistical programs but also of program priorities and quality practices across their entire portfolio. They should also consider ways to improve program cost-effectiveness by combining data from multiple sources, particularly because fewer people and organizations are responding to surveys than in the past. It is increasingly urgent to determine whether there are alternative data sources to surveys that also offer better quality (e.g., see NASEM, 2017b, 2017c, 2017d, 2018a, and Practice 5).

Statistical agencies that fully follow practices related to an active research program (Practice 5), openness (Practice 10), dissemination of statistical data products (Practice 9), and commitment to quality and professional standards (Practice 3) will likely be in a good position to make continuous assessments of and improvements in the relevance, quality, and efficiency of their data collection systems. Yet even the

Suggested Citation:"Practice 6: Strong Internal and External Evaluation Processes for an Agency's Statistical Programs." National Academies of Sciences, Engineering, and Medicine. 2021. Principles and Practices for a Federal Statistical Agency: Seventh Edition. Washington, DC: The National Academies Press. doi: 10.17226/25885.
×

best-functioning agencies will benefit from an explicit program of internal and independent external evaluations to obtain fresh perspectives.

Evaluating Quality, Relevance, Efficiency

Evaluation of data quality for any kind of data collection program begins with regular monitoring of quality indicators that are readily available to users. Agencies should use broad quality frameworks (see Practice 3 and Appendix C) and assess the costs and benefits of using alternative data sources (see Practice 5 and NASEM, 2017b, 2017d). These evaluations should be undertaken periodically and the results made public (see Practice 10 and NRC, 2007b).

When it is disruptive to implement improvements on a continuing basis, a common practice is to bundle changes to implement several at the same time. For example, classifications such as the North American Industry Classification System (NAICS) are updated every 5 years, and agencies may implement other changes at the same time as this. Agencies should ensure that the intervals between innovations do not become so long that data collection programs deteriorate in quality, relevance, and efficiency. Regular, well designed program evaluations, with adequate budget support, are key to ensuring that data collection programs do not deteriorate. Having a set schedule for research and development efforts will enable data collection managers to ensure that the quality and usefulness of their data are maintained and help prevent locking in less optimal procedures.

As part of ongoing evaluation, the relevance of an agency’s data collection programs and products needs to be continually assessed. The question of relevance is whether the agency is “doing the right thing,” in contrast to whether the agency is “doing things right.” Relevance should be assessed not only for particular programs or closely related sets of programs, but also for an agency’s complete portfolio in order to make the best choices among program priorities given the available resources (see Practice 1).

Engaging and consulting with stakeholders—through such means as regular meetings, workshops, conferences, and other activities—is important to ensuring relevance (see Practice 9). Including other federal statistical colleagues in this communication, both as users and as collaborators, can be valuable (see Practice 7).

Suggested Citation:"Practice 6: Strong Internal and External Evaluation Processes for an Agency's Statistical Programs." National Academies of Sciences, Engineering, and Medicine. 2021. Principles and Practices for a Federal Statistical Agency: Seventh Edition. Washington, DC: The National Academies Press. doi: 10.17226/25885.
×

Statistical agencies commonly find it difficult to discontinue a particular data series, even when it has largely outlived its usefulness relative to other series, because of objections by users who have become accustomed to it. In the face of limited resources, however, discontinuing a series is preferable to across-the-board cuts in all programs, which would reduce the accuracy and usefulness of both the more relevant and less relevant data series. Regular internal and external reviews and a documented priority-setting process or framework can help an agency not only in reassessing its priorities but also in developing the justification and support for changes to its portfolio.

Finally, statistical agencies should review their statistical programs for efficiency and cost-effectiveness.24 Federal statistics as a public good represent a legitimate draw on public resources, and statistical agencies in turn are properly called on to analyze the costs of their programs on a continuing basis to ensure the best return possible on tax dollars. For this purpose, statistical agencies should develop complete, informative models for evaluating the costs of current procedures and of possible alternatives and follow best practice in the design of statistical production processes.25

Types of Reviews

Regular statistical program reviews should include a mixture of internal and external evaluation. Agency staff should set goals and timetables for internal evaluations that involve informed staff outside the program under review. Independent external evaluations should also be conducted on a regular basis. The frequency of these external evaluations should depend on the importance of the data, how quickly the phenomena being measured change, and how quickly respondent behavior and data collection technology may adversely affect a program change.

External reviews can take many forms. They may include recommendations from advisory committees that meet at regular intervals (typically, every 6 months). However, advisory committees should never be the sole

___________________

24 “Efficiency” is generally defined as an ability to avoid waste (of materials, energy, money, time) in producing a specified output. “Cost-effectiveness” connotes a broader, comparative look at inputs and outputs to assess the most advantageous combination. (“Cost-benefit” analysis attempts to add monetary values to outputs.) In the context of federal statistical programs, cost-effectiveness analysis would assess the costs of conducting a program for different combinations of desired characteristics, such as improved accuracy or timeliness and reduced burden on respondents.

25 See Generic Statistical Business Process Model of the United Nations Economic Commission for Europe in Appendix C

Suggested Citation:"Practice 6: Strong Internal and External Evaluation Processes for an Agency's Statistical Programs." National Academies of Sciences, Engineering, and Medicine. 2021. Principles and Practices for a Federal Statistical Agency: Seventh Edition. Washington, DC: The National Academies Press. doi: 10.17226/25885.
×

source of outside review, because the members of such committees rarely have the opportunity to become deeply familiar with agency programs. External reviews can also take the form of a special committee or panel established by a relevant professional association, such as the American Statistical Association, or by some other recognized group, such as the National Institute of Statistical Sciences (also see NRC, 2009a).

Suggested Citation:"Practice 6: Strong Internal and External Evaluation Processes for an Agency's Statistical Programs." National Academies of Sciences, Engineering, and Medicine. 2021. Principles and Practices for a Federal Statistical Agency: Seventh Edition. Washington, DC: The National Academies Press. doi: 10.17226/25885.
×
Page 75
Suggested Citation:"Practice 6: Strong Internal and External Evaluation Processes for an Agency's Statistical Programs." National Academies of Sciences, Engineering, and Medicine. 2021. Principles and Practices for a Federal Statistical Agency: Seventh Edition. Washington, DC: The National Academies Press. doi: 10.17226/25885.
×
Page 76
Suggested Citation:"Practice 6: Strong Internal and External Evaluation Processes for an Agency's Statistical Programs." National Academies of Sciences, Engineering, and Medicine. 2021. Principles and Practices for a Federal Statistical Agency: Seventh Edition. Washington, DC: The National Academies Press. doi: 10.17226/25885.
×
Page 77
Suggested Citation:"Practice 6: Strong Internal and External Evaluation Processes for an Agency's Statistical Programs." National Academies of Sciences, Engineering, and Medicine. 2021. Principles and Practices for a Federal Statistical Agency: Seventh Edition. Washington, DC: The National Academies Press. doi: 10.17226/25885.
×
Page 78
Next: Practice 7: Coordination and Collaboration with Other Statistical Agencies »
Principles and Practices for a Federal Statistical Agency: Seventh Edition Get This Book
×
Buy Paperback | $60.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Government statistics are widely used to inform decisions by policymakers, program administrators, businesses and other organizations as well as households and the general public. Principles and Practices for a Federal Statistical Agency, Seventh Edition will assist statistical agencies and units, as well as other agencies engaged in statistical activities, to carry out their responsibilities to provide accurate, timely, relevant, and objective information for public and policy use. This report will also inform legislative and executive branch decision makers, data users, and others about the characteristics of statistical agencies that enable them to serve the public good.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!