Measurement
and Analysis of
Public Opinion
AN ANALYTIC FRAMEWORK
Dylan Rebstock, Tina M. Winters, and Emma Fine, Editors
Board on Behavioral, Cognitive, and Sensory Sciences
Division of Behavioral and Social Sciences and Education
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This activity was supported by contracts between the National Academy of Sciences and the Intelligence Community. Support for the work of the Board on Behavioral, Cognitive, and Sensory Sciences is provided primarily by a grant from the National Science Foundation (Award No. BCS-1729167). Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-27340-4
International Standard Book Number-10: 0-309-27340-4
Digital Object Identifier: https://doi.org/10.17226/26390
Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2022 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2022. Measurement and Analysis of Public Opinion: An Analytic Framework. Washington, DC: The National Academies Press. https://doi.org/10.17226/26390.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.
This page intentionally left blank.
EXPERT CONTRIBUTORS
ELIZABETH ZECHMEISTER (Co-Moderator), Vanderbilt University
CHARLES LAU (Co-Moderator), RTI International
JAMES N. DRUCKMAN, Northwestern University
WILLIAM P. EVELAND, The Ohio State University
MICHELE GELFAND, Stanford University
D. SUNSHINE HILLYGUS, Duke University
MICHAEL HOUT, New York University
COURTNEY KENNEDY, Pew Research Center
ZACHARIAH MAMPILLY, City University of New York
LISA MUELLER, Macalester College
DIANA C. MUTZ, University of Pennsylvania
SCOTT E. PAGE, University of Michigan
Staff
DYLAN REBSTOCK, Project Director
TINA M. WINTERS, Associate Program Officer
EMMA FINE, Associate Program Officer
JACQUELINE COLE, Senior Program Assistant
BARBARA WANCHISEN, Senior Advisor for Behavioral Sciences
SAMANTHA CHAO, Associate Executive Director
This page intentionally left blank.
Acknowledgment of Reviewers
This analytic framework was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each publication as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the statement of task. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We thank the following individuals for their review of this report:
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the key themes of this analytic framework nor did they see the final draft before its release. We also thank Jonathan Moreno, Department of Medical Ethics and Health Policy, University of Pennsylvania, who served as review coordinator. He
was responsible for making certain that an independent examination of this analytic framework was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authors and the National Academies.
Additional Acknowledgments
Many individuals contributed their time and expertise to this project and preparation of the framework. The project was initiated by the US Government (USG). USG staff readily assisted the project through their active participation at meetings and responses to a large number of requests for background materials about the daily tasks of an analyst and real-world problems they could plausibly encounter.
Special appreciation is extended to the members of the expert contributors who are largely responsible for shaping the project’s course:
Their focus on providing actionable support to the sponsor never faltered, and their collective dedication and positive attitude were impressive, especially considering the collaborative limits imposed on the project by the
pandemic. Extra thanks are extended to Elizabeth Zechmeister and Charles Lau, who in their capacity as co-moderators, provided excellent guidance that kept the project moving forward while never sacrificing the main objectives of providing maximum utility to the end-user intelligence analyst.
Appreciation is also extended to the authors of the foundational papers included in this framework:
They boldly signed up to help make a difference, possessed a seemingly inexhaustible supply of patience, and provided the project’s core substantive material. Additional thanks go to consultant Rona Briere, who took on the arduous task of writing the synthesis layer, condensing four incredibly detailed and technical documents into a single summary. Rona was supported in her work by Allison Boman, whose careful notetaking during discussions of the content of the synthesis informed its development, and who provided invaluable assistance in preparing the overview graphic in the synthesis and formatting and proofreading the text. This framework was edited by Kevin Morrow, whose careful attention to detail strengthened the final product. We thank all of these individuals for their flexibility in making needed changes to the documents between the prepublication copy and final publication.
The core project team at the National Academies of Sciences, Engineering, and Medicine included Dylan Rebstock, project director, who had the honor of leading a superlative team; Tina Winters, associate program officer, whose foresight and ubiquitous good judgment helped avoid mistakes and keep the project on track; Emma Fine, associate program officer with the Health and Medicine Division, who, among other things, served as an invaluable bridge by advocating best practices from the previous Framework project; and Jacqueline Cole, senior program assistant, whose relentless cheerfulness and attention to detail ensured nothing slipped through the cracks. The work they did was often behind the scenes, but it was the sine qua non to the project’s success.
Several other National Academies staff members made important contributions to this project. Samantha Chao, associate executive director, National Research Council Programs; Barbara Wanchisen, senior advisor for behavioral sciences, Division of Behavioral and Social Sciences and Education; and Adrienne Stith Butler, former director of the Board on Behavioral, Cognitive, and Sensory Sciences, each provided critical guidance
and support to the core project team. Kirsten Sampson Snyder, director of reports, managed the review process; Holly Sten, design coordinator, shepherded the graphic layer through production; Elizabeth Molyé, senior contract manager, provided thoughtful guidance on the contracting process; and Patrick Burke, senior finance business partner, relieved a lot of stress by managing the finances so adroitly.
This page intentionally left blank.
Acronyms
| AAPOR | American Association for Public Opinion Research |
| ACASI | Audio Computer Assisted Self Interview |
| AI | Artificial Intelligence |
| API | Application Programming Interface |
| ARIMA | Autoregressive Integrated Moving Average |
| CASM | Cognitive Aspects of Survey Methodology |
| CATI | Computer-assisted Telephone Interviewing |
| CDS | Computerized Delivery Sequence |
| CNN | Cable News Network |
| COVID-19 | Coronavirus Disease 2019 |
| ESOMAR | European Society for Opinion and Marketing Research (formerly) |
| ESS | European Social Survey |
| FAQ | Frequently Asked Questions |
| FIPS | Federal Information Processing Standards |
| GP | General Population |
| IAT | Implicit Association Test |
| IC | Intelligence Community |
| IRB | Institutional Review Board |
| IVR | Interactive Voice Response |
| LLC | Limited Liability Company |
| MAGA | Make America Great Again |
| MRP | Multilevel Regression and Post-Stratification |
| NATO | North Atlantic Treaty Organization |
| NORC | National Opinion Research Center |
| PHI | Personal Health Information |
| PII | Personal Identifiable Information |
| PPS | Probability Proportional to Size |
| RDD | Random Digit-Dialing |
| RDE | Random Device Engagement |
| RDS | Respondent Driven Sampling |
| RICS | Redirected Inbound Call Sampling |
| SAIPE | Small Area Income and Poverty Estimates |
| SMS | Short Message Service |
| SRS | Simple Random Sampling |
| TSE | Total Survey Error |
| USAID | U.S. Agency for International Development |
| ZTRAX | Zillow Transaction and Assessment Dataset |
Contents
Dylan Rebstock, Tina M. Winters, and Emma Fine
LAYER 2 – SYNTHESIS: USING PUBLIC OPINION RESEARCH TO ANSWER AN INTELLIGENCE QUESTION
LAYER 3 – FOUNDATIONAL WHITE PAPERS
3A: Drawing Inferences from Public Opinion Surveys: Insights for Intelligence Reports
3C: Ascertaining True Attitudes in Survey Research
