Applications of an Analytic Framework on Using Public Opinion Data for Solving Intelligence Problems Proceedings of a Workshop (2022) / Chapter Skim
Currently Skimming:
2 Hypothetical Applications of the Analytic Framework
2 Hypothetical Applications of the Analytic Framework
Pages 21-46
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 21... ...
Evaluation of Data Sources Elizabeth Zechmeister (co-lead expert contributor for the Analytic Framework, and Cornelius Vanderbilt Professor of Political Science and director of the Latin American Public Opinion Project at Vanderbilt University) observed that because the research question was assigned to the analyst and data are available, the first step is to narrow the scope, with consideration for the quality and potential use of the data.
|
From page 22... ...
a well-executed mobile phone survey conducted by the U.S. Department of State 3 months ago; and (3)
|
From page 23... ...
Thus, it is important for Rachel to know that she could test quickly to determine whether the poll is gaining traction in the social media sphere and whether the perception matters. Face-to-Face Survey Eveland posited that to evaluate the data from the face-to-face survey, Rachel would consult "Drawing Inferences from Public Opinion Surveys: Insights for Intelligence Reports" as well as "Ascertaining True Attitudes in Survey Research," by Kanisha Bond (assistant professor of political science at Binghamton University, State University of New York)
|
From page 24... ...
Perhaps more important, since Rachel is a novice, he suggested that she write down what she thinks the data will reveal before looking at them to prevent being biased; if the results do not align with her hypotheses, it might be worthwhile for her to consult with a senior analyst to understand whether the flaw is in the hypothesis or the data. Mobile Phone Survey Page commented that if the mobile phone survey is divided by e thnic group, it is likely both credible and sound; however, considering the commentary on survey language selection from Michele Gelfand (expert contributor for the Analytic Framework; John H
|
From page 25... ...
Referencing both "Drawing Inferences from Public Opinion Surveys: Insights for Intelligence Reports" and "Ascertaining True Attitudes in Survey Research," Eveland pointed out that mobile phone surveys allow people to avoid answering certain questions more easily than face-to-face surveys. He encouraged Rachel to pay close attention to the "do not know" or "refuse" answers in both the face-to-face and mobile phone surveys as a point of comparison to understand whether the survey mode influenced the results.
|
From page 26... ...
Pasek observed that the face-to-face and mobile phone surveys might be very different in the context of varied ethnic groups. For example, if the mobile phone survey is done well and mobile phone coverage is good, a clear picture of the e thnic groups could emerge.
|
From page 27... ...
Pasek advocated for analysis within subgroups in the face-toface survey and the mobile phone survey: even if the mobile phone data are lacking entire populations, if the populations that are well represented in both surveys have similar estimates, they could be used as a basis for assert ing relative similarity. Page added that because some parties have ethnic affiliations, it is important for Rachel first to determine whether the prime minister is from a party that has historic ethnic affiliations.
|
From page 28... ...
IC analyst task Tanya is tasked with developing a plan for how the State Department can understand public opinion within 1 month. Data sources available Tanya has access to no existing data sources.
|
From page 29... ...
Instead, Tanya's next step could be to scrape social media data to understand the extent of existing biases in the population. Reflecting on other possible strategies to gather new data, Kennedy explained that if Tanya decides to conduct a survey, a phone poll or online data collection might be the best approaches.
|
From page 30... ...
An analyst asked how to assess whether the results of an experiment are reliable in an environment where no previous data are available for comparison. Mueller explained that in a list experiment, for example, people would not be asked direct questions about their attitudes toward the new government.
|
From page 31... ...
In a context where no previous survey data exist, it is important to evaluate social media environments and their data as well as how value could be extracted from those data. Even though no data source is perfect, she continued, Tanya should still ensure that the methods she selects will extract worthwhile information.
|
From page 32... ...
Are data from the frame available? He directed analysts to the American Association for Public Opinion Research's (AAPOR's)
|
From page 33... ...
He suggested that Tanya use the month to track those data down, study them, and consult with experts who are located in the United States and could provide insight to develop a plan for understanding public opinion in the country of interest. Although this scenario's discussion introduced several best practices for data collection, he continued, it did not consider the constraints for what analysts have to produce afterward.
|
From page 34... ...
IC analyst task Luis is tasked with describing public opinion in the region, with a focus on recommending countries where popular discontent with the regime might emerge in the next year. Data sources available 20 surveys (5 mobile phone, 5 social media nonprobability Web surveys, 10 face-to-face)
|
From page 35... ...
Lau explained that although the mobile phone surveys might be more credible than the social media surveys, a lack of documentation about weighting suggests that the mobile phone surveys might not be nationally
|
From page 36... ...
36 FIGURE 2-1 A rating tool for survey quality.
|
From page 37... ...
Therefore, Luis would have to assign bias to these surveys. Lau reiterated that the Analytic Framework is not meant to be directive but rather to spark discussion: how does such a rating system work on a day-to-day basis for intelligence analysts?
|
From page 38... ...
He contemplated how these considerations would influence the IC's practice, and he encouraged workshop participants to identify related gaps in the Analytic Framework and to discuss how they could be addressed. Discussion A representative from the IC observed that an early career intelligence analyst could be challenged and overwhelmed by an assignment to conduct regional analysis and would likely take several months to complete it.
|
From page 39... ...
When comparing the mobile phone surveys to the face-to-face surveys, the analyst wondered how to determine what is closer to true attitude if a systematic difference in attitude emerges. Lau highlighted sample coverage as one key factor (i.e., mobile phone surveys exclude people without mobile phones)
|
From page 40... ...
Individual plots can be generated for each country assuming that the differences between the countries are relatively stable, with the exception of any added discontinuities -- the effects of those discontinuities then become visible. Pasek mentioned that although an early career intelligence analyst like Luis would not be doing this type of work, the toolkit to do so exists.
|
From page 41... ...
The analyst is expected to present the first deliverable in 2 weeks, but the project will likely continue for 1 year. Data sources available ~5 years of biannual mobile phone surveys in 5 of the 10 nations in the region and some yearly aggregate international polls, but no data exist for the remaining nations.
|
From page 42... ...
Another analyst observed that a translator might be needed to conduct these assessments because the files contain the original questionnaire in Russian. Pasek mentioned that although most in the public opinion community know Langer's reputation well, if that was not the case, it would be important for the intelligence analyst to conduct research into his credibility.
|
From page 43... ...
provide context about the stability of public opinion -- that is, how does the 58 percent approval rating cited in the Washington Post article compare to the approval ratings in those instances? Following the guidance of the Analytic Framework, Eveland reiterated the importance of understanding the nature of the survey provider and how the public views that provider, which could also affect the results of the poll.
|
From page 44... ...
Lau pointed out that the concept of "support" is ambiguous and interpretable among respondents, and Amaya suspected significant nonresponse biases in the poll results. Kennedy posited that addi tional opposition could be "hiding," perhaps in the nonresponses to the poll, and it would be helpful for an intelligence analyst to better understand the cooperation rate of the public.
|
From page 45... ...
In the case of an attack within a particular country or a natural disaster, it relates to a temporary change in public opinion in response to the significant event that deviates from the normal narrative and recedes within weeks or months. Pasek remarked that while "rally around the flag" is useful in some cases, intelligence analysts likely rely on other phenomena to explain public opinion that do not appear in the academic literature.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.