National Academies Press: OpenBook

Learning from the Science of Cognition and Perception for Decision Making: Proceedings of a Workshop (2018)

Chapter: 2 Data and Analysis in the Intelligence Community

« Previous: 1 Introduction
Suggested Citation:"2 Data and Analysis in the Intelligence Community." National Academies of Sciences, Engineering, and Medicine. 2018. Learning from the Science of Cognition and Perception for Decision Making: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25118.
×

2

Data and Analysis in the Intelligence Community

Thomas Fingar, Stanford University, set the stage for the day’s discussions by asserting that the role of the Intelligence Community (IC) and the reports it produces is to serve policy making—to help decision makers better understand the world and anticipate trajectories of specific events and developments. In his remarks, he highlighted several important aspects of the IC’s role, as well as some of the difficulties it faces in managing data and answering specific questions for policy makers.

Fingar pointed out that the IC collects an extremely large amount of data—both publicly available and clandestinely acquired—each day. The main challenge, he stressed, is marshaling the right data at the right time in order to address a particular set of concerns. He explained that meeting this challenge is difficult because not only are the data of uneven quality and uncertain relevance, but they may also contain deliberate misinformation, which must somehow be separated from the factual information.

The questions the IC faces are difficult and have serious consequences, Fingar continued, and a sufficient amount of the right data for full insight into specific questions is often lacking. He added that the IC is often working within a tight decision timeline.

Fingar then provided examples of questions asked of the IC 10 years ago. These examples included whether the events in Iraq could be considered a civil war and whether Iran could be persuaded to give up or freeze its weapons program. Fingar pointed out that analysts’ understanding of the political environment and surrounding context is often very important to answering the questions asked of the IC, noting that such knowledge is

Suggested Citation:"2 Data and Analysis in the Intelligence Community." National Academies of Sciences, Engineering, and Medicine. 2018. Learning from the Science of Cognition and Perception for Decision Making: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25118.
×

helpful in determining whether findings will be controversial and in raising other questions that should be considered.

Fingar went on to explain that projections made by the IC should help policy makers either maintain or intervene in the trajectory of events. To illustrate this point, he provided the example of a group of clerics in northern Nigeria who were convinced that inoculations against polio were a Christian plot to sterilize Muslims, and therefore blocked immunizations in the one location in the world where polio had not been eradicated. To determine how to alter this trajectory of events, he said, the IC had to ask such questions as, “Who can persuade these people to change their minds?”, “Are there Islamic companies that manufacture vaccines?”, and “Who might the clerics listen to?” He emphasized that this type of situation relies on the ability of the IC both to deeply understand issues of culture and to reach out to others who can help them intervene in the events.

In addition to answering questions directly posed to it, Fingar continued, the IC serves a warning function. He elaborated by saying, “It’s not just looking for needles in a haystack; it’s discovering needles that you didn’t know were there, [ones] that might have significance for American interests, from national security to the health and safety of the American public.” Thus, he stressed, analysts need to be trained specifically to discover unanticipated information of possible significance.

Fingar acknowledged that the focus of intelligence analysis is on making sense of available data to determine their relevance and communicate useful information derived from the data such that it will be fully understood by policy makers. He then cited multiple aspects of accurate communication to decision makers: being clear about what is known, what is not known, what assumptions were used to bridge the information gaps, why some assumptions were used over others, and how changing an assumption or weighing certain evidence differently might affect the interpretation of the data. He also stressed the importance of communicating with clarity instead of arrogance.

An additional challenge faced by the IC, noted Fingar, is that because the community relies almost entirely on training provided externally, in universities, new techniques and methodologies from the university setting come to the IC in the form of new recruits. He pointed out that the more senior people in the IC therefore must adapt to utilize the skill sets brought by these younger people, so the community might do well to consider formal mechanisms of “teaching up” from younger to more senior analysts.

To conclude, Fingar reiterated that the primary responsibility of the IC is to make sense of developments and information, and to communicate that information to policy makers accurately and effectively. He stressed that it is critical for the IC to provide timely information specific to the problems at hand and for the information to be communicated in such a

Suggested Citation:"2 Data and Analysis in the Intelligence Community." National Academies of Sciences, Engineering, and Medicine. 2018. Learning from the Science of Cognition and Perception for Decision Making: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25118.
×

way that the recipients fully understand what they are being told. He stated, “Being right is nice. Being useful and being perceived as useful is absolutely essential. Without demonstrated utility, you don’t get the confidence of the people you work with. Without the confidence and trust that you’ve done the work correctly, you’re just one more opinion.” He closed by pointing out that although every lobbyist, foreign official, and member of the public has an opinion, the IC is the only contributor to the policy-making process institutionally identified to provide objective insight.

Following Fingar’s presentation, workshop participants were invited to ask questions. Peter Pirolli, Institute for Human & Machine Cognition, asked Fingar about the possibilities for having artificial intelligence mediate between the flood of data and the analyst. Fingar replied that the IC should use machines to do the things they do well (such as the initial sorting of vast amounts of data, pattern recognition, identification of trends over time, and flagging of new information) to free up human beings for the things only they can do.

In response to a question regarding the concept of “being right” versus “being useful,” Fingar provided an example from the 1990s in which the overriding analytic judgment suggested that North Korea would not give up its weapons program. However, he continued, the information was quite ambiguous, and some analysts in the IC did not believe the prediction. Those analysts were persuasive enough that the Clinton administration took a politically risky course that resulted in the freezing of North Korea’s nuclear weapons program. Fingar stressed that the work of the IC is estimative, not predictive, noting that many uncertainties, time constraints, and conflicting data prevent the IC from being right all the time. In his estimation, the IC is right 80 to 85 percent of the time. Sometimes, he added, the most useful information for policy makers is clarification of the level of uncertainty in a given situation.

Another workshop participant, recognizing that data collection and analysis are separate functions within the IC, asked Fingar whether he sees a need for greater emphasis on fast feedback between those engaged in these two functions, particularly when new, unanticipated problems arise. Fingar replied that analysis should drive collection, with analysts reporting to those collecting data the questions being considered, the types of data that would be useful to answer those questions, and where the data might be found. However, he acknowledged, this process works better in theory than in practice because budgets in the IC are overwhelmingly on the data collection side, leaving the analytic community with little leverage. In addition, he noted, technologies for collecting data are outpacing those for useful analysis, which results in funneling volumes of data of unknown relevance to the analysts.

Victoria Stodden, University of Illinois at Urbana-Champaign, asked

Suggested Citation:"2 Data and Analysis in the Intelligence Community." National Academies of Sciences, Engineering, and Medicine. 2018. Learning from the Science of Cognition and Perception for Decision Making: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25118.
×

Fingar to speak further on the topic of transparency, particularly with regard to reporting inferences drawn from data and models and communicating with end users about how changing aspects of the data collection methods or the models could change the inferences drawn. Fingar replied that he believes trust is the critical link because policy makers cannot always attend to all the intricacies of the data. However, he continued, analysts need to be prepared to show their work, if necessary, in order to be as transparent as possible about the complexity of the data. This entire process, he added, hinges on policy makers’ trust in the IC’s objectivity, in the capability of the individual analyst, in the analyst’s having tapped into the broader IC and outside expertise as appropriate, and in the analyst’s ability and willingness to show the work if requested.

Suggested Citation:"2 Data and Analysis in the Intelligence Community." National Academies of Sciences, Engineering, and Medicine. 2018. Learning from the Science of Cognition and Perception for Decision Making: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25118.
×
Page 7
Suggested Citation:"2 Data and Analysis in the Intelligence Community." National Academies of Sciences, Engineering, and Medicine. 2018. Learning from the Science of Cognition and Perception for Decision Making: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25118.
×
Page 8
Suggested Citation:"2 Data and Analysis in the Intelligence Community." National Academies of Sciences, Engineering, and Medicine. 2018. Learning from the Science of Cognition and Perception for Decision Making: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25118.
×
Page 9
Suggested Citation:"2 Data and Analysis in the Intelligence Community." National Academies of Sciences, Engineering, and Medicine. 2018. Learning from the Science of Cognition and Perception for Decision Making: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25118.
×
Page 10
Next: 3 Forecasting and Anticipatory Thinking »
Learning from the Science of Cognition and Perception for Decision Making: Proceedings of a Workshop Get This Book
×
Buy Paperback | $52.00 Buy Ebook | $41.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Beginning in October 2017, the National Academies of Sciences, Engineering, and Medicine organized a set of workshops designed to gather information for the Decadal Survey of Social and Behavioral Sciences for Applications to National Security. The fourth workshop focused on the science of cognition and perception, and this publication summarizes the presentations and discussions from this workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!