1

Introduction

In September of 2020, the Department of Defense (DoD) published a data strategy (DoD 2020a), intended to support the National Defense Strategy and Digital Modernization program. While not focusing on or mentioning acquisition explicitly, the DoD Data Strategy (summarized in Figure 1.1) includes guiding principles, data goals, and essential capabilities needed to meet those goals. The strategy envisions a DoD that is a “data-centric organization that uses data at speed and scale for operational advantage and increased efficiency.” (p. 2). The strategy frames data as a strategic asset and concludes that “every leader must treat data as a weapon system, stewarding data throughout its lifecycle and ensuring it is made available to others” (p. 11). See Box 1.1. However, successful implementation of a data strategy requires the support of leadership across all aspects of the organization, adequate funding, and a trained workforce. While the DoD Data Strategy is in the early phase of deployment, the importance of data as a strategic resource and enterprise asset for DoD cannot be understated.

In 2019, the Office of Under Secretary of Defense for Acquisition and Sustainment (USD(A&S)) recognized the rapidly expanding tools and techniques available in data science and commissioned this report to identify how DoD can enable the defense acquisition system and its workforce to make better use of data science. To that end, this report explores the critical roles for data within the defense acquisition system, the importance of aligning and operationalizing a data strategy to improve acquisition processes and insights, and approaches toward developing an acquisition workforce better able to apply data-driven methods to their work and interpret their results. Data alone will not remedy many of the complex problems facing

the acquisition system, nor does every acquisition professional need to become a data scientist or data engineer. Nevertheless, improved appreciation, sharing, use, and analysis of data along with data-informed decision making have the potential to dramatically improve overall efficiency and effectiveness of defense acquisition. This report conveys existing opportunities to leverage advances in data science for defense acquisition. As data science and its tools are rapidly evolving, DoD will need to continually refine and update its understanding of data science opportunities and tailor them to defense acquisition.

HARNESSING DATA FOR DEFENSE ACQUISITION

Data science—the multidisciplinary field that encompasses the technologies, processes, and systems to extract knowledge and insight from data and enable data-informed decision making under uncertainty, to be discussed later in this report—has become a pervasive and transformative field unto itself over the past decade. Building a skilled workforce that is able to apply and adapt to the quickly evolving domain of data science

is essential for organizations looking to improve performance through more optimal use of data. A skilled workforce—one capable of extracting value from data—will include a number of different data science roles and teamwork among them. Some team members will have technical expertise in data science, while others are more specific to the “domain” (contracting, acquisition, materials, testing, cyber, etc.). All are more fully described later in this report.

With respect to data scientists specifically, private industry holds a significant advantage in recruiting and retention, in part because of the high salaries offered and the flexibility of these organizations to more rapidly adopt new approaches. In comparison, currently DoD and other government organizations have a more limited toolset available to recruit, train, and retain skilled data scientists. As DoD acquisition contemplates the use of data science teams in support of its acquisition workforce, the cost and challenges of recruiting sophisticated data scientists may be a driver in how many and where such teams are established and the mix of talent between military, civil service, and contracted specialists.

In spite of the recruiting and retention challenges in the data science, an increasing number of the defense acquisition teams are embracing aspects of data science—in particular, the use of data and data analysis—and are reaping the benefits. The USD(A&S) has an important mission of developing policy and processes to enable more effective and efficient acquisition by the DoD Components¹ that result in operational benefits to warfighters. With the potential to provide products for rapidly evolving missions, accelerate logistics, lower costs, and save lives, both the Office of the Secretary of Defense (OSD) and the Components are striving to collect, manage, and make better use of data. Examples are described in Box 1.2.

The successes of teams—such as those cited in Box 1.2—are encouraging, yet even these successes had their share of challenges with access to data, modern tools, and talent. Studies of existing DoD acquisition data systems often cite the enormity, complexity, and siloed nature of data within the Defense community and the related challenges these issues cause. For example, in the January 2019 Report of the Advisory Panel on Streamlining and Codifying Acquisition Regulations (Section 809 panel), Recommendation 89 identified the “[t]he proliferation of different data architectures throughout the acquisition and financial system leads to countless marginal inefficiencies in DoD,” and recommending that Congress “Direct DoD to

___________________

¹DoD Components are the Office of the Secretary of Defense, Office of the Chairman of the Joint Chiefs of Staff and the Joint Staff, Combatant Commands, Office of Inspector General, Military Departments, Defense Agencies, DoD field activities, and other organizational entities, which includes the National Guard Bureau (DoD 2020b).

consolidate or eliminate competing data architectures within the defense acquisition and financial system” (DoD 2019, p. 1).

As a result of the proliferation of data architectures and long-standing challenges in data labeling and terminologies, the DoD acquisition system often misses opportunities to share and contextualize data, account for un-

certainties, and consistently incorporate data use and analysis into decision-making processes. Data use for acquisition insights has been siloed within disciplines and organizational elements, and widespread data sharing and use of more sophisticated tools has been limited. Moreover, the pool of data science experts that understand defense acquisition is small. These

data-related challenges are common across government organizations and all sectors of industry. However, due to the size and complexity of the DoD acquisition system, described later in this report, it can be challenging to determine a universal approach for improving data use. In general, though, extracting decision quality information from data will depend on use of the data life cycle, to be described later in this report.

A comprehensive data strategy, specific to USD(A&S), could help move beyond pockets of innovation and meet the need for data-informed decision making throughout the defense acquisition system. Because the defense acquisition system relies on data generated and maintained throughout DoD, a successful implementation of a data strategy would guide which data to generate and collect and how that data should be stored, managed, shared, and used. As USD(A&S) determines its data strategy consistent with the 2020 DoD Data Strategy framework (Figure 1.1 and Box 1.2), it needs to frame, inform, prioritize, and guide all aspects of the data life cycle in order to fully capitalize on the value of data already generated within the acquisition system, as well as tap into and feed data into other DoD data systems.

LEADERSHIP DRIVING CHANGE IN A DATA-CENTRIC ORGANIZATION

Chief data officers (CDOs) for OSD and the military departments play a key role in the implementation of the DoD Data Strategy. Brought in prior to the release of the strategy, these CDOs are collectively seeking order and discipline in data use.

During the April 2020 National Academies workshop on “Improving Defense Acquisition Workforce Capability in Data Use,” Mr. Tom Sasala, the chief data officer for the Department of the Navy, noted that two of the Navy’s key challenges are that (1) data are not available for mission commanders, warfighters, and decision makers, and (2) data silos limit real-time use of data and linked data, and force duplication of efforts (Sasala 2020). As CDO of the Navy, he stated that much of his role is to make data available to address the challenges of accessibility, sharing, and security. “Our goal is just to maximize access to tools and data and allow people to be innovative.” Current priorities for Navy include data specifications, common platform development, and data integration, which are the data engineering tasks that prepare data for use in analysis and decision making (Sasala 2020).

Ms. Eileen Vidrine, CDO of the Air Force, shared her office’s goal to “make data part of every airman’s core DNA” (Vidrine 2020). The Air Force CDO oversees a small laboratory based at Andrews Air Force Base. This lab considers topics proposed by Air Force personnel and develops teams composed of a data scientist, data architect, and a subject-matter

expert to work on small use cases that could have enterprise-level impact. In their work, the lab identifies certain data sets that have been required multiple times and prioritizes their cleaning, sharing, and accessibility (Vidrine 2020). Additionally, the lab discovered useful data at the tactical level that are stored on an individual’s desktop.

In a January 2020 interview, Vidrine explained, “In our first year, we just wrapped up our 15th use case, and several of them are now being scaled across the enterprise. We take a problem set, we use data science to solve it, and then we use that as the seed to grow the next use case moving forward” (Miller 2020). A key goal is to leverage cloud infrastructure to allow the airmen to register the data and make them discoverable so that the data can support broader strategic questions.

The DoD CDOs operate at the Service level, not specifically in the acquisition lane. But the data fundamentals are the same: good, clean, shareable data.

The long-term vision and opportunity for data science is integrating it into operations rather than just a side support function providing insight. As one moves from the concept of data science as a separate activity to one where the use and analysis of data are fully integrated within an organization’s operations, management, and decision making, system interoperability not only facilitates data sharing (to enable analysis) but also data-informed decision making and operations. To achieve this, system interoperability, data sharing, and data accessibility are critical. DoD has taken some steps in improving the syntactic and semantic interoperability of some key data systems that collect acquisition data on large acquisition programs through open interfaces and data governance, but continued progress is needed (Anton et al. 2019). See Box 1.3 for the Federal Data Strategy approach to data accessibility and insight for DoD. For example, integrated data science is seen in DoD systems (e.g., in the integration of analytics in autonomous systems), but expanded integration in acquisition processes holds the potential for other systems as described in Chapter 4.

Adopting a data-informed approach to acquisition processes and management will require fundamental changes across DoD, and senior DoD leaders have an important role to play in making that vision a reality. The successful use and application of data are often facilitated by leaders who value data and encourage their use and who have secured some resources, established a routine workflow, and acquired personnel with the necessary skills for applying aspects of data science to acquisition, most commonly data analytics. The Secretary of Defense and other senior leaders can facilitate adoption by expressing an expectation that decisions are based on the most accurate information available. If these leaders insist on using data-driven processes to make these data-informed decisions, they must also demand that the data are high quality and as complete and accurate

as possible and provide the necessary resources. Such expectations and requirements will push these data-centric practices and the required use of data analytics and other aspects of data science down into other parts of the acquisition system. By making decisions informed by data, leadership sets an example that can promote the cultural changes necessary to facilitate data-informed decisions across the entire department.

The USD(A&S) has provided strong verbal support for data-informed decision making in defense acquisition—the following are just a few such

examples. In her remarks on October 28, 2019, at the National Academies, Ms. Stacy Cummings, Principal Deputy Assistant Secretary of Defense for Acquisition, stated that the USD(A&S) prioritizes bringing innovation into the defense acquisition system through data, data-informed policy making, and an empowered workforce. Critical to this study, Ms. Cummings added that there is a need to transform training so that there is a baseline set of data-use skills throughout the defense acquisition workforce.

The USD(A&S)’s June 2020 memorandum, Data Transparency to Enable Acquisition Pathways (USD(A&S) 2020), highlights the value of providing decision makers access to comprehensive and accurate data. These statements in support of more data-informed decision making have been reinforced in the September 9, 2020, release of DoD Directive 5000.01 by the Deputy Secretary of Defense as well as the January 23, 2020, release of DoD Instruction 5000.02 by the USD(A&S).

In early 2020, USD(A&S) established the Adaptive Acquisition Framework, a set of acquisition pathways to help the workforce to tailor strategies to deliver better solutions faster (see DoD Instruction 5000.02, 2020). Meanwhile, through the Human Capital Initiatives,² USD(A&S) is initiating the Back-to-Basics initiative (USD(A&S) 2020), which is a major reform of the defense acquisition workforce management framework that will align with the Adaptive Acquisition Framework.

Given the recent adoption of the DoD Data Strategy 2020 Action Plan and the strong support from leadership, defense acquisition is well-positioned—from a policy perspective—to strengthen its data infrastructure and data-centric skills of its workforce.

STUDY ORIGIN, OBJECTIVES, AND APPROACH

Over recent years, the National Academies of Sciences, Engineering, and Medicine has convened several activities to identify the challenges

___________________

²See https://www.hci.mil/.

of, and highlighting best practices in, postsecondary data science education (NASEM 2018, 2020a; NRC 2015). During discussions around these activities, representatives from DoD highlighted needs for strengthened appreciation and understanding of data science within the defense acquisition workforce, leading to discussions about creating a consensus study tailored to this context.

Around the same time, Congress passed the 2018 National Defense Authorization Act, directing DoD to establish a “set of activities that use data analysis, measurement, and other evaluation-related methods to improve acquisition program outcomes” (Section 913). These activities are to include the establishment of focused research and educational activities at the Defense Acquisition University³ and appropriate private-sector academic institutions to support enhanced use of data management, data analytics, and other evaluation-related methods to improve acquisition outcomes.

At the request of USD(A&S), the National Academies convened a committee of diverse experts to help address the need for improved data use within the defense acquisition workforce. This study was conducted over 18 months, included a public workshop in April 2020 with a published proceedings (NASEM 2020b), and produced this consensus report. The complete description of the committee’s charge can be seen in Box 1.4. A list of committee members along with their biographies is shown in Appendix F, and a list of meetings and information-gathering sessions held by the committee is shown in Appendix A.

Terminology—from both the DoD acquisition and data science communities—and a common understanding of terms are critical to addressing the Statement of Task. In particular, there are challenges in translations and contexts of terms and phrases; for example, “analysis” has different definitions in the two communities. Accordingly, throughout the study and within this report, the study committee sought to establish a common understanding of critical terms and be consistent with their use. Please see Appendix E for a glossary of these terms and phrases.

REPORT OUTLINE

This report is structured as follows. For short versions of background topics, Chapter 2 gives a brief summary of defense acquisition and its workforce and Chapter 3 discusses data science and the data life cycle. Chapter 4 explores the current uses of data in defense acquisition and identifies opportunities to exploit data and improve program and institutional performance. Chapter 5 identifies the portfolio of skills and organizational options for data science teams, and how supervisors with non-technical

___________________

³Recently renamed just “DAU.”

backgrounds can effectively manage data science projects. Techniques and programs for upskilling the acquisition workforce are explored in Chapter 6 using best practices and lessons learned from academia, industry and other government agencies. Finally, this report’s findings, conclusions, and recommendations are listed in Chapter 7.

REFERENCES

Miller, J. 2020. “The Big Data Challenge Getting Smaller for Army, Air Force as CDOs Mature,” Federal News Network. January 24. https://federalnewsnetwork.com/ask-the-cio/2020/01/the-big-data-challenge-getting-smaller-for-army-air-force-as-cdos-mature/.

NASEM (National Academies of Sciences, Engineering, and Medicine). 2018. Data Science for Undergraduates: Opportunities and Options. Washington, DC: The National Academies Press. https://doi.org/10.17226/25104.

NASEM. 2020a. Improving Defense Acquisition Workforce Capability in Data Use: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/25922.

NASEM. 2020b. Roundtable on Data Science Postsecondary Education: A Compilation of Meeting Highlights. Washington, DC: The National Academies Press. https://doi.org/10.17226/25804.

NRC (National Research Council). 2015. Training Students to Extract Value from Big Data: Summary of a Workshop. Washington, DC: The National Academies Press. https://doi.org/10.17226/18981.

Sasala, T. 2020. “Perspectives from the Chief Data Officers.” Presentation at the Workshop on Improving Defense Acquisition Workforce Capability. Virtual. April 14.

USD(A&S) (Under Secretary of Defense for Acquisition and Sustainment). 2020. “Data Transparency to Enable Acquisition Pathways.” Department of Defense. https://www.acq.osd.mil/aap/assets/docs/USA000854-20%20Signed%20Memo.pdf.

Vidrine, E. 2020. “Perspectives from the Chief Data Officers.” Presentation at the Workshop on Improving Defense Acquisition Workforce Capability. Virtual. April 14.