National Academies Press: OpenBook

Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science (2021)

Chapter: Appendix C: Data Science Case Studies in Defense Acquisition

« Previous: Appendix B: Defense Acquisition Notes
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×

C

Data Science Case Studies in Defense Acquisition

The use of data science to support decision making is not new to the acquisition community and has facilitated critical program support and even informed acquisition strategies for decades. However, the advent of increasingly robust data tools, associated visualization techniques and data analytics employed in the context of the burgeoning data science discipline offer the Department of Defense (DoD) an opportunity to institutionalize what once was a sporadic and often ad hoc process into a cohesive, consistent expectation across DoD programs.

This appendix highlights examples where data science has improved DoD acquisition processes. These examples illustrate how data analytics inform the understanding of existing processes and serves to measure the effect of new innovations. Analytics do not provide all the answers because human innovation, leadership, and decision-making are key considerations. What measurement and analysis enable are insights into the effectiveness of a system and how new ideas and approaches impact performance.

CURRENT USES OF DATA IN DOD ACQUISTION PROCESS

Program management in the DoD acquisition community is a data-driven discipline that involves controlling the cost, schedule, and performance of a project or set of projects. Defense acquisition program managers are responsible for achieving development, production, and sustainment objectives of their projects. The following case studies demonstrate how data science has been key to decision making in DoD acquisition programs for decades.

Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×

Data Science Application: Tracking costs through multi-year programs through common database and analysis platforms.

For decades, the Office of the Secretary of Defense Cost Assessment and Program Evaluation (CAPE) and its predecessors kept a database of contractor costs for DoD acquisition contracts that analysts used to make cost estimates. That database was assembled by individual analysts who manually entered data from individual contractor cost data reports submitted as part of the acquisition contracts. Beginning in 2013, analysts in CAPE set DoD on a path to capture this data automatically. The result of that effort is the Cost Assessment Data Enterprise (CADE).

CADE reduced the data entry burden on both the contractors and CAPE analysts, allowing more time for data analysis. Through CADE, CAPE’s goal is “to increase analyst productivity and effectiveness by collecting, organizing and displaying data in an integrated single web-based application….” CADE has made these data readily available in a format that is easily searchable to facilitate better and more consistent cost analysis across the entire DoD.

By using the data in CADE, cost analysts across DoD can more readily calculate the expected single-year cost of a procurement to support effective contract negotiations. This has led to significant savings for DoD. For example, from the FY 2017 Annual Report on Cost Assessment Activities, CAPE completed five analyses supporting multi-year procurements. Three were for aircraft programs, and CAPE estimated the resulting savings ranged from 10 to 13.7 percent (DoD 2018). Two were for ship programs with savings ranging from 5 to 10 percent. The total amount of savings from these five multi-year procurements is approximately $4.5 billion (in then-year dollars).

While DoD cost analysts in CAPE and in the military departments have done analysis to support multi-year procurements for a long time, CADE has made it easier for the analysts and has supported more consistent results across DoD enabling more effective contract negotiations. This enhanced use of data is summarized in Box C.1.

Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×

Weapons System Test and Evaluation

Data Science Application: Supporting statistical modeling and applying common tools and training to enhance testing and evaluation of weapons systems.

A key function of the DoD acquisition workforce is testing and evaluating new systems. The goal of operational test and evaluation is to understand how new or upgraded systems will perform under realistic combat conditions prior to full-rate production and fielding to combat units (DoD 2016). Understanding the capabilities and limitations of systems before they are fielded helps commanders in the field employ the systems effectively and

Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×

allows for corrective actions before large numbers of systems are produced to help minimize expensive retrofitting.

Test and evaluation is inherently data-driven, as it uses a series of events to gather data to understand the system’s effectiveness and suitability. This data supports decisions on design changes, corrective actions, and full-rate production.

Since 2008, the office of the Director of Operational Test and Evaluation (DOT&E) has supported a variety of efforts to upskill the test and evaluation (T&E) workforce in data science. Specific areas of focus include statistical design of experiments (DOT&E 2010), the design and analysis of surveys, design for reliability and statistical reliability assessment (DOT&E 2016), and methodologies for combining information from multiple tests (DOT&E 2016).

These efforts have focused on providing training resources tailored to the T&E community and on developing a community of practice.

  • Since 2012, the Institute for Defense Analyses (IDA) has offered a variety of 1- to 3-day on-site courses for the service operational test agencies on topics such as test planning foundations, design of experiments, statistical analysis, and introduction to statistical thinking.
  • The Test Science website1 provides case studies, online tools, free R software packages tailored for T&E professionals (Haman 2020; Morgan-Wall and Khoury 2020), webinars, and training videos.
  • The Science of Test (SOT) Research Consortium, sponsored by DOT&E and the Test Resource Management Center, is an academic research consortium supporting faculty experts in experimental design, statistical analysis, reliability, and modeling and simulation from the Naval Postgraduate School, the Air Force Institute of Technology, and six additional universities.
  • The SOT Research Consortium provides data science expertise by developing new techniques, producing content for the Test Science website, collaborating with DoD professionals, and training students for the workforce (DOT&E 2016).
  • The Defense and Aerospace Test and Evaluation Workshop,2 sponsored by DOT&E, the National Aeronautics and Space Administration (NASA), the American Statistical Association, and IDA, provides an opportunity for practitioners to interact with researchers, develop collaborations, take short courses, and receive feedback on projects and issues.

___________________

1See https://testscience.org/.

2See https://dataworks.testscience.org/.

Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×

These training efforts have had impacts at a variety of levels. In 2007, Congressional concern about body armor testing led to DOT&E working closely with the services to develop test protocols for the ballistic components of combat helmets. Applying statistical design of experiments at a program level allowed DoD to establish a testing and analysis program that identified an initial design for the Enhanced Combat Helmet as providing insufficient protection against small arms penetrations. This analysis led to a key design change in the ballistic shell laminate material of the combat helmet (DOT&E 2016; Hester et al. 2015).

Similarly, a detailed data analysis on the operational evaluation of the Littoral Combat System directly influenced Defense Secretary Hagel’s decision to direct the Navy to consider a new frigate.

Within defense acquisition, the T&E community have a long track record of valuing and using data. Training programs are robust, civilian and military career paths support the T&E demand, and a common lexicon is in place. Data are valued and demanded by leadership and decision makers.

Velocity Management: How Data Analysis Helped to Improve the Army’s Logistics System

Data Science Application: Tracking maintenance data and replacement parts fulfillment.

In the realm of Army logistics, a classic example of how data science and analytics can enable a major transformation of critical defense systems is the U.S. Army’s Velocity Management effort. Starting in 1994, the Army sought to improve its order fulfillment and related processes for parts. As shown in Figure C.1, the key to improving this logistics process was to define and establish performance metrics and associated data to understand how well the system is performing at baseline, diagnosing performance drivers, and informing process improvements. Army logisticians monitor data and metrics and the effects of any changes driving iterative improvements.

The Army improved its data collection systems to better measure data on order submissions, item movement, and delivery locations. These data were a key enabler for the effective use of performance metrics. Figure C.2 illustrates some of the basic elements in the order fulfillment process for Army units in the continental United States; Army units abroad have additional elements (Dumond et al. 2001).

To highlight how data analysis improved Army logistics, consider how Velocity Management examined whether to add or remove an item from the warehouse inventory. Algorithms calculated the tradeoff between the effect of stocking an item and its associated cost. Analysis revealed that

Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Image
FIGURE C.1 Steps in the Define-Measure-Improve Method. SOURCE: Dumond et al., 2001; Dumond and Eden, 2005. Courtesy of RAND.
Image
FIGURE C.2 Order fulfillment process for army units in the continental United States. SOURCE: Dumond et al., 2001; Dumond and Eden, 2005. Courtesy of RAND.
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×

stocking additional inexpensive items reduced customer wait time with little investment (Dumond et al. 2001). Many of these low-cost items were small, so stocks remained mobile with little increase in total weight and volume. Other analyses identified items that were no longer in demand and could be removed from storage. In addition, data analytics identified when warehouses and storage containers could be reconfigured to improve space utilization for higher-demand items and generating credit for returning unnecessary items (Dumond et al. 2001).

Figure C.3 illustrates the number of days to fulfill orders from Fort Bragg for repair parts before and after Velocity Management (not counting backordered parts). At the median (the point at which half of the order parts are fulfilled), the delay was reduced from 18 days to 5 days. The delay on three-quarters of ordered parts was reduced from 28 days to 7 days. Thus, military equipment can be put back into the field in about a quarter of the time compared to when Velocity Management was not in place.

These data do not specifically reveal which repairs were accelerated, which equipment are considered critical, and what the associated warfighting effect is of the equipment (i.e., it could be that the most critical equipment still takes a long time while simpler equipment is turned around faster). However, the fact that the speed-up extends into the 95th percentile affects overall operational readiness and availability of the warfighter.

Image
FIGURE C.3 Improvement trends for order fulfillment times before and after velocity management. SOURCE: Dumond and Eden, 2005. Courtesy of RAND.
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×

Dumond and Eden (2005) report that data-driven measurement, metric feedback, simplified performance rules, and new information systems were key elements. These metrics and associated data helped to inform the status of the Army logistics system and demonstrated that other key elements were effective in speeding delivery, including leadership commitment, streamlined reviews, scheduled truck deliveries, higher fulfilment from the main depot, and direct deliveries to customers. A brief summary of this improved use of data for the Army logistics system appears in Box C.2.

Business Analytics Saves Air Force $2 Billion and Counting

Data Science Application: Business analytics to inform acquisition contracting.

Category management is an effort to move from managing purchases and prices individually across thousands of procurement items to managing spend and total cost for entire categories of purchases. In 2014, the Air Force published the Air Force Category Management Concept of Operations (CONOPS), leveraging an industry best practice employed by leading companies (e.g., Walmart, UPS, Kroger). In the category management construct, the Air Force determined common categories of products and services (i.e., heating and air conditioning, airfield paving, working dogs, etc.), assigned Air Force-level responsibility to a category manager with

Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×

expertise in their assigned category and who now manages their categories as a virtual mini-business with its own set of strategies.

The backbone of this Category Management CONOPS is the Business Intelligence Competency Cell (BICC) under the Air Force Installation Contracting Center at Wright-Patterson AFB in Ohio. Its process couples historical spend data and functional data to provide complete “as-is” Air Force benchmarks; engages industry on cost drivers, best practices, benchmarks, and customary commercial practices; and identifies gaps in cost, performance, and implements execution plans to address the gaps.

BICC capitalized on existing commercial data tools. For example, it procured an enterprise license for an industry leading market intelligence service covering more than 900 reports on common commercial goods and services and acquired a license for Tableau, a modern commercial off-the-shelf data visualization software. Using these tools, the BICC team linked existing government data sets such as Federal Procurement Data System-Next Generation, the System for Award Management, and the Contracting Business Intelligence Service, which collects data from Air Force contract writing systems and Government Purchase Card programs to create the Air Force Business Intelligence Tool. This centralized team of 9 people delivered $2 billion in savings in its first 5 years of operation. The BICC team includes data analysts with domain expertise and data engineers. Five of the 9 team members are contracted, which means that BICC could not find the needed skills in the Air Force acquisition workforce so hired the talent by contract. Organizationally, this small team services the entire Air Force on strategic sourcing and category management initiatives. It also supports 78 Air Force Contracting Squadrons, and shares its tools and training with the many others including the Army, Navy, DHS, and GSA.

Data Driven Analysis of Alternatives Restructures C-17 Globemaster II Program

Data Science Application: Analysis of alternatives and cost comparison.

The U.S. Air Force’s C-17 Globemaster II is one of the most successful airlift platforms in world and widely flown by the Air Force. In addition to transoceanic flight and supply missions (goods, vehicles, and personnel), the C-17 performs tactical airlift, medical evacuation, and airdrop missions. While widely acknowledged as successful aircraft, the acquisition program to develop and purchase the aircraft was nearly canceled in its infancy due to rising cost and underperformance in the aircraft itself—proving earlier assessments of low development cost and risk inaccurate. This prompted Congress to mandate that DoD conduct an independent assessment of the program that would ultimately be executed by IDA.

Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×

IDA based its analysis on the data collected and developed a series of models and simulations to drive the analysis. For instance, to assess the C-17 performance and compare it against alternative models, the IDA developed the Airlift Loading Model (ALM) for loading, the detailed simulation model Mobility Analysis Support System (MASS) for airlift movement and cargo flows, and the Airlift Cycle Analysis Spreadsheet (ACAS) for assessments of cargo and passenger movement. ALM, MASS, and ACAS all served to collate data on aircraft performance and assess it against alternative systems. The IDA team also utilized cost estimates to support its analysis and compliment the performance models’ assessment.

Given the effectiveness of individual platforms, the potential of a mixed fleet, and potential changes to Army vehicle development outside of the Air Force’s programs purview, the report’s final analysis identified and prioritized the options and alternatives for DoD to consider.

While the initial decision following the IDA study was to cap the number of C-17s the Air Force would purchase—40 aircraft—and pursue a mixed fleet that would include 747s, the study prompted a change in the C-17 program. Using data analysis to lay bare how the aircraft compared to alternative options and identifying inefficiencies in their program, the study pushed the C-17 program to restructure itself. This restructuring combined with follow-up studies conducted by the Air Force and DoD with regard to program requirements, led to a decision in 1995 to proceed with the original purchase of 120 C-17s (IDA 2010). A brief summary of the use of performance and cost assessment data for the C-17 program appears in Box C.3.

REFERENCES

DoD (Department of Defense). 2016. Performance of the Defense Acquisition System: 2016 Annual Report. Under Secretary of Defense for Acquisition, Technology, and Logistics. October. http://www.dtic.mil/docs/citations/AD1019605.

DoD. 2018. FY 2017 Annual Report on Cost Assessment Activities. February. https://www.cape.osd.mil/files/Reports/CA_AR2017v10Final.pdf.

DOT&E (Director of Operational Test and Evaluation). 2010. “Director, Operational Test and Evaluation (DOT&E) FY 2010 Annual Report.” https://www.dote.osd.mil/Publications/Annual-Reports/2016-Annual-Report/.

DOT&E. 2016. “Director, Operational Test and Evaluation (DOT&E) FY 2016 Annual Report.” https://www.dote.osd.mil/Publications/Annual-Reports/2016-Annual-Report/.

Dumond et al. 2001. Dumond, J., M.K. Brauner, R. Eden, J.R. Folkeson, K.J. Girardini, D.J. Keyser, E. Peltz, E.M. Pint, and M.Y.D. Wang. 2001. “Velocity Management: The Business Paradigm That Has Transformed U.S. Army Logistics.” RAND Corporation. https://www.rand.org/pubs/monograph_reports/MR1108.html.

Dumond, J., and R. Eden. 2005. “Improving Government Processes: From Velocity Management to Presidential. Appointments.” RAND Corporation. https://www.rand.org/pubs/reprints/RP1153.html.

Haman, J.T. 2020. September.

Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×

Hester, J., T. Johnson, and L. Freeman. 2015. “Managing Risks: Statistically Principled Approaches to Combat Helmet Testing.” In Research Notes. Fall. Institute for Defense Analyses, Alexandria, VA.

IDA (Institute for Defense Analysis). 2010. “An Application of Cost-Effectiveness Analysis in a Major Defense Acquisition Program: The Decision by the U.S. Department of Defense to Retain the C-17 Transport Aircraft.” https://www.ida.org/-/media/feature/publications/a/an/an-application-of-costeffectiveness-analysis-in-a-major-defense-acquisitionprogramthe-decision-by/ida-document-d-4218.ashx.

Morgan-Wall, T., and G.A. Khoury. 2020. “Design of Experiments Suite: Generate and Evaluate Optimal Designs [R package skpr version 0.64.2].” Corpus ID: 226069590. https://www.semanticscholar.org.

Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 109
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 110
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 111
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 112
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 113
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 114
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 115
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 116
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 117
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 118
Suggested Citation:"Appendix C: Data Science Case Studies in Defense Acquisition." National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. doi: 10.17226/25979.
×
Page 119
Next: Appendix D: Skills for Data Science Mastery »
Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science Get This Book
×
Buy Paperback | $50.00 Buy Ebook | $40.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The effective use of data science - the science and technology of extracting value from data - improves, enhances, and strengthens acquisition decision-making and outcomes. Using data science to support decision making is not new to the defense acquisition community; its use by the acquisition workforce has enabled acquisition and thus defense successes for decades. Still, more consistent and expanded application of data science will continue improving acquisition outcomes, and doing so requires coordinated efforts across the defense acquisition system and its related communities and stakeholders. Central to that effort is the development, growth, and sustainment of data science capabilities across the acquisition workforce.

At the request of the Under Secretary of Defense for Acquisition and Sustainment, Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science assesses how data science can improve acquisition processes and develops a framework for training and educating the defense acquisition workforce to better exploit the application of data science. This report identifies opportunities where data science can improve acquisition processes, the relevant data science skills and capabilities necessary for the acquisition workforce, and relevant models of data science training and education.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!