6

Preparing and Sustaining a Data-Capable Defense Acquisition Workforce

Workforce development in data analytics and data science is critical to the Department of Defense’s (DoD’s) current and future acquisition functions, and it supports a broader DoD effort to make the department a data-centric organization, as outlined in the 2020 DoD Data Strategy (DoD 2020). While not all defense acquisition workforce members will or should be data scientists, becoming a data-centric organization depends on personnel that understand the value of data, the conditions that affect the quality of the data, and how data can better inform decisions in the acquisition environment. The key to success, however, will be attracting, developing, managing, and retaining personnel with robust data skills and experience; whether that be as a military member, civil servant, or a contracted resource.

A review of relevant training opportunities currently available to personnel provides a crucial background for the development and adoption of future programs and strategies. The committee undertook site visits to Defense Acquisition University (DAU), the Air Force Institute of Technology (AFIT), and the Naval Postgraduate School (NPS) to gather key information on current and prospective data training efforts available to acquisition personnel and leaders (see Appendix A). The committee also solicited input on data upskilling from organizations outside DoD through site visits and invited testimony. Taken together, information gathered from the site visits and invited speakers provided key insights for future data analytics and data science training programs for acquisition personnel.

OVERVIEW OF TRAINING APPROACHES IN DATA SCIENCE AND ANALYTICS

Given the rising use of data across many sectors and domains, it is important to recognize the variety of training approaches currently available. Many of these programs are evolving as new technologies and data platforms become available. Demand for data talent is expanding across industries, driving demand for multiple models of training delivery. Importantly, without operational context and an understanding of the specific programmatic needs, one type of training is not better than another, and therefore the committee does not endorse any specific training program or approach. However, understanding the characteristics of various training approaches can provide opportunities for acquisition professionals and managers to map training programs to their data needs and available resources. Below are brief descriptions of training approaches currently available for increasing data capabilities. Table 6.1 provides a summary of these training options and thus can be a guide for determining which platforms or approaches are appropriate for meeting specific data needs.

Higher Education

Higher education institutions, including 2-year and 4-year colleges, offer a variety of data-science training opportunities. They include traditional courses and degree offerings spanning associate’s, bachelor’s, master’s, and doctoral programs. Higher education institutions are externally and independently accredited to ensure that recognized standards are met. For most degree programs, learners must apply and be accepted into the program, but individual courses offered through continuing education programs may only require prerequisite courses for admittance. The majority of the data analytics and data science courses currently offered in higher education institutions occur over the span of a semester or quarterly schedule and are either in person, online, or in a hybrid model that synchronously combines in-person and virtual attendance. Currently, there are efforts for developing ABET (Accreditation Board of Engineering and Technology) accreditation of data science programs. Course fees range depending on the institution, but they can be costly if not offset by scholarships or employer contributions. Importantly, the data courses and programs in non-defense higher education institutions are not tailored for defense acquisition applications and scenarios.

At the time of this report’s publication, higher education is undergoing substantial adaptation as a result of the COVID-19 pandemic. Given safety precautions associated with in-person learning during the pandemic, higher education has rapidly adapted to increase access to courses through delivery

TABLE 6.1 Characteristics of Different Training Offerings for Data Analytics and Data Science

modes that allow for fully remote and hybrid participation.¹ Institutions have expanded digital infrastructure to support multiple education delivery modes, and are adapting pedagogical approaches that support digital learning environments. The adaptations not only protect the health of faculty and learners, but also connect institutions to a broader range of learners. The educational adaptations developed in response to the pandemic are creating a window of opportunities for higher education institutions to collaborate with a broader set of external stakeholders and organizations. Higher education is primed to partner with defense acquisition to provide accessible and tailored course offerings in data analytics and data science.

Certificate Programs

Over the past decade, there has been substantial growth in certificate programs in data science and data analytics. The growth in data certificates has occurred in both the higher education and commercial sectors, and some organizations have developed their own in-house certification programs. Typically, certificate programs are a collection of courses that can be delivered either in person or online and take less time to complete than traditional academic programs. However, some certificate programs are self-paced, and completion of a certificate can span from a few weeks to two years. Admission into a data science certificate program operated by academic institutions is similar to applying for an undergraduate program. Fees for certificate programs range across institutions, but they often require less investment than a graduate degree program.

Certificate programs are usually geared toward professionals that already have some business or computer science experience and can serve as an indicator of proficiency in data science concepts. They can also offer opportunities for learners to broaden their data analysis techniques and master new data tools. It is important to note that data certificate courses are generally not easier than master’s degree courses. In fact, learners in higher education data science certificate programs may be enrolled in graduate-level courses, simply taking fewer courses than required for a graduate degree.

Data Bootcamps

Data analytics and data science bootcamps are typically characterized as short, intensive programs designed to quickly teach a combination of

___________________

¹“Here’s Our List of Colleges’ Reopening Models” from The Chronicle of Higher Education available at https://www.chronicle.com/article/heres-a-list-of-colleges-plans-for-reopening-inthe-fall/ (as of March 11, 2021).

theoretical concepts and applications through interactions with the full data life cycle. Content can be delivered either in person or virtually in data bootcamps and typically involve 6–15 weeks of instruction. Some bootcamps are project-based, meaning the learners complete a project that demonstrates their abilities, as an immersive experience where learners can refine and apply their skills.

While some higher education institutions offer bootcamps for data analytics and data sciences, most are offered in the commercial sector at variable cost ranging from $25 a month to nearly $20,000 for a full program. A strong appeal of some data bootcamps are promises of job interviews or placements after successful completion of the program. However, bootcamps are not accredited, and it may be difficult to ensure the programs meet the recognized standards associated with higher education institutions.

Micro-Credentials

Micro-credentials are a form of specialized certification that verifies or attests to the proficiency of specific skills or competencies. They differ from traditional degrees and certificates in that they are generally offered in shorter or more flexible time spans and are more narrowly focused. Micro-credentials can be offered online, in the classroom, or in hybrid approaches through higher education institutions and third-party providers. Sometimes called digital badges or nano-degrees, micro-credentials can also be “stacked” by building upon one another as the learner continues mastering specific skills or competencies.

There are many benefits to using micro-credentials for increasing data analytic skills, including their specificity, short duration, flexibility, affordability, and personalization. Micro-credentials can also serve as an indicator of proficiencies and professional development if they are recognized by the learner’s industry or employer. Depending on the provider, micro-credential courses are free to low cost and relatively easy to adapt for specific use cases.

Massive Open Online Courses

Another opportunity for learners to acquire data science and data analytics skills is through Massive Open Online Courses (MOOCs), which are fully online courses that can support thousands of learners at a time. MOOCs are often free, open-access courses where learners do not typically have to apply or complete prerequisites to enroll. The learners are self-directed and do not engage with the instructor. While course completion does not often lead to a certificate, learners may have the option to pay their MOOC provider for a verified certificate or digital badge indicating the

successful completion of the course. Well-known MOOC providers include Coursera, Udacity, Khan Academy, and edX (see, for example, Ngo 2020).

The challenges of using MOOCs in defense domains include concerns about security, rigor, and the reliability of the information provided. Students must also self-regulate and set their own goals. However, several DoD training institutions have partnered with select MOOC providers to deliver training or course content that is readily available, rather than spending months developing new courses. Through these established partnerships, MOOC courses can count toward certificates or digital patches to verify proficiency in various skills and competencies.

Executive and Leadership Training

The growing demand for data analytics and data science skills has seen a concomitant demand for team leaders and decision makers who can facilitate collaboration between technical and business personnel, manage data science teams, communicate relevant data findings, and apply data to decision making. Transforming a team or organization to become increasingly data-centric is a complex endeavor and leaders play a critical role in supporting that transformation. Beyond learning skills specific to the data life cycle associated with their role, non-data skills and strategies can help leaders advance the use of data on their team.

While there is no clear agreement on the non-data skills necessary for team leaders and decision makers, the National Association of Corporate Directors does offer some research-based principles to advance the oversight of digital transformations (Van der Oord et al. 2019). These principles include approaching technology as a strategic imperative, developing continuous technology learning goals and development paths, realigning leadership to reflect the growing significance of technology, and demanding frequent reporting on technology initiatives. While not specific to data use, learning how to apply these or similar principles could help acquisition leaders guide their teams through effective data use.

There are a variety of training programs in data analytics and data science for non-technical managers and leaders throughout higher education and the commercial sectors, including certificate programs. Data science programs for leaders may offer opportunities to learn both data and non-data skills, and many utilize a blend of lectures, interactive discussions, and case studies. Content can be delivered either in person or virtually, and costs can range based on the content and training provider. Additionally, some organizations partner with higher education institutions or third-party providers to develop executive leadership programs with content applicable to the leaders’ needs. However, it is important to note that assessments and evaluations of these programs are scarce. As leaders consider training op-

tions for their data needs, it is key that they identify programs that align with the skills associated with their data roles.

DATA USE AND ANALYSIS TRAINING OPPORTUNITIES FOR DEFENSE ACQUISITION PERSONNEL—CURRENT PROGRAMS

Personnel in the defense acquisition workforce, as well as a large proportion of support contractors, receive training for their jobs through Defense Acquisition University (DAU). The mission of DAU is to “provide for—(1) the professional educational development and training of the acquisition workforce; and (2) research and analysis of defense acquisition policy issues from an academic perspective” (10 USC, Sec. 1746). DAU has several “brick and mortar” campuses across the United States, but many of its courses and training programs are provided online. The institution both develops its own courses and works with external vendors to offer a broad selection of courses relevant to the defense acquisition workforce.

Within defense acquisition, personnel are certified for their respective positions. DAU provides certification across 14 different career fields, and within each career field, there are three tiers of certification. Based on testimony provided to the committee in October 2018 by Ms. Megan McKernan from the RAND Corporation, DAU offers more than 220 courses related to data and/or analytics, and enrollments in these courses topped 200,000 in the 2018 fiscal year. In addition, DAU offered a webinar series in 2020 on digital readiness that covered some fundamentals of data science (DAU 2020a). The learning outcomes for these courses are unclear, but the time frame for completion is often brief. Ms. Darlene Urquhart, Director of the Enterprise Integration Directorate at DAU, informed the committee during its December 2019 meeting that DAU was interested in training learners to identify problems that data analytics could help them solve. However, Urquhart shared that DAU had not yet hired a data scientist to its faculty and there is an institutional preference to utilize commercially available training for data analytics training, such as MOOCs.

A site visit to DAU in January 2020 provided insights into the institution’s current and future efforts to meet data analytics and data science needs for acquisition personnel. During the site visit, committee members heard from DAU leadership and interacted with faculty and learners in a

300-level program management course and a 400-level program manager case study course. When asked about training needs in data analytics, learners in the 300-level classes expressed their interest in learning how to ask “the right questions,” find relevant data, and use the appropriate skills for data analysis. However, trainees shared that the data they require are sometimes inaccessible or have been manually entered into basic spreadsheets, making the data difficult to manipulate, analyze, and verify.

During the classroom visits, several committee members also observed that the data sets used in case studies and computer simulations incorporated oversimplified data, and learners did not need to address uncertainty in their decision making. Uncertainty relates to the understanding that while data are rarely perfect, when properly analyzed they can be sufficient for guiding decision making. It is important that people who make decisions based on data recognize that real data are often messy, variable, and represent only a sample. Learners are often surprised by how much effort is involved in finding and cleaning data. Similarly, when they encounter real data sets in the workplace, there is concern about ambiguity because they did not encounter it in their training. For instructors with some level of data acumen or data literacy, incorporating realistic data sets into courses and discussing the value and complexity of those data may enable learners to better integrate the methods from class into their daily work (American Statistical Association 2016).

At both the site visit and the committee’s data-gathering workshop, the Director of User Experience and Platform Optimization at DAU, Ms. Maryann Watson, shared that high interest in data and analytics courses drove efforts at DAU to develop an optional credentialing program in October 2019 to deepen skills beyond its three-tier certification program. The credentialing program provides acquisition personnel with targeted and job-specific learning experiences. Some DAU credentialing courses, such as for agile software development and data analytics, are offered by commercial MOOC vendors such as Coursera (Pearson and Trevino 2019). While current credentialing courses are focused on entry-level workers, DAU anticipates developing more advanced credentials in the future. As of summer 2020, DAU offered the certification program Data Analytics for DoD Acquisition Managers.

As DAU expands data analytics training through its new credentialing pathway, the Naval Postgraduate School (NPS) and the Air Force Institute of Technology (AFIT) offer graduate-level programs in data analytics that are open to acquisition personnel. In addition to their graduate offerings, both institutions also provide continuing education opportunities and formal certificates where acquisition personnel can access training in data analytics.

NPS offers a four-course certificate in data science that includes a blend of lecture and lab hours. The objective of the NPS data science certificate

program is to provide learners with training in statistical and machine learning techniques to manage and gain insights from data. Learners enrolled in the program are required to have some background in statistics and programming languages.

NPS also offers a certificate program in Data Sciences and Analytic Management. Dr. William Muir, Assistant Professor in the Graduate School of Defense Management at NPS, informed the committee during its October 2019 meeting that one of the goals of the data science and analytics programs is to train NPS graduates for effective collaboration among data analysis, scientists, statisticians, and leaders. NPS faculty incorporate publicly available and relevant data into their courses, both within and outside the certificate program in Data Sciences and Analytic Management. As a result, NPS students can access real-time acquisition data, including streaming data, which emphasizes the need for the interoperability of data and data sharing. Muir shared that there are data sets that can take an acquisition officer between 6 and 12 months to acquire, but that same data can be accessed by NPS students in fewer than 15 minutes. There is potential for downstream frustration when graduates re-engage in their acquisition projects and are unable to readily access relevant data at the speed NPS provided.

AFIT offers a certificate in data analytics and another in data science. The data analytics certificate program is a five-course distance learning experience that includes introduction to data analytics, data and databases, introduction to machine learning, statistics, and computer programming with Python. The certificate program in data analytics is focused on the use and understanding of data applications and tools, not mathematical theory and algorithm development. AFIT’s data science certificate program (DSCP) offers graduate-level training on advanced analytic techniques, large and complex data sets, and computer programming. In addition to training learners in all aspects of the data life cycle, the DSCP also provides training on speaking to disparate groups within an organization to implement data science applications and solutions.

Defense acquisition personnel can enroll in data analytics degree programs, certification pathways, and continuing education courses at DoD-affiliated institutes such as DAU, NPS, and AFIT, but there are also opportunities for personnel to receive data training at non-DoD institutions. In some instances, DoD educational institutions engage with external partners to fulfill training criteria. For example, DAU offers an Equivalency Program, which provides an opportunity for higher education institutions, other DoD schools, commercial vendors, and professional associations to offer courses, programs or certifications that DAU will accept as equivalent (DAU 2020b). As of July 2020, at least 16 colleges and universities provided courses approved by DAU’s Equivalency Program, but none of

the approved courses included data analytics or data science. However, the DAU Equivalency Program could provide opportunities in the future for acquisition personnel to engage in additional training that builds on the new data analytics certification program. Broadening DAU’s educational partnerships allows DoD to outsource some training and potentially develop hiring pipelines at partner institutions.

While not specific to defense acquisition, the Army launched a higher education partnership program in 2020 to build personnel capabilities in data science and artificial intelligence. The Army Artificial Intelligence Task Force is a two-year pilot program with Carnegie Mellon University (CMU) providing master’s degrees in data science and data engineering to both uniformed and civilian Army personnel (Army Futures Command 2020). Army also is also partnering with CMU to launch a future artificial intelligence executive education program to provide Army leadership with a high-level understanding of data science to make informed and strategic decisions to integrate data capabilities into Army operations (CMU 2020).

At its data-gathering workshop in April 2020, the committee learned about emerging data training efforts across the U.S. Department of the Navy (DON) and the U.S. Department of the Air Force. While not specific to the acquisition workforce, Mr. Thomas Sasala, Chief Data Officer of DON, noted that DON has created partnerships with commercial companies to offer micro-degrees to build up data acumen, and public universities, such as Old Dominion University, to offer on-site training. Similarly, Ms. Eileen Vidrine, Chief Data Officer of the Air Force, shared how the upcoming establishment of the Air Force Digital University (the Digital University) will include block-style short courses on various topics including data use and data science for airmen and Air Force civilian personnel (Kanowitz 2020; Barnett 2020a, 2020b). The Digital University has engaged in partnerships with commercial training companies, such as Udacity and Pluralsight, to deliver technical content and boost digital literacy in the Air Force. Courses in the Digital University will also be designed to provide accessible courses for senior executives and high-ranking uniformed officers.

Other federal efforts are under way to provide training resources for critical data skills. According to Action 13 of the Federal Data Strategy 2020 Action Plan, the General Services Administration (GSA) will develop a curated catalogue of existing training providers, programs, courses, certifications, and other opportunities for federal personnel to practice and apply new data skills (Federal Data Strategy 2020). While not specific to data skills, DoD’s Enterprise Course Catalog will centralize tens of thousands of training and course offerings so that they are searchable through a single web portal (Advanced Distributed Learning Initiative 2020). These catalogues could be useful tools in the future for accessing data training resources and developing data training plans.

An emerging DoD acquisition training program for artificial intelligence (AI) could reveal additional strategies for training acquisition personnel in data science and data analytics. In October 2020, the AI Education Strategy was piloted with 84 DoD acquisition and requirements personnel to increase AI capabilities (DoD 2020). Key to the AI Education Strategy are workforce archetypes with specific AI roles and training needs. Each AI archetype has subcategories of learning outcomes and competencies tiered by levels of proficiency that connect with curricular recommendations. Close monitoring of this program and the utility of the archetypes could be informative to future data science and data analysis training efforts for defense acquisition personnel.

Overall, DoD already has access to a variety of data training options and partnerships that could potentially be expanded for defense acquisition personnel. Opportunities will continue to evolve in response to demand for training in data science and data analytics, but it is key that as programs are developed and adapted for acquisition personnel, they are assessed and evaluated for their success in increasing learner competencies in data use for acquisition contexts. The committee encourages DAU, NPS, and AFIT to continue piloting new approaches to delivering data analytics and data science training to acquisition personnel, but evaluations and assessments are essential to those piloting efforts. As the acquisition workforce engages in new data training programs, clear evaluation plans will help inform future efforts for scaling and modifying the programs to best address acquisition data needs.

INDUSTRY AND GOVERNMENT TRAINING APPROACHES IN DATA SCIENCE AND ANALYTICS

Given the competition to hire data talent, many organizations across the United States are addressing their increased demand for data capabilities by developing their own training programs, pathways, or partnerships. While not all the non-DoD training programs reviewed by the committee are directly applicable to the defense acquisition experiences, there are opportunities to apply some industry and government training strategies.

On a virtual site visit to Lockheed Martin, Mr. Bruce Litchfield (Vice President, Sustainment Operations at Lockheed Martin) shared that increasing data use at such a large company required a cultural transformation. While technological capabilities are critical for effective data use, success is tied to leadership, people, and processes. Lockheed Martin started with clarifying its vision to align with its desired outcomes: aiming to build a data-centric enterprise that collects, integrates, and analyzes data to continually improve performance. Once the data vision was established, transformation efforts at Lockheed Martin focused on building a system that addressed the data needs most relevant to its vision. To improve data literacy for approximately 50,000 employees at Lockheed Martin, the company selected the basic data skills its employees need to add value to the corporation while not detracting from processes and policies already in place. A catalogue of training modules was developed around the selected skills and include many free or low-cost modules offered by commercial vendors. Employees then take a data literacy assessment to inform them on which training the individuals should complete. A more intense training program is offered for Lockheed Martin employees working toward the position of data scientist or data engineering. While the specifics of this

training program were not shared at the site visit, approximately 1,000 out of approximately 50,000 Lockheed Martin employees participate in the program.

Mr. Melvin Greer, Chief Data Scientist with Intel USA and committee member, discussed at the data-gathering workshop how Intel approaches its goal of training all its employees in data science. While Intel is known as a semiconductor manufacturer, it has pivoted its focus to data analysis. Intel has identified the development, growth, and analysis of data as central to its ability to innovate and grow. As a result, all employees are provided training in data science and AI. Intel’s training was developed internally, and the curriculum provides a foundational understanding of data science. Intel’s required training is not intended to develop every employee into a data engineer or analyst, but to instead ensure that every employee has a basic understanding of how data science contributes to Intel’s development and growth as a data-centric organization.

Federal agencies outside of DoD also offer personnel opportunities for data training, and training material and structure may be scalable to DoD-led efforts. For example, a program known as Commerce Academy at the Department of Commerce piloted freely available courses, including a course on data science for leadership. When the Commerce Academy was running, a high level of interest was reported in both the interactive and recorded courses. No evaluation of the program was conducted, but former Commerce Academy director, Natassja Linzau, shared her reflections of the strengths and weaknesses of the training program. According to Linzau, the training courses were most beneficial when Commerce Academy staff directly provided the training because they could offer contextual framing of the learning materials through relevant case studies. The training program offered multiple learning formats, including in-person and synchronous virtual courses, to accommodate various personnel schedules. However, the Commerce Academy staff encountered difficulties providing necessary software on learners’ computers, utilizing a viable set of relevant data in the training modules, and sustaining the program through staff transitions.

In response to increasing data collection, the U.S. Department of Health and Human Services (HHS) developed the Data Science CoLab to build a community of employees as skilled data scientists.² A cohort from HHS participates in an eight-week program that includes a blend of courses in data science, statistics, and programming, and concludes with a capstone project. Courses are taught by external experts, but the cost of the program is fully covered by HHS. As of 2021, the Data Science CoLab is a fully

___________________

²HHS, “The HHS Data Science CoLab,” last reviewed on August 14, 2020, https://www.hhs.gov/cto/initiatives/data-science-colab/index.html.

virtual training experience and accessible to contractors as well as HHS employees.

Some industry organizations invest in partnerships with higher education institutions to offer learners, or potential employees, opportunities to hone their training in data analytics or data science through real-time application. For example, the University of Massachusetts at Amherst offers a summer program that matches data science undergraduate and graduate students to MassMutual in order to work on external projects. MassMutual also worked with the university to establish a college of data science and a master’s degree program. In this scenario, the higher education institution serves as a talent pool for the company, and MassMutual can hire graduates of the program with foundational skills in data science.

ENVISIONED FUTURE FOR TRAINING IN DATA USE CAPABILITIES FOR DEFENSE ACQUISITION PERSONNEL

Defense acquisition handles large volumes of data across different tools and platforms. Personnel need to be empowered to harness those data. As outlined in this chapter, there are a variety of training programs to increase data capabilities for the defense acquisition workforce so they can use that data to inform better decision making. Of the programs listed in this chapter, there is no one-size-fits-all training approach. Rather, acquisition teams can consider their current and projected data needs and select training programs that enable them to meet those needs. Training requirements will shift depending on professional roles, the application of data, the scale of data, and mission objectives. Certain positions may require more domain-expertise while others are more data-centric, but importantly, high levels of both domain and data science expertise for an acquisition personnel are unreasonable (Anton et al. 2019). An effective data training strategy will involve coordination across teams to balance data-related capabilities with acquisition-specific expertise.

Training and sustaining a data-capable workforce in defense acquisition are essential and require resources and leadership, especially as the field of data science continues to evolve and advance. To keep up with these advancements, training programs for the acquisition workforce will need to continually adapt while being accessible to learners of all backgrounds with limited available time and resources. Providing a variety of training opportunities for increasing the data capabilities of the defense acquisition workforce will enable data-informed decision making to improve cost, schedule, and performance across DoD.

REFERENCES

Ngo, C. 2020. “10 Best Free and Affordable Platforms for Online Courses,” Fort Mill, SC: Best Colleges, April. https://www.bestcolleges.com/blog/platforms-for-online-courses/.

Pearson, D., and A. Trevino. 2019. “Refreshing Acquisition Workforce Skills.” Defense Acquisition University. https://www.dau.edu/library/defense-atl/blog/Refreshing-Acquisition-Workforce-Skills.

Van der Oord, F., S. Mezeu, L. Chacko, and R. Lam. 2019. Governing Digital Transformation and Emerging Technologies: A Practical Guide. The National Association of Corporate Directors.