National Academies Press: OpenBook
« Previous: Front Matter
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

Summary and Recommendations

Despite the decades of success that the Department of Defense (DOD) has had in leveraging information technology (IT) across the defense enterprise to build the world’s most powerful military force, its acquisition of IT systems continues to be burdened with problems. Briefings to the National Research Council’s (NRC’s) Committee on Improving Processes and Policies for the Acquisition and Test of Information Technologies in the Department of Defense from the staff of the Office of the Secretary of Defense (OSD) show that the acquisition of major automated information systems (MAIS) remains especially slow. This problem has been broadly recognized for years, and there have been many efforts at reform in the past. Nevertheless, today’s processes for the acquisition and testing of the DOD’s IT systems all too often deliver solutions that are too late to satisfy the needs of the user community. Current fielding cycles are, at best, two to three times longer than successful commercial equivalents according to presentations to the committee—representing multiyear delays in delivering improved IT systems to warfighters and the organizations that support them. As a result, the DOD is often unable to keep pace with the rate of IT innovation in the commercial marketplace, cannot fully capitalize on IT-based opportunities, and is unable to deliver IT-based capabilities rapidly to meet urgent requirements. Moreover, to the extent that adversaries have access to state-of-the-art IT and can put it to operational use rapidly, the DOD faces risks across the battlespace.

Regarding the adoption and use of IT, the commercial sector has also experienced delays and failures as well as extraordinary successes. Of

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

special interest in the commercial sector, however, is the extent to which new or adaptive corporations using agile approaches for the acquisition and operational deployment of IT capabilities have outstripped industrial giants that were beset with their own ponderous, process-bound, industrial-age management structures.

As called for in the committee’s statement of task (see Box P.1 in the preface), this report examines the acquisition, culture, practices, processes, and rules within the DOD as they apply to information technology; assesses whether the DOD could adopt best practices from the commercial sector for IT acquisition, systems engineering, and test and evaluation (T&E); and makes recommendations to improve the speed and effectiveness of IT acquisition programs.

Many studies have recommended reforms to the defense acquisition system—that is, the institutions, processes, and rules that govern the development, procurement, testing, and fielding of new capabilities. A number of these, including a recent study by the Defense Science Board,1 have focused on IT acquisition and concluded that there is a need for a unique acquisition process for IT. This study reaches the same fundamental conclusion but adds another dimension in its elaboration of differing types of IT systems and offers a suggested acquisition process for each.

SCOPE AND VOCABULARY FOR INFORMATION TECHNOLOGY ACQUISITION PROGRAMS

Information technology is used for a wide variety of purposes in the DOD, a breadth suggested by the definition of “information enterprise” in DOD Directive (DODD) 8000.1.2 However, the issues, challenges, and potential solutions are not the same for all elements of the information enterprise. Moreover, the present defense systems acquisition vocabulary

1

Defense Science Board, Report of the Defense Science Board Task Force on Department of Defense Policies and Procedures for the Acquisition of Information Technology, Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, Washington, D.C., March 2009.

2

DOD Directive 8000.1 (Management of the Department of Defense Information Enterprise) defines “information enterprise” as follows: “The DoD information resources, assets, and processes required to achieve an information advantage and share information across the Department of Defense and with mission partners. It includes: (a) the information itself and the Department’s management over the information life cycle; (b) the processes, including risk management, associated with managing information to accomplish the DoD mission and functions; (c) activities related to designing, building, populating, acquiring, managing, operating, protecting, and defending the information enterprise; and (d) related information resources such as personnel, funds, equipment, and IT, including national security systems.”

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

embodied in the law and regulations governing IT acquisition3 does not provide a taxonomy of IT program characteristics suitable to allowing program types to be matched with appropriate acquisition approaches. Consequently, the committee created new definitions for a well-defined subset of the information enterprise that has parallels in the commercial sector.

The committee decided to consider IT systems to be just those systems that support the DOD information enterprise,” especially those systems expected to run on or interface with existing infrastructure and systems that are user-facing, and limited to those that are delivered through the acquisition process (and not systems “homegrown” in individual commands). (An IT program is defined in this report as the process for acquiring an IT system as defined here.) Excluded from this subset are the IT-based components embedded in weapons systems or DOD-specific hardware. The committee believes that as a general rule such embedded components (for example, firmware that controls a missile’s flight) are so integral to the weapons system that they are best managed using the same processes as those used for managing the other elements of the weapon. The committee considers it likely that at least some of the conclusions in this report would also apply to other systems with significant IT content—but this report and its recommendations are focused on opportunities to improve the acquisition of IT systems as defined here.

To better align the acquisition approach with technical characteristics and risk, the committee further subdivides IT systems into two categories:

  • Software development and commercial off-the-shelf (COTS) integration (SDCI) programs—those focused on the development of new software to provide new functionality or focused on the development of software to integrate COTS components, and

  • COTS hardware, software, and services (CHSS) programs—those focused exclusively on COTS (hardware, software, or services) without modification for DOD purposes (that is, the capabilities being purchased are determined solely by the marketplace and not by the DOD).

By separating programs that involve software development and/or integration from programs that entail simply adopting COTS technology, these categories facilitate the comparison of defense acquisition processes

3

See, for example, DOD Directive 5002.15000 series regulations; Title 10, U.S.C., Chapters 144 (“Major Defense Acquisition” Programs) and 144A (“Major Automated Information Systems Programs”); and DOD Directive 8000.1.

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

with commercial best practices and the adoption of practices focused on effective and timely leveraging of the commercial IT cycle.

FINDINGS AND RECOMMENDATIONS

IT Acquisition Process and Culture (Chapter 2)

Findings

The acquisition framework of the Department of Defense4 prescribes elaborate governance mechanisms and cost thresholds that trigger varying levels of oversight and review. In response to acquisition problems from the past, more oversight and governance, some of it excessively process-centric and adversarial, have been added. Collectively, these well-intended changes have made the timely delivery of IT capabilities more difficult. Today’s IT acquisition process often focuses on review documents and other process artifacts, and the acquisition culture does not reward early and transparent feedback on capabilities and limitations. Success as determined by process metrics in acquisition does not necessarily align with success metrics based on the timely delivery of end-user capability.

The DOD’s perceived need for caution over speed is understandable. Given the criticality and danger of its mission, its worldwide operations and large workforce, and the frequent need for clear, decisive action, the DOD by its nature is an organization with a classic command-and- control culture. However, if current trends continue, it is likely that processes and systems will become even more top-down and centralized, in spite of the DOD’s desire to move to an integrated, cross-Service environment with empowered decision making at all levels of command.

DOD systems acquisition policies, expertise, practice, and culture— including those applied to IT systems—reflect the practices, policies, and cultural norms associated with large weapons systems programs. This report does not address the issue of whether the process for weapons systems is appropriate. Other studies and reports have focused on that question—and many have pointed out shortcomings. With respect to IT, however, there is a long-standing reluctance to deviate from standard weapons system acquisition processes, and acquisition personnel are not trained or led to differentiate the unique aspects of IT systems acquisition.

4

As set forth in Title 10, U.S.C., Chapters 144 (“Major Defense Acquisition Programs”) and 144A (“Major Automated Information System Programs”) and DOD Directive 5002.15000 series regulations.

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

The application of current, weapons-system-based acquisition processes to IT systems has a number of deleterious effects on the DOD’s ability to deliver needed end-user capability, including the following:

  • With the exception of hardware and licensed software purchased through vehicles such as the DOD Enterprise Software Initiative contracts, applicable COTS technologies are insufficiently leveraged, excessively tailored, inefficiently tested, and excessively delayed. Many programs have experienced acquisition or integration lead times that significantly exceed the life cycles of the underlying COTS technology. The discrepancy between DOD fielding cycles and COTS life cycles is stark, and measured in years.

  • The oversight process focuses too much on shortcomings of COTS products and services and inhibits the timely delivery of meaningful (albeit imperfect) end-user capabilities.

  • IT program requirements are often written with overly detailed specifications, take a long time to develop, and are not consistent with the pace of technological change or the rapid delivery of end-user capabilities.

  • DOD acquisition, budgeting, and requirements processes, which are designed for large weapons systems acquisition programs, are being inappropriately applied to relatively low-dollar IT programs.

  • Dollar thresholds are used to assign the level of oversight for IT programs. These levels are significantly lower than the dollar levels used for determining oversight levels for weapons system programs. This disparity subjects too many IT programs to time-consuming, high-level DOD oversight and prevents the delegation of oversight to lower levels that are more agile.

  • The DOD’s acquisition training curriculum does not adequately address the special challenges of IT system acquisition or prepare program managers to run IT programs effectively. This shortfall impedes the DOD’s ability to assess, adapt, and adopt applicable commercial methods, processes, products, and services.

Recommendations

Recommendation 1. Adopt a new acquisition process tailored for IT systems.

For IT systems, the acquisition processes, which are currently defined by the 5000 series of DOD regulations, should be replaced with a new process designed specifically for the timely and effective acquisition of

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

IT systems. Elements of this process are detailed in Chapter 3 and in Appendixes B and C.

The supporting recommendations that follow address key areas where an adjustment of acquisition practice and cultural change are needed.

Recommendation 1.1. Emphasize timeliness and end-user mission success in the DOD IT acquisition culture rather than rigid oversight and process compliance.

Cultural aspects of the DOD acquisition process that have an impact on the potential success of IT acquisition efforts include the following: the bias that larger is better, the sense that oversight personnel have no accountability for delaying needed IT capabilities, an emphasis on process risk (executing the acquisition process correctly) rather than on the risk of late delivery of end-user capability, an unwillingness to admit program failure, an emphasis on process over product, a belief that the DOD is genuinely unique, and the belief that what is good for large weapons systems should be good enough for IT systems.

In addition to implementing a new acquisition process for IT capabilities, the DOD should institute a companion program to address the cultural changes required to make the new acquisition process successful. As examples, the DOD should require that all personnel in the oversight process have accountability for the program (that is, for helping the program succeed or helping terminate programs that are fatally flawed or no longer required). The focus should be on providing products with value to end users, not the oversight process per se. It is also important that the oversight culture support small, incremental cycles for IT development and fielding.

Recommendation 1.2. State IT systems requirements as top-level mission expectations (that is, “big-R” requirements) rather than as detailed processes or technical solutions; develop the details (“small-r” requirements) by iterative refinement with users.

Today, there is an extensive requirements-definition period for the typical IT program, resulting in a large volume of requirements documentation that must be formally validated by the Joint Requirements Oversight Council for major joint programs. This requirements-definition and budgeting process encourages the aggregation of requirements into larger programs. In turn, these requirements documents are used to justify program budget requests, and the approval of these requests is a condition for proceeding to successive acquisition program milestones. Because the requirements-definition and approval processes introduce

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

significant delays, requirements documents for IT programs often have become inaccurate descriptions of user needs by the time that funding is obtained and the acquisition process is initiated.

Instead, IT systems requirements should be defined at the mission capability level. Top-level requirements (big-R) should be rapidly developed and validated, and more detailed requirements (small-r) should be developed as an integral part of the acquisition process.5 Specifically, based on iterative interactions with the actual end users of the IT capability as well as on assessments of available technology, more detailed requirements would be developed for individual programs or projects. As the acquisition effort progresses, feedback from operational users after deployment of increments of capability would be used to feed the continued evolution of (small-r) requirements for the IT acquisition effort. Doing so will permit iterative refinement with users and the best use of commercially available technologies. At the same time, this recommendation should not be construed to suggest that big-R requirements will not change. Feedback on those is also important and should be accommodated.

The committee’s recommended process is similar to one recently established by a Joint Capabilities Integration Development System policy. The approach is currently being applied to a small set of programs; the committee encourages the expansion of this concept to encompass all IT programs.

Recommendation 1.3. Leverage flexibilities within IT acquisition funding to achieve speed and agility in the new acquisition process.

The DOD’s process for obtaining funding for new acquisition programs typically takes a number of years. In summary, a DOD capability shortfall is linked to a request to Congress for funding that would be provided in a future year. For solutions that will rely on IT, the time frame for seeking funding can be many times longer than the actual time needed to develop or procure the solution. To achieve rapid delivery of IT solutions, a more responsive process is needed for justifying and allocating funding to address capability shortfalls.

In the short term, the DOD should work with Congress to explore how to make use of flexibility consistent with current legal requirements.

5

“Big-R” requirements convey a widely recognized purpose, mission, and expected outcome. “Small-r” requirements provide a set of more detailed requirements associated with specific user interfaces and utilities that will evolve within the broader specified architecture as articulated in the initial big-R requirements.

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

(Even where legislative changes are not needed, flexibility must nonetheless be negotiated with the congressional defense oversight committees.) For example, the DOD has authority to allocate funds for urgent war-fighter needs and to reallocate funding after congressional appropriation as a result of changing needs. This authority could be used to begin the development of a needed capability in weeks or months. Also, acquisition funds are sometimes allocated by Congress to a larger mission or program area, or in some cases to a portfolio of projects identified with an area of mission need—for example, to fund software upgrades in a particular mission area or system. Funding could rapidly be allocated to meet demands for IT capabilities within these areas. In both cases, transparent reporting to Congress will be essential to ensure proper oversight and to demonstrate that the flexibility yields a more rapid delivery of capability to the field.

In the longer term, the DOD should work with Congress to establish a new set of funding mechanisms for meeting IT requirements that would align congressional funding with mission or capability areas rather than with individual acquisition programs. Under this concept, Congress would allocate funding to a mission area that would be governed in the DOD with a portfolio-management-like process. In implementing this concept, DOD officials would be responsible for setting priorities and allocating the funding to individual projects following appropriations for a portfolio of mission requirements. This approach would ensure appropriate justification of funding needs tied to mission requirements during budget submission as well as the rapid allocation of appropriated funding consistent with the pace of evolving mission requirements and technology advancements. Currently, the DOD uses a process similar to this concept for funding maintenance upgrades to aircraft avionics software. Likewise, a somewhat similar process is also used for managing IT projects funded through working capital funding processes.

Recommendation 1.4. Provide IT systems acquisition professionals with education in modern IT systems and establish minimum competency standards.

Training a professional acquisition workforce in modern IT program management is critical. IT systems-acquisition professionals must understand key discriminants of the process, be able to evaluate proposals, and make trade-offs. Their training should include requirements specification; development, integration, and test processes; listening to and incorporating “the voice of the user”; and the use of COTS.

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

Recommendation 1.5. Use pilot programs to institutionalize the new IT acquisition process recommended in this report.

Pilot programs would provide the DOD with the means to apply new acquisition approaches and to capture valuable lessons learned and develop the necessary guidance for applying the knowledge acquired to larger and future programs. The committee believes that the establishment of 10 pilot programs for the rapid acquisition of IT systems capabilities under the alternative IT acquisition process described in this report would greatly help in moving the acquisition process forward. These pilot programs could be the catalyst for institutionalizing a new and updated IT systems acquisition process and provide an opportunity for the DOD to deepen the IT systems acquisition experience of its workforce.

Recommendation 1.6. Propose legislative and regulatory changes (1) to codify a new agile process for acquiring IT systems and (2) to revise dollar thresholds for the oversight of IT systems acquisition in order to foster decentralization.

The committee proposes that two major initiatives be undertaken to update legislation and regulations governing IT acquisition in the DOD. The first initiative would involve the adoption of a new agile process as the default for the acquisition of IT systems. Central to the new process would be the concept of the iterative, incremental acquisition of capabilities that are delivered to end users as successive packages that are aggregated over time into comprehensive capabilities—a concept that is consistent with today’s best commercial practices. This concept would apply to the acquiring of custom IT capabilities that are developed or integrated and are deployed on existing IT infrastructures (SDCI systems) as described above. In addition, a companion process would be tailored to the acquisition of off-the-shelf IT capabilities (CHSS systems). Processes for both types of IT acquisition are described in detail in this report.

The second initiative would be to restructure and decentralize the IT acquisition oversight process in order to align it with the fast-paced cycles of agile and rapid acquisition. Today’s acquisition oversight process in the DOD is designed for the disciplined management of large, expensive, and complex weapons systems. However, the dollar thresholds for designating oversight levels for IT programs are significantly lower than those used for weapons systems (by a factor of five). Moreover, the current legislation has no provision for MAIS programs to be designated as acquisition category (ACAT) II, which would provide for oversight at the Service or agency level. One approach to solving the problem of highly centralized oversight with its attendant delays would be to use the same

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

dollar thresholds in effect for major defense acquisition programs for the designation of ACAT levels to MAIS programs. This change in thresholds for IT programs would foster decentralization and would better align authority for IT program oversight to the appropriate levels—at OSD, the Services and agencies, and lower echelons.

IT Systems and Software Engineering Processes and Practices (Chapter 3)

Findings

Information technology programs are profoundly affected by the rapid and relentless pace of change in the underlying technologies. Hardware capability per unit expenditure doubles roughly every 18 months. Software capability is driven by the even-faster pace of technology change in the Internet environment and by the elevated level of end-user expectations that this causes. DOD IT systems acquisition programs progress at a markedly slower pace. As a result, the DOD is unable to keep pace with the rate of IT innovation in the commercial marketplace, cannot fully capitalize on IT-based opportunities, is unable to deliver IT-based capabilities rapidly, and, accordingly, will not have the requisite agility.

Historically, system development has followed a “waterfall” process calling for formally documented specification, followed by a request for bids, followed by contracting, delivery, installation, and maintenance. Major elements of the waterfall method, which is document-intensive, attempt to satisfy management’s goals for ensuring that projects will successfully meet their objectives, but they end up emphasizing the acquisition process rather than the capability being delivered.

To deliver software capability more rapidly, the commercial world has widely embraced the iterative, incremental development (IID) approach, which addresses two issues of central importance for IT systems—(1) the need for user interaction in setting requirements and (2) complexity.6 Key attributes of the IID approach include (1) the prominence of the end user’s voice, (2) a focus on big-R requirements during early planning, (3) the close integration of developmental and operational T&E into the development cycle, and (4) the breaking down of a project into incremen-

6

There is another, alternative strategy based on incremental development strategies—evolutionary acquisition, which addresses a concern somewhat different from that addressed by IID, namely, the issue of technology maturation. Under evolutionary acquisition, early increments provide end-user capabilities based on mature technology, and work on later increments is deferred until needed technology has been matured.

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

tally deliverable parts. The commercial world has widely embraced the IID approach.

Although the DOD’s current governance and oversight structure permits tailoring and provides the flexibility needed for a milestone decision authority and program manager to adjust how the process is applied to specific programs, there is no established best practice or accepted template for tailoring. Instead, each decision maker must independently address on an ad hoc basis the misalignment between an oversight system designed for hardware and weapons system development and the very different types of issues that contribute to risk in an IT system program.

With the existence of multiple oversight bodies and large numbers of participants in the program oversight and review process, the current system gives undue leverage to groups that often are not true stakeholders in the process. This can have negative effects, including too many detailed and ad hoc small-r requirements placed on the program, an inability to prioritize requirements effectively, and “corner case” requirements (focused on rare situations that only occur outside normal operating parameters) that can be contradictory or extremely difficult to implement.

Recommendations

Recommendation 2. Adopt an iterative, incremental approach for acquiring information technology systems.

The following supporting recommendations are aimed at enabling more rapid and nimble IT systems acquisition processes and fostering the institutionalizing of IID practices.

Recommendation 2.1. Establish iterative, incremental development (IID) processes based on agile software development and related approaches as the default for IT system development.

Software engineering, although still a young discipline, has advanced substantially over the past four decades, and much is understood about how to structure development efforts to better manage the unique risks inherent in IT system programs and to improve the probability of success. In particular, it is now widely appreciated that the waterfall development process is not appropriate for most software development projects. One emerging approach to IID that has many attractive attributes is agile software development (ASD),7 which has at its core ongoing, integrated developmental and operational testing and a close coupling between

7

A. Cockburn, Agile Software Development, Addison-Wesley, Boston, Mass., 2001.

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

those responsible for requirements, for development, and for testing. It emphasizes frequent testing and interaction over formal, up-front documentation. The adoption and implementation of an agile-inspired IID approach do not equate simply to compressing traditional waterfall models into shorter time periods, nor does appending current document-centric oversight processes to a series of release phases equate to the use of ASD.

IID and ASD are proven approaches for building capabilities in line with end-user expectations and needs and for rapidly iterating requirements and solutions based on end-user feedback. Such approaches have been embraced in the commercial world as being highly effective ways to deliver incremental improvements to the field rapidly. Their adoption not only would conform to widely accepted commercial best practices but also would implement more than 20 years of recommendations from forums such as the Defense Science Board. The adoption of IID approaches coupled with a focus on the end-user experience does not mean, however, that other stakeholders and nonfunctional requirements (such as information assurance, reliability, and so on) are unimportant. Historically, other stakeholder voices have dominated the process to the exclusion of the end user. The committee urges a rebalancing and a focus on end-user mission success, because it is the end user who is in the best position to judge whether a capability is useful and should be fielded.

Recommendation 2.2. Allocate top-level DOD mission expectations (i.e., big-R requirements) across increments and use each increment to define and satisfy detailed requirements (i.e., small-r requirements).

Beyond the highest set of (big-R) requirements, there is a more detailed set of (small-r) requirements for such things as specific user interfaces and utilities that will be developed and will evolve within the context of the big-R requirements. Stated another way, users cannot effectively articulate their requirements without interacting with (even partially) fielded systems—in IT systems, one cannot establish what is needed without a sense of what is possible. Requirements such as the expected user interface and user paradigms and integration with other concurrently evolving systems and security practices all dictate that the initial specification be limited to broad system goals and physical operating conditions and that more detailed requirements be evolved in concert with user feedback garnered during incremental development cycles.

Recommendation 2.3. Establish separate and distinct strategies and processes for acquiring custom versus off-the-shelf IT systems.

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

There are fundamentally different classes of issues involved when dealing with software development and commercial off-the-shelf integration, or SDCI, versus the commercial off-the-shelf hardware, software, and services, or CHSS, components of IT system programs; therefore, different strategies are appropriate when addressing them. In both cases, rapid change in virtually all aspects of the technology—requirements, capabilities, user expectations, usage and development environments, and so on—is a fundamental factor that must be addressed, and IID acquisition strategies are appropriate. Where the requirement includes COTS hardware or licensed software that can be procured at volume discounts through DOD enterprise contract vehicles (such as Enterprise Software Initiative contracts), decentralized procurement should be encouraged. When the DOD awards contracts requiring that such components are to be integrated into an IT system solution, those COTS products should be acquired by the most cost-effective means available. Options should include authorizing the contractor to procure COTS components from established government vehicles such as Enterprise Software Initiative contracts and providing the COTS components as government-furnished equipment to avoid unnecessary handling or procurement fees. This is particularly true for licensed software when existing enterprise contracts permit direct downloading.

Recommendation 2.4. Establish, employ, and report measures of success that emphasize the end-user experience, including timeliness to field.

The committee fully anticipates that any new IT systems acquisition process that is defined and adopted will need to evolve over time. The committee has identified shortcomings in the present system, and recommends a new direction. An appropriate evaluative framework should consider end-user capability, end-user satisfaction, timeliness, quality, operating costs, and acquisition (that is, improvements in the process and overall competitive strategy).

Recommendation 2.5. Provide a stable budget profile across multiple increments for iterative, incremental development of IT programs.

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

Stable budget profiles for the IT program acquisition cycle are necessary to ensure that end users’ requirements can and will be addressed, some at the outset and others in future increments. This will avoid the unintended but real consequence of users attempting to overload requirements into the first capability increment. Multiple time-boxed8 capability increments will fit within each Planning, Programming, Budgeting, and Execution budget cycle. The confidence of key stakeholders, including users, will increase, as will the transparency of overall program execution status.

IT System Testing (Chapter 4)

Findings

Today, testing discipline is integrated too late and serially into DOD IT systems acquisition practices. Indeed, programs generally defer the testing of IT systems in realistic operational environments until the mandated operational test. Without regular feedback from a user perspective on IT system development, program managers and milestone decision authorities lack critical information for managing and supporting programs, and other key stakeholders lack the knowledge that can build their confidence. This approach stands in contrast to best commercial practice, whereby user testing is performed early and often to ensure that user feedback guides all stages of system development. Because DOD end-user engagement is nonexistent or limited, the test community is unfairly tasked with representing the perspective and needs of end users and is often engaged too late to have a substantive impact on requirements, system architecture, or system functionality. Finally, the acquisition community has been reluctant to embrace virtualized testing and has been overtly precluded from reusing or accessing operationally relevant test data and environments.

Recommendations

Recommendation 3. Perform continuous testing, with early involvement from end users, in acquiring DOD information technology systems.

8

Time boxing refers to a deadline-driven approach to system development in which work items may slip from one iteration to the next, but iterations are completed according to schedule, thus affording the opportunity to quickly identify erroneous estimates of the time required to complete deliverables and ensuring continuous user input regarding priorities.

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

The following supporting recommendations would establish testing and evaluation policies and practices in support of the committee’s proposed DOD IT systems acquisition process.

Recommendation 3.1. Adopt continuous testing in DOD IT systems development, and insist on the use of metrics, especially emphasizing measures of end-user satisfaction.

Continuous user testing is an integral part of successful IT systems development and deployment. An acceptance team emphasizing the perspective of the end user and composed of operational end users, development testing and operational testing stakeholders, security certification and accreditation stakeholders, and interoperability stakeholders should be continuously integrated into the development process. Such integration is critical to ensuring both that the big-R requirements are the right ones and that the system is meeting both big-R and small-r requirements. To facilitate measuring success, a process involving a robust set of metrics should be in place through an IT system’s life cycle, from development to ultimate decommissioning. A process that collects, aggregates, and analyzes metrics on end-user service consumption and experiences will provide visibility into what features end users are actually using. Successful commercial suppliers of IT systems place considerable emphasis on metrics that capture actual end-user behavior measured by online interactions with deployed IT systems.

Recommendation 3.2. Emphasize the needs of end users by having the acceptance team play a lead role in recommending deployment decisions.

Continuous user input is critical to the successful development of new IT systems. There are numerous examples of new IT systems meeting every formal aspect of their specifications but not delivering the expected value to their users; conversely, a system that may not meet all aspects of a specification may already be an improvement over existing systems. With these situations in mind, the acceptance team should play a leading role in recommending deployment decisions. This would prevent the deployment of systems that are not adding value (independent of their readiness on paper per the specification), and it would also ensure more rapid deployment of systems that can improve the effectiveness of DOD users (especially the warfighter). Deployments may take the form of small trials, large-scale “beta” programs, or even full production deployments.

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×

Recommendation 3.3. Test with users in their actual work or field environment (sometimes referred to as a beta deployment).

It is essential to put capabilities into the hands of users early and to measure performance and otherwise obtain feedback. Engaging expected users of the system early enough makes it possible to provide meaningful feedback to developers and influence the small-r requirements. One especially useful approach is to engage users through pilot projects deployed in the field, possibly operating in parallel with production systems.

Recommendation 3.4. Accept certification and functional IT system component test results across organizational boundaries.

The DOD has broadly adopted a set of networking capabilities that are integral to every IT system. As a matter of DOD policy, such capabilities should not be required to undergo separate revalidation or recertification during operational testing. Examples include such capabilities as Internet Protocol and Domain Name System software modules. As more of the technology stack becomes commoditized or provided as a service, the set of associated capabilities not requiring revalidation should likewise grow. Current examples include public key infrastructure and on-demand computing and storage. The same policy should apply to DOD-deployed and DOD-certified software modules that are reused across DOD programs. The testing of the composite system product that is evaluated in a realistic environment for operational effectiveness and suitability is sufficient.

Recommendation 3.5. Use virtual test environments to support both continuous feedback and certification of IT program increments.

A variety of opportunities to establish virtual test environments already exist, as identified in Chapter 4. The definition of test environment could, over time, be significantly extended to include the use of beta testing in actual operational environments, the use of commensurate commercial data and/or operating environments, and the extension of virtual test environments to include operations monitoring as a source of continuous test feedback. The DOD should codify this approach and encourage the use of virtual environments. Operational realism is important but must be balanced against pragmatic considerations.

CONCLUSION

The current DOD approach to IT acquisition has not been broadly successful in delivering needed capability in a timely manner. To lever-

Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 1
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 2
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 3
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 4
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 5
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 6
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 7
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 8
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 9
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 10
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 11
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 12
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 13
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 14
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 15
Suggested Citation:"Summary and Recommendations." National Research Council. 2010. Achieving Effective Acquisition of Information Technology in the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/12823.
×
Page 16
Next: 1 Introduction »
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!