Improving Metrics for the Department of Defense Cooperative Threat Reduction Program (2012) / Chapter Skim
Currently Skimming:
3 Improvements to CTR Metrics
3 Improvements to CTR Metrics
Pages 29-54
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 29... ...
program because the programs illustrate the points well.23 Those sections are followed by recommendations on independent evaluation and factoring time and change into metrics. Chapter 3 closes with a set of important issues for Cooperative Threat Reduction (CTR)
|
From page 30... ...
A proxy measure typically does not encompass all of the important aspects of status or performance, so managers sometimes use multiple proxy measures. Multiple proxies can increase the data collection requirements and only provide marginal capability to assess the achievement of the objective, but they may be all that are available.
|
From page 31... ...
Measures Time FIGURE 3-1 Metrics organized into six categories using systems thinking. The term on the top of each category is the term the committee uses in the report to assess the CTR metrics.
|
From page 32... ...
, but same model forms the basis for program strategy and planning. Proxy measures can be derived from the model, but the most desired outcomes (no WMD attacks)
|
From page 33... ...
(Ball et al., 2004) Surveys may be helpful in CTR programs.
|
From page 34... ...
Objectives and Partnership For each program in the DoD Metrics Report, DoD should include a concise statement of its objectives and of how the program is intended to reduce threat or risk. Objectives for projects and the overall CTR Program in a partner country are developed jointly between the United States and the partner country.
|
From page 35... ...
The DoD Defense Security Cooperation program also shares important similarities with the DoD CTR Program and may in some cases serve as a model. Not only can DoD learn from other agencies, but DoD will be working with other agencies in a "whole of government" effort, and they might already have mechanisms in place for measuring impact and effectiveness that would be useful to DoD CTR.
|
From page 36... ...
The typical approach TABLE 3-2 Comparison of the CTR Program with other DoD programs that face challenges developing metrics Capacity Building Program Cooperative Threat Defense Security DoD Capabilities in Iraq and Feature Reduction Cooperation Based Planning Afghanistan Reduce nuclear, Build the Increase security biological, and infrastructure to cooperation to Develop future U.S. Purpose chemical WMD risk provide for security support U.S.
|
From page 37... ...
A few examples may provide useful lessons for the DoD CTR programs. The field of global public health assistance (see e.g., Unite for Sight, 2010)
|
From page 38... ...
Consider the Chemical Weapons Elimination program. Years were spent planning and constructing the chemical weapons destruction facility.
|
From page 39... ...
Any of several decision-making or prioritizing frameworks would work, including the decision analysis technique of swing-weight analysis and the DoD capabilities based planning process. As noted in Chapter 2, DoD developed each set of metrics ab initio, using a different approach for each program, i.e.
|
From page 40... ...
A key technique for prioritizing metrics is the swing weight matrix (Parnell et al., 2011) , which can and has been used as the prioritization tool in a capabilities based planning exercise.
|
From page 41... ...
The levels of importance and variation can be thought of as constructed scales that have sufficient clarity to allow program managers to uniquely place every metric in one of the cells. Continuing from the discussion of priorities in Chapter 2, an illustrative swing weight matrix for the CBEP program is displayed in Table 3-3.
|
From page 42... ...
Other easily collected and compiled border security metrics include: • miles of pedestrian and vehicle fencing • number of border patrol agents • numbers of various classes of equipment deployed such as unmanned aircraft systems, remote video surveillance systems, mobile surveillance systems, unattended ground sensors 28 See Parnell et al.
|
From page 43... ...
In addition, questions have been raised about whether the "operational control" measure properly reflects the overall objectives of border security. In Congressional testimony, Border Patrol Chief Michael J
|
From page 44... ...
The DoD Metrics Report mentions that "one way to measure the effectiveness of enhancements objectively is through testing; that is, standardized exercises that can be conducted before and after enhancements are made to measure the impact of those enhancements." In addition, the Report presents a set of more detailed project-level metrics that are intended to capture a wide range of specific impacts or effects in relation to a detailed hierarchy of goals and objectives. The committee recognizes the difficulty of establishing a compelling overall effectiveness metric.
|
From page 45... ...
Table 3-4 provides the committee's tally of the CBEP metrics (the DoD Metrics Report TABLE 3-4 Assessment of the Cooperative Biological Engagement Program Metrics Partner Partner Capability Domestic Partner Total Stability Outcome Process Output (Environment) Input (Existence)
|
From page 46... ...
• The presented structure in the DoD Metrics Report that suggests moving from program objective to capability is too cumbersome because the capabilities being developed through a CBEP program may support multiple objectives. This causes metrics to be Box 3-3 Acquisition and Metrics DoD has conducted the CTR Program as an acquisition program: DoD contracts with companies to implement CTR plans.
|
From page 47... ...
The committee's overall assessment is that the CBEP metrics could be reasonably assessed with about 11 metrics instead of the 49 metrics listed in the DoD Metrics Report. The 11 metrics would be: 2 input (budgets)
|
From page 48... ...
Although the DoD acquisition process establishes many of the requirements for the implementation of CTR programs, the DoD Metrics Report does not make clear the relationship between the current metrics and the DoD acquisition process. For example, it is not clear whether DoD will use the IOC and the Full Operational Capability (FOC)
|
From page 49... ...
These table-top and field exercises were used as training events. The training exercises were conducted in a nonthreatening manner, so that the host country participants could gain experience and confidence in the use of the new infrastructure (e.g., biological safety level-2 laboratories and secure repositories for especially dangerous pathogens)
|
From page 50... ...
should be similar, if the question is how well a country is safely and securely responding to especially dangerous pathogens. Exercises under an expanded CTR Program could be similar in concept to what was previously tested in the FSU, but it is likely that DTRA will only be funding discrete parts of the disease surveillance system or supplementing existing programs.
|
From page 51... ...
, clearer planning for how changes and metrics results will feed into decision making will make the metrics more credible and useful for both DoD and the partner country. As mentioned in Chapter 2, the DoD Metrics Report deliberately does not consider future missions or changes in objectives,31 and although it notes that programs may change and expand to new geographic areas, some sections of the DoD Metrics Report make CTR appear to be a static program.
|
From page 52... ...
Some aspects of this evolution are already evident at least implicitly in the metrics described in the DoD Metrics Report, for example, the increased emphasis on sustainability and on partner country follow-through. The committee believes this concept of evolution of metrics should be more explicitly considered in future iterations of CTR metrics so that the progression of CTR activities through the successive phases would be more visible and prominent.
|
From page 53... ...
The key point is that for CTR programs there is often a natural progression of stages from conception to completion, and natural metrics for each stage. Although details may differ from program to program, a series of questions often arise at each stage: • What is the scope of the threat reduction challenge?
|
From page 54... ...
funding stops? -- are critically important for the CTR programs, particularly the capacity building programs.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.