National Academies Press: OpenBook
« Previous: References
Suggested Citation:"Appendix A: Letter Report of the Panel to the Army Test and Evalution Command." National Research Council. 2003. Improved Operational Testing and Evaluation: Better Measurement and Test Design for the Interim Brigade Combat Team with Stryker Vehicles: Phase I Report. Washington, DC: The National Academies Press. doi: 10.17226/10710.
×
Page 99
Suggested Citation:"Appendix A: Letter Report of the Panel to the Army Test and Evalution Command." National Research Council. 2003. Improved Operational Testing and Evaluation: Better Measurement and Test Design for the Interim Brigade Combat Team with Stryker Vehicles: Phase I Report. Washington, DC: The National Academies Press. doi: 10.17226/10710.
×
Page 100
Suggested Citation:"Appendix A: Letter Report of the Panel to the Army Test and Evalution Command." National Research Council. 2003. Improved Operational Testing and Evaluation: Better Measurement and Test Design for the Interim Brigade Combat Team with Stryker Vehicles: Phase I Report. Washington, DC: The National Academies Press. doi: 10.17226/10710.
×
Page 101
Suggested Citation:"Appendix A: Letter Report of the Panel to the Army Test and Evalution Command." National Research Council. 2003. Improved Operational Testing and Evaluation: Better Measurement and Test Design for the Interim Brigade Combat Team with Stryker Vehicles: Phase I Report. Washington, DC: The National Academies Press. doi: 10.17226/10710.
×
Page 102
Suggested Citation:"Appendix A: Letter Report of the Panel to the Army Test and Evalution Command." National Research Council. 2003. Improved Operational Testing and Evaluation: Better Measurement and Test Design for the Interim Brigade Combat Team with Stryker Vehicles: Phase I Report. Washington, DC: The National Academies Press. doi: 10.17226/10710.
×
Page 103
Suggested Citation:"Appendix A: Letter Report of the Panel to the Army Test and Evalution Command." National Research Council. 2003. Improved Operational Testing and Evaluation: Better Measurement and Test Design for the Interim Brigade Combat Team with Stryker Vehicles: Phase I Report. Washington, DC: The National Academies Press. doi: 10.17226/10710.
×
Page 104

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix A Letter Report of the Pane! to the Army Test and Evaluation Command

THE NATIONAL ACADEMIES Advisers to the Nation on Science, Engineering, and Medirine Division of Behavioral and Social Sciences and Education Committee on National Statistics Panel on Operational Test Design and Evaluation of the Interim Armored Vehicle (IAV) Frank John Apicella Technical Director Army Evaluation Center U.S. Army Test and Evaluation Command 4501 Ford Avenue Alexandria, VA 22302-1458 Dear Mr. Apicella: 500 Fifth Street, NW Washington, DC 20001 Phone: 202 334 3408 Fax: 202 334 3584 Email:jmcgee~ nas.edu October 10, 2002 As you know, at the request of the Army Test and Evaluation Com- mand (ATEC) the Committee on National Statistics has convened a panel to examine ATEC's plans for the operational test design and evaluation of the Interim Armored Vehicle, now referred to as the Stryker. The panel is currently engaged in its tasks of focusing on three aspects of the operational test design and evaluation of the Stryker: (1) the measures of performance and effectiveness used to compare the Interim Brigade Combat Team (IBCT), equipped with the Stryker, against a baseline force; (2) whether the current operational test design is consistent with state-of-the-art meth- ods in statistical experimental design; and (3) the applicability of models for combining information from testing and field use of related systems and from developmental test results for the Stryker with operational test results for the Stryker. ATEC has asked the panel to comment on ATEC's current plans and to suggest alternatives. The work performance plan includes the preparation of three reports: · The first interim report (due in November 2002) will address two 101

102 IMPROVED OPERATIONAL TESTING AND EVALUATION topics: (1) the measures of performance and effectiveness used to compare the Stryker-equipped IBCT against the baseline force, and (2) whether the current operational test design is consistent with state-of-the-art methods . . . . . In statlstlca . experlmenta . ( ~eslgn. · The second interim report (due in March 2003) will address the topic of the applicability of models for combining information from test- ing and field use of related systems and from developmental test results for the Stryker with operational test results for the Stryker. · The final report (due in luly 2003) will integrate the two interim reports and add any additional findings of the panel. The reports have been sequenced and timed for delivery to support ATEC's time-critical schedule for developing plans for designing and imple- menting operational tests and for performing analyses and evaluations of the test results. Two specific purposes of the initial operational test of the Stryker are to determine whether the Interim Brigade Combat Team (IBCT) equipped with the Stryker performs more effectively than a baseline force (Light In- fantry Brigade), and whether the Stryker meets its performance require- ments. The results of the initial operational test contribute to the Army's decisions of whether and how to employ the Stryker and the IBCT. The panel's first interim report will address in detail factors relating to the effec- tiveness and performance of the Stryker-equipped IBCT and of the Stryker; effective experimental designs and procedures for testing these forces and their systems under relevant operational conditions, missions, and scenarios; subjective and objective measures of performance and effectiveness for cri- teria of suitability, force effectiveness, and survivability; and analytical pro- cedures and methods appropriate to assessing whether and why the Stryker- equipped IBCT compares well (or not well) against the baseline force, and whether and why the Stryker meets (or does not meet) its performance requirements. In the process of deliberations toward producing the first interim re- port that will address this broad sweep of issues relevant to operational test design and to measures of performance and effectiveness, the panel has discerned two issues with long lead times to which, in the opinion of the panel, ATEC should begin attending immediately, so that resources can be identified, mustered, and applied in time to address them: early develop- ment of a "straw man" (hypothetical draft) Test and Evaluation Report (which will support the development of measures and the test design as

APPENDIXA: LETTER REPORT 103 well as the subsequent analytical efforts) and the scheduling of test partici- pation by the Stryker-equipped force and the baseline force so as to remove an obvious test confounder of different seasonal conditions. The general purpose of the initial operational test (JOT) is to provide information to decision makers about the utility of and the remaining chal- lenges to the IBCT and the Stryker system. This information is to be generated through the analysis of IOT results. In order to highlight areas for which data are lacking, the panel strongly recommends that immediate effort be focused on specifying how the test data will be analyzed to address relevant decision issues and questions. Specifically, a straw man Test Evalua- tion Report (TER) should be prepared as if the IOT had been completed. It should include examples of how the representative data will be analyzed, specific presentation formats (including graphs) with expected results, in- sights one might develop from the data, draft recommendations, etc. The content of this straw man report should be based on the experience and intuition of the analysts and what they think the results of the IOT might look like. Overall, this could serve to provide a set of hypotheses that would be tested with the actual results. Preparation of this straw man TER will help ATEC assess those issues that cannot be informed by the opera- tional tests as currently planned, will expose areas for which needed data is lacking, and will allow appropriate revision of the current operational test plan. The current test design calls for the execution of the IBCT/Stryker vs. the opposing force (OPFOR) trials and the baseline vs. the OPFOR trials to be scheduled for different seasons. This design totally confounds time of year with the primary measure of interest: the difference in effectiveness between the baseline force and the IBCT/Stryker force. The panel believes that the factors that are present in seasonal variations weather, foliage density, light level, temperature, etc. may have a greater effect on the differences between the measures of the two forces than the abilities of the two forces themselves. We therefore recommend that serious consideration be given to scheduling these events as closely in time as possible. One way to address the potential confounding of seasonal affects, as well as possible effects of learning by blue forces and by the OPFOR, would be to inter- sperse activities of the baseline force and the IBCT/Stryker force over time. The panel remains eager to assist ATEC in improving its plans and processes for operational test and evaluation of the IBCT/Stryker. We are grateful for the support and information you and your staff have consis- tently provided during our efforts to date. It is the panel's hope that deliv-

104 IMPROVED OPERATIONAL TESTING AND EVALUATION Bring to you the recommendations in this brief letter in a timely fashion will encourage ATEC to begin drafting a straw man Test Evaluation Report in time to influence operational test activities and to implement the change in test plan to allow the compared forces to undergo testing in the same season. Sincerely yours, Stephen Pollock, Chair Panel on Operational Test Design and Evaluation of the Interim Armored Vehicle

Next: Appendix B: Force Exchange Ratio, Historical Win Probability, and Winning with Decisive Force »
Improved Operational Testing and Evaluation: Better Measurement and Test Design for the Interim Brigade Combat Team with Stryker Vehicles: Phase I Report Get This Book
×
Buy Paperback | $43.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The U.S. Army Test and Evaluation Command (ATEC) is responsible for the operational testing and evaluation of Army systems in development. ATEC requested that the National Research Council form the Panel on Operational Test Design and Evaluation of the Interim Armored Vehicle (Stryker) to explore three issues concerning the initial operation test plans for the Stryker/Interim Brigade Combat Team (IBCT). First, the panel was asked to examine the measures selected to assess the performance and effectiveness of the Stryker/IBCT in comparison both to requirements and to the baseline system. Second, the panel was asked to review the test design for the Stryker/IBCT initial operational test to see whether it is consistent with best practices. Third, the panel was asked to identify the advantages and disadvantages of techniques for combining operational test data with data from other sources and types of use. In this report the panel presents findings, conclusions, and recommendations pertaining to the first two issues: measures of performance and effectiveness, and test design. The panel intends to prepare a second report that discusses techniques for combining information.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!