THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
Computer Science and Telecommunications Board
June 7, 2004
The Honorable Robert S. Mueller III
Federal Bureau of Investigation
935 Pennsylvania Avenue, NW Washington, DC 20535
Dear Director Mueller:
This letter provides comments and observations regarding the briefing provided on May 20, 2004, by the FBI to the National Research Council’s Committee on the FBI’s Trilogy Information Technology Modernization Program. The committee was provided with an update on recent FBI activities with respect to the Trilogy program and other applications.
The May 20 briefing was necessarily limited in scope. This briefing contained information not previously presented to the committee when it prepared its original report, and focused on the period since the beginning of 2004. The committee saw in what the FBI presented evidence that the FBI has made significant progress on a number of fronts in the last 5 or so months, such as the development of key plans and management approaches for information technology (IT) systems and the deployment of a functional prototype of an investigative data warehouse with significant operational utility. These actions are well aligned with recommendations made by the committee in its original report. In other areas, the briefing of May 20 about FBI plans and intentions since delivery of the original report were reassuring. At the same time, the committee recognizes that the FBI has many remaining challenges before it in the IT area and that it will take some time to see the results from the many IT plans that the FBI has recently put into place. The committee looks forward to seeing the FBI follow through with implementation of these plans. Finally, the committee was impressed that the FBI has hired a chief information officer who is aware of and responsive to the need for strong leadership, and also sensitive to the FBI’s mission and to the many complex issues of design, procurement, implementation, and management of its information technology systems.
On May 10, 2004, the National Research Council (NRC) report entitled A Review of the FBI’s Trilogy Information Technology Modernization Program (hereafter referred to as “the original report” and available at http://books.nap.edu/catalog/10991.html) was delivered to FBI management and a briefing of the study committee’s findings, conclusions, and recommendations was given to you, the chief information officer, and some other members of your team. As noted in the report, the committee’s findings and conclusions were based primarily on FBI briefings to the NRC’s Committee on the FBI’s Trilogy Information Technology Modernization Program on October 27-28, 2003, and December 15-16, 2003. Shortly after the report was delivered to the FBI and the Congress and was publicly released, FBI
management advised the NRC that there had been substantial progress on a number of the issues outlined in the report since FBI management and staff had last briefed the committee, and requested that the committee review this new information.
As a result, on May 20, 2004, the committee convened for an update briefing conducted by FBI Chief Information Officer (CIO) Zalmai Azmi, Executive Assistant Director Wilson Lowery, and Sherry Higgins, the Assistant Director for FBI Program Management. The briefing dealt with items that were included in the five areas of concern identified in the original report: (1) the transition from the Automated Case Support system to the Virtual Case File, (2) enterprise architecture, (3) system design, (4) program and contract management, and (5) human resources. This letter report summarizes the May 20 briefing and presents the committee’s commentary on it.
The next five sections describe and comment on what was covered in the May 20 briefing relative to the five areas of focus of the original report.
THE CONVERSION FROM THE AUTOMATED CASE SUPPORT (ACS) SYSTEM TO THE VIRTUAL CASE FILE (VCF)
In its original report, the committee recommended that the FBI refrain from undertaking a “flash cutover” conversion from the Automated Case Support (ACS) system to the Virtual Case File (VCF), and that a validated contingency plan for capability to revert partially or completely to ACS be put into place before any major transition proceeds. Mr. Azmi, the CIO, told the committee on May 20 that he concurs with this view and that he is in the process of developing alternatives to the flash cutover conversion.
Observations and comments: The report that the FBI is developing alternatives to a flash cutover is consistent with the committee’s recommendations. Planning for such an alternative also underscores the importance of a backup capability if the VCF is to be deployed without substantial operationally testing. The committee recognizes that backup capabilities will necessitate some additional cost, but considers the cost a good investment relative to the possible catastrophic consequences should an inadequately tested VCF fail during initial operational use. The committee further notes that if the ACS system does not remain available for use throughout the VCF phase-in period including an incremental phase-in of VCF functions, then the VCF implementation is effectively a flash cutover.
In its original report, the committee noted that the success of the FBI's information technology efforts will require the development of a close linkage between IT and a coherent view of the bureau's mission and operational needs. The committee warned that the development of this strategic linkage—the enterprise architecture—cannot be delegated inside the bureau to the CIO or outside to contractors, and that only the senior leadership of the FBI can establish the policies, define the operational frameworks and priorities, and make the tradeoffs that are necessary to formulate this strategic view.
Against this backdrop, the committee urged the FBI to create an enterprise architecture (EA) as a matter of the highest priority, and that top FBI operational management be substantively engaged in the process of creating the EA. In the May 20 briefing, Mr. Azmi stated his personal commitment to the creation of the EA as a matter of high priority. In an important step toward obtaining authoritative operational input, he reported that he is reconstituting the Enterprise Architecture Board to include senior representatives of 11 operational divisions of the FBI. In addition, he reported that he intends to require signoff on the EA by the bureau’s executive assistant directors and division assistant directors. Finally, Mr. Azmi stated that he has engaged contractor assistance in the creation of the EA and that the contract includes specific
milestones and a specific timeline for defining the “as-is” EA as a baseline for implementing the new “to-be” EA.
Observations and comments: The reported steps to obtain significant operational input to and validation of the “as-is” enterprise architecture are aligned with the committee’s recommendations for proceeding with the development of an EA for the FBI. The committee notes that the need for senior operational leadership involvement, including that of the director, will increase significantly as the FBI addresses the “to-be” enterprise architecture, because it is the vision and policies of the operational leadership that must be incorporated in the “to-be” EA for full implementation of it to be successful. Development of an “as-is,” followed by a “to-be” enterprise architecture, will be major milestones for the FBI.
In its original report, the committee recommended that the FBI refrain from using the VCF as the primary foundation on which to build its analytical and data management capabilities for the intelligence processes supporting the counterterrorism mission. Rather, the committee said that the FBI should fashion an architecture for the counterterrorism mission from scratch, and then design explicit interfaces to the VCF when information must flow between them. The committee’s focus on this issue was driven by the fact that in the FBI’s October 2003 briefing to the committee, FBI managers represented the VCF as a system intended to support both the criminal investigation and counterterrorism missions of the FBI.
On May 20, 2004, Mr. Azmi emphasized that the VCF will be primarily a tool for workflow automation in the criminal investigative process, and additionally will serve as one of many “data feeds” into a system known as the Investigative Data Warehouse (IDW). The IDW provides single-query access to data in multiple databases (including data currently resident in the ACS system) and provides a portfolio of intelligence-oriented analytical tools (e.g., “entity extraction tools” to locate names, phone numbers, and other information in documents, or “relationship-tracking tools” to identify who is in contact with whom).
Mr. Azmi also addressed the issue of security briefly, stating that a security group is in place and is undertaking actions such as a red-team analysis of Trilogy security.
Observations and comments: The concept of the IDW as related to the committee on May 20 is consistent with the spirit of the committee’s recommendation for separate but linked enterprise subarchitectures. The committee was not briefed on May 20 about progress on the subarchitecture for the counterterrorism mission and urges continued attention to the creation of such an architecture along the lines of its recommendations in the original report. Within the limits of a short demonstration of the IDW to the committee on May 20, the IDW appears to provide some of the key capabilities necessary for intelligence use. More detailed examination and review would be needed to determine the extent and nature of enhancements to today’s IDW that are needed to serve effectively the full set of operational needs for intelligence analysis.1
The FBI managers providing the May 20 briefing on the IDW noted that internal demand for the capabilities provided today by the IDW was strong and that the current user base of 500 is expected to grow rapidly in the future. The committee believes that if growth in demand
continues, the IDW will have met an important internal “market” test that it can serve important analytical needs in both the counterterrorism and criminal investigation missions.
Regarding security, the committee believes that the FBI has paid appropriate attention to this subject. From the information made available in the October and December 2003 briefings, and again on May 20, 2004, the committee concluded that staffing and plans for security appeared sound. However, as with all security plans, the devil is in the details, and the committee urges the FBI to undertake comprehensive and detailed review of such plans (including technical, organizational, and legal dimensions) on a regular basis.
PROJECT AND CONTRACT MANAGEMENT
In its original report, the committee recommended that prototypes be developed and used before committing to large-scale development, and that adequate testing time must be allowed in development schedules even if deployment is delayed as a consequence. It also recommended that contracts with developers make use of functional specifications, specific milestones, frequent contract reviews, and “earned-value” metrics.
On May 20, Mr. Azmi reported to the committee the establishment of a new IT project management office, staffed largely by people transferred from the Information Resources Division of the FBI, and said that some of these individuals with substantial project management experience had been assigned as project managers for the VCF project. He also noted that communication with the senior management of key contractors is frequent, and he expressed a commitment to exploit methodologies such as “earned-value management” to enhance the return on IT investments and the pace of progress. Regarding testing, Mr. Azmi reported that the FBI intends to standardize a life cycle methodology for all IT projects within the FBI that will include adequate time for testing. Finally, Mr. Azmi indicated that the IDW application demonstrated to the committee on May 20 was based on a prototype that has now been in operation for a number of months.
Observations and comments: In the committee’s judgment, the management steps taken and in process—when fully implemented—will be important elements of an effective and restructured management plan for the Trilogy program and beyond. To the best of the committee’s knowledge, these elements were not in place in December 2003, and the committee concurs that these new steps, when fully implemented, are consistent with the committee’s recommendations.
Regarding the test program for the VCF, about which the committee had expressed concern in its original report, Mr. Azmi reported on May 20 that the FBI is developing a number of new options for how to proceed with the completion and deployment of the VCF, but he did not present to the committee a detailed test plan or schedule. The committee notes, however, that the dates projected for the VCF to achieve initial operating capability have been extended, so, at least from a schedule point of view, additional time seems to be available for testing. As indicated in the original report, the committee underscores the need for thorough testing of the VCF before its deployment, including operational testing by a limited number of users before the VCF is deployed across the bureau.
Regarding the use of prototypes in the development process, the committee believes that the apparent progress illustrated in the IDW demonstration to the committee on May 20 reflects the fact that the IDW is based on the use of prototypes. In October 2003, the committee was briefed on both the Secure Counterterrorism Operational Prototype Environment (SCOPE) and an early concept of the IDW. At that time, the plans for the IDW appeared poorly conceived, an observation included in the original report. On May 20, the committee learned that in mid-
December 2003, driven by pressing operational needs SCOPE and IDW were combined and placed into operational use, resulting in the current IDW product.
In its original report, the committee had recommended that the FBI significantly grow its internal expertise in IT and contract management, that the FBI fill the position of CIO with a permanent incumbent, and that it develop an improved system for internally reviewing the state of progress in key IT programs and for communicating relevant findings to key stakeholders. The FBI filled the position of CIO with a permanent incumbent—Mr. Azmi—in May 2004. (Mr. Azmi was named acting CIO on December 30, 2003). A May 6, 2004, FBI press release indicates that Mr. Azmi is responsible for the FBI's overall information technology efforts, including developing the FBI’s IT strategic plan and operating budget; developing and maintaining the FBI's technology assets; and providing technical direction for the reengineering of FBI business processes.2 The FBI has also appointed an acting chief technology officer, whose responsibilities are reported to focus initially on centralizing the bureau's current IT projects and ensuring the best way to share bureau intelligence data on common platforms.
In addition, Mr. Azmi and Mr. Lowery reported having taken a number of other steps, including working with the Department of Justice to prepare a package for submission to the Office of Personnel Management that would enable senior executive service members in a certified performance appraisal system to earn up to $201,600 (in 2004) in base salary plus performance bonuses; engaging the services of two executive recruitment firms to assist in the identification and recruitment of individuals for IT and other specialized areas; contacting chief executive officers of several private companies and senior managers of government agencies in an effort to identify individuals who might be borrowed by the FBI to assist with IT work; preparing a request to the Department of Justice for authority to establish senior-level positions for information technology; preparing a human capital plan that establishes a framework for FBI-wide human capital planning, recruitment, and development; and undertaking a reengineering initiative on workforce training and development that resulted in a 5-year plan for designing and implementing a training and development program for every job series within the FBI, including those in the technical and scientific areas.
Observations and comments: The committee believes that the appointment and subsequent leadership of Mr. Azmi have been key factors in creating a new environment for progress in several IT areas in the last 4 to 6 months. Mr. Azmi appears to have the knowledge and the insight needed to provide the strong direction required for the Trilogy program. Furthermore, the CIO now reports directly to the FBI director rather than to an executive assistant director, and Mr. Azmi reported on May 20 that he meets with the director daily. These changes indicate the director’s commitment to exploiting IT in support of the FBI’s missions, and the committee believes that the progress indicated in this letter report is based largely on the leadership of the CIO.
Beyond this strengthening of the senior ranks of information technologists within the FBI, progress in growing IT expertise has been mixed to date, and several efforts have been taken or are now under way in this area as described above. The committee supports these actions, even as it reiterates the importance of growing internal FBI expertise. To be successful in executing its information-intensive missions, the FBI must have a first-rate IT staff capability throughout the organization, even if significant use is made of external contractors.
Overall, the committee saw clear evidence of progress in the May 20 update briefing it received from the FBI. Whereas the committee’s original report concluded that the FBI was not on a path to success in its IT modernization program, a conclusion based on information received through December 2003, the update briefing identified a number of important steps taken by the FBI since last year that move the FBI ahead in its IT modernization program. Indeed, clear progress was visible in the appointment of a permanent CIO, the IDW demonstration to the committee, and the formation of a staffed program office for improved IT contract management. In other areas, the update briefing described recent plans and intentions to carry out many of the actions that the committee had recommended, such as a commitment to develop an alternative to the previously planned flash cutover. Given that such plans and intentions were apparently not in place in October and December 2003, their very existence today demonstrates important progress. Since December 2003, the FBI appears to have made progress at a more rapid rate than the committee would have expected from the information made available to it at that time.
At the same time, the committee observes that many challenges remain. Many large organizations have learned that following an enterprise-wide plan through to successful completion is almost always more difficult than the initial development and articulation of the plan, important though the latter is. Sustained success in IT will require strong and forceful leadership over an extended period of time.
Furthermore, comprehensive management information flow is key to success in any organization, especially one undergoing significant change. In the course of the original review in late 2003, some of the presumably authoritative FBI briefers providing information to the committee seemed to be unaware of a number of important IT activities under way within the FBI and, hence, only recently were these activities reported to the committee. Integrating IT into the fabric of FBI operations will better prepare the bureau for the new challenges it faces in the 21st century, and the committee urges the FBI to establish effective bureau-wide communications pathways for IT strategies, plans, and efforts under way.
Finally, the committee wishes to emphasize some points raised by or implicit in the original report. The FBI’s missions constitute increasingly information-intensive challenges, and the ability to integrate and exploit rapid advances in IT capabilities will only become more critical with time. Consequently, even with perfect program management and execution, substantial IT expenses on an ongoing basis are inevitable and must be anticipated in the budget process if the FBI is to maximize the operational leverage that IT offers. Thus, the FBI’s budget overseers should not expect an automatic decrease in expenses for IT when the Trilogy program is completed.
The committee hopes that you find this letter report useful.
COMMITTEE ON THE FBI’s TRILOGY INFORMATION TECHNOLOGY MODERNIZATION PROGRAM
JAMES C. MCGRODDY, IBM (retired), Chair
EDWARD BALKOVICH, RAND
RICHARD BASEIL, Holmdel, New Jersey
MATT BLAZE, AT&T Laboratories
W. EARL BOEBERT, Sandia National Laboratories
MARC DONNER, Morgan Stanley
MICHAEL McGILL, Columbus, Ohio
JAMES NOGA, Massachusetts General Hospital
CARL O’BERRY, The Boeing Company
KEN ORR, The Ken Orr Institute
JAMES PATTON, The MITRE Corporation
MARK SEIDEN, MSB Associates
GEORGE SPIX, Microsoft Corporation
CHARLES E. STUART, Competitive Enterprise Solutions, LLC
GIO WIEDERHOLD, Stanford University
HERBERT S. LIN, Senior Scientist and Study Director
KRISTEN BATCH, Research Associate
DAVID DRAKE, Senior Project Assistant until November 2003