4
Vision Setting and the
Pre-Application Process
The implementation of a research program requires establishing a vision for the program that focuses on the needs of its constituents as well as a process for identifying the research that is most likely to achieve that vision. For Congressionally Directed Medical Research Programs (CDMRP), these two activities are undertaken by programmatic panels. As described in Chapter 3, programmatic panels are composed of scientists, clinicians, consumers, and program managers from other Department of Defense (DoD) agencies or external organizations with experience in the relevant health condition(s).
This chapter begins with a discussion of the stakeholders meeting that occurs at the beginning of each new program and helps establish the mission and goals of the new research program. The chapter then focuses on the processes that a programmatic panel uses to establish an annual vision and investment strategy for the research program. The programmatic panel’s activities with regard to reviewing applications are discussed in Chapter 6 because they occur after the peer review process (discussed in Chapter 5) is complete. The annual investment strategy identifies the award mechanisms to be used for each research program; the award mechanisms result in program announcements that drive the application process (see Figure 4-1). In this chapter, the “vision setting” process is considered in terms of the activities and deliverables that occur prior to, during, and immediately after the vision setting meeting.
STAKEHOLDERS MEETINGS
When congressional appropriations are received for a new research program, CDMRP holds a stakeholders meeting prior to the start of the review process (see Figure 4-1). The purpose of the stakeholders meeting is to “survey the research landscape and identify gaps in both the scientific and consumer interest areas” (CDMRP, 2015a). A stakeholders meeting is usually held only once—at the start of a new research program. Additional stakeholders meetings are held infrequently and typically are “state of the science” meetings (Salzer, 2016b).
When a new research program is initiated, the CDMRP program manager assigned to the new program initiates the program process (with the help of the programmatic review contractor) by identifying, recruiting, and convening a group of stakeholders in that health area (Salzer, 2016b). The stakeholders include researchers from all types of organizations (e.g., academia and industry), clinicians, consumers (that is, people affected by the health condition and their relatives or caregivers), and representatives from other federal agencies and nongovernmental organizations that may
* As needed.
SOURCE: Adapted from Salzer, 2016a.
fund or conduct research on the health condition. The size of the stakeholders group varies by program.
The committee heard at its second public session (held on March 3, 2016) that stakeholders participate in the initial meeting only by invitation from CDMRP (or its contractor). Thus, the meetings are not open to the general public, they are not publicly announced, and there is no mechanism to receive any input from the public prior to, during, or after the stakeholders meeting. The committee finds this exclusivity to be an area of concern as it may preclude CDMRP from hearing from the broadest number of stakeholders, including individuals who may have contrary or unusual perspectives or input.
At the meeting, presentations are made on the congressional intent when initiating the program, followed by discussions of the current research landscape and knowledge gaps and existing federal and nonfederal research initiatives (Davey, 2016). All meeting discussions are captured in minutes of the meeting, but no decisions on the direction of the program are made (Katz, 2012). The meeting minutes are used to inform the subsequent vision setting process as described later in this chapter.
The intended goals of the stakeholders meeting are to develop a clear idea of how the program should evolve and to facilitate the vision setting meeting that follows (CDMRP, 2015a). The meeting may also result in a better understanding of the state of the science and an identification of knowledge gaps (Salzer, 2016b). A stakeholders meeting is not considered by CDMRP to be subject to the Federal Advisory Committee Act (FACA) of 1972 (Public Law 92-463), because it “does not form a formal advisory panel and is for information gathering only and therefore it is not covered by the FACA” (Salzer, 2016b). The program manager may use the meeting to identify potential members for the programmatic and peer review panels and suggest them to the respective contractors.
THE VISION SETTING PROCESS
For each annual funding cycle, the programmatic panel begins its work with a vision setting meeting to assess the mission and goals of the program and to define an investment strategy for that year. The investment strategy may be modified each year in response to scientific advances, patient care needs, and congressional funding.
The CDMRP provided the committee with 10 vision setting booklets for 6 research programs. These booklets—along with the investment strategies that they help inform—delineate the short-term goals of the program, the research gaps to be addressed, congressional intent, research initiatives and funding in other organizations, the award mechanisms to be used to achieve program goals, and the amount of funding to be
allocated for each award mechanism. For example, in 2013 the Prostate Cancer Research Program focused on five broad areas of emphasis: resources, innovation, impact, health disparities, and training. Within each area of emphasis there can be multiple award mechanisms, such as the Prostate Cancer Pathology Resource Network Award and Clinical Consortium Award in the resources area. Some programs such as the Autism Research Program do not specify areas of emphasis when selecting award mechanisms.
On the basis of the investment strategy, the CDMRP program manager then develops a program announcement for each recommended award mechanism with “consideration being given to avoiding duplicative research funding by other agencies” (Salzer, 2016a). The activities that occur prior to, during, and after the vision meeting are described in this section.
Pre-Meeting Activities
Prior to the vision setting meeting, the CDMRP program manager needs to understand the intent of the annual congressional appropriation language that funds—and, in some cases, specifies—the scientific focus of the research program (e.g., that the Prostate Cancer Research Program should fund imaging studies). The program manager may also need to query congressional staff for clarification of that language (Salzer, 2016a). Once the intent of the congressional language is understood, the program manager, with the help of the programmatic panel contractor, initiates a review of the current scientific landscape for the health condition. Information gathering may include searching the published literature, attendance at interagency meetings, discussions with the panel chair and panel members, and consultations with other organizations, both governmental (e.g., the National Institutes of Health) and nongovernmental (e.g., the American Cancer Society). The purpose of this review is to help inform the panel about those organizations’ research activities, the gaps in their research portfolios, and emerging areas of scientific interest.
CDMRP reports that prior to the vision setting meeting, the CDMRP program manager and the programmatic panel contractor conduct a teleconference with the chair of the panel. The program manager and panel chair discuss topics and ideas to be covered during the meeting and award mechanisms to be considered by the panel. The chair and other panel members may also be asked to recommend speakers for the meeting. The program manager may reach out to leaders in the field to request presentations to the programmatic panel during the vision setting meeting (Salzer, 2016b).
In its responses to committee questions, CDMRP stated that the pro-
gram manager may request that the contractor survey the programmatic panel members to collect information about the research knowledge gaps and funding landscape. Information from these surveys may be included in the vision setting booklet and discussed at the meeting to inform the investment strategy (Salzer, 2016c). The committee was not provided with any prototype surveys or sample responses from panel members.
Information gathered from all the above sources is included in the vision setting booklet for each CDMRP research program. The booklet, prepared annually by the program manager, is provided to the programmatic panel members at least 1–2 weeks in advance of the meeting, preferably earlier. Although the vision setting booklets are not available on the CDMRP website for any of the research programs, CDMRP provided several examples to the committee. There is no required format for these documents as they are designed to address the specific needs of each research program and thus are highly variable. These booklets may provide information on prior and current program funding, the short- and long-term goals of the program, career development needs for researchers, and considerations for funding future research to address program goals (Salzer, 2016a). Information on funding strategies for organizations that are not represented on the programmatic panels may also be included in the vision setting booklet; for example, the 2016 vision setting booklet for the Spinal Cord Injury Research Program included information on research funding around the world (SCIRP, 2016). Thus, most program managers rely on panel members, literature searches, and interagency contacts to learn about new developments in the field (Salzer, 2016c).
The CDMRP science officer, who is responsible for award management for the research program, may present the research program’s portfolio and answer questions about the past performance of funded awards (Salzer, 2016b). Although the vision setting booklet and presentations from the science officer may include information on current and past research efforts, the committee is uncertain of how the results of earlier CDMRP grants are tracked and disseminated and how this information is used by the panel to inform its investment strategy. The committee also notes that there does not appear to be a standardized method for ensuring that the panel members are seeing the most relevant or comprehensive information on developments in the field, particularly the results of CDMRP awards.
The Vision Setting Meeting
The vision setting meeting for the upcoming fiscal year occurs after the programmatic panel completes its review of the previous year’s applications. Outgoing programmatic panel members depart the panel after
the programmatic review is complete and new members join when the vision setting meeting begins. The vision setting meeting may take only a few hours or up to a full day or more. Vision setting meetings are nearly always held in person.
At the vision setting meeting, the CDMRP program manager briefs the panel on the goals of the meeting, including a review of the research program’s vision and mission statements, the current award portfolio, and information on the program announcements, such as an estimation of the number of anticipated awards per mechanism, and application criteria (Salzer, 2016b). The panel may also hear presentations from researchers in the field or from other research funding organizations to help clarify what new research is being undertaken and where research gaps remain.
Each CDMRP research program has a different investment strategy to fit the needs of that program. At the vision setting meeting, the chair leads the discussion on investment strategies, focus areas, funding landscapes and award mechanisms to inform the identification of research gaps and opportunities. There are many award mechanisms available to the panels for addressing a variety of research needs, but if none appear to be appropriate for a panel’s investment strategy, it may design a new one. Examples of award mechanisms range from the Ovarian Cancer Research Program’s Clinical Translation Award, which supports studies associated with ongoing or completed clinical trials, to the Reconstructive Transplant Research Program’s Discovery Award that is intended to “support innovative scientific approaches to regenerative medicine research that may be untested but may reveal breakthroughs or new avenues” (CDMRP, 2015e).
In response to committee questions about what guidance is given to programmatic panel members to help them identify “innovative” and “high-impact” research, CDMRP replied:
Innovation and high impact concepts are, by definition, specific to the different research programs. What may be innovative or high impact in one disease type or condition may not be important or critical in another disease type or condition. In program announcements such as the Amyotrophic Lateral Sclerosis Research Program (ALSRP) Therapeutic Idea Award, the terms innovation and impact are defined as to assist the applicants, the peer reviewers, and the programmatic reviewers. Research deemed innovative may introduce a new paradigm, challenge current paradigms, introduce novel concepts or technologies, or exhibit other uniquely creative qualities that may lead to potential therapeutics for Amyotrophic Lateral Sclerosis (ALS). Research must also have the potential for a major impact and accelerate progress toward treatment for ALS. The impact may be near-term or long-term, but must be significant and move beyond an incremental advancement. (Salzer, 2016b)
The committee received no information on how programmatic panel members are trained to define or operationalize these concepts.
Other program goals may include enhancing the number or expertise of researchers in a particular field or addressing health disparities. These goals might be achieved, for example, through award mechanisms that specifically encourage applications from junior investigators and through awards that address health disparities. Examples of specific award mechanisms in this area include the Career Development, Pre- or Post-doctoral Research Awards, the New Investigator Award, and the Health Disparities Research Award.
Post-Meeting Activities and Deliverables
After the investment strategy recommendations are developed by the programmatic panel, they are reviewed by the CDMRP program manager and the U.S. Army Medical Research Acquisition Activity (USAMRAA) contract officer to ensure that all outcomes such as the selection of award mechanisms follow DoD acquisition regulations. The investment strategy must also be reviewed and concurred with by the commanding general of the U.S. Army Medical Research and Materiel Command (Salzer, 2016b).
The CDMRP program manager, with support from the contractor, may create a pre-announcement for posting on the CDMRP website to inform the research community of the upcoming funding opportunities. Prior to the release of the program announcements, the program manager anticipates the possible topic areas that might be covered and the number of pre-applications and applications that might be received in response to the new program announcements in order to help guide the selection of reviewers. The program manager (again with contractor support) also writes and posts the program announcements (or broad agency announcements) and posts the list of programmatic panel members on the CDMRP website along with any new versions of the vision and mission statements for the program.
After the meeting, the program manager may also conduct an assessment of the panel members to ensure that there are no issues or gaps in representation. This assessment is given to the contractor for action (Salzer, 2016b). The committee received no further information on the nature of the assessment or on whether the information is used to inform the subsequent selection and training of programmatic panel members or on the speed by which any concerns are addressed.
Program Announcements
Program announcements—or broad agency announcements, depending on the program—along with the general application instructions provide guidance on the preparation and submittal of pre-applications and applications. CDMRP has a template for program announcements, including the content instructions and the timeframe for release, submissions, and review. Interested individuals may sign up on the CDMRP website to receive notifications when program announcements are available.
The program announcement and the general application instructions, both of which are listed on the CDMRP website by research program, specify the intent and objectives of the program as well as the particular award mechanism, a description of the type of studies being requested, eligibility and submission requirements, and criteria for the review of pre-applications and for the peer and programmatic review of full applications (CDMRP, 2015a). The submission requirements in a program announcement depend on the specific award mechanism, but typically they include the project narrative, supporting documentation, a technical abstract, a lay abstract, a statement of work, an impact statement, a submissions statement, a transition plan, a key person profile, and the budget.
The CDMRP program manager is responsible for the entire program announcement creation process, from the award-specific language as recommended by the programmatic panel through approval. These internal CDMRP processes include writing, revising, and editing through several stages of review that involve the integrative program team (see Chapter 3), other program managers and leaders, the Defense Health Agency, the judge advocate general, and USAMRAA. The program announcement milestones are determined by the program manager and include the pre-application receipt date, the date when the applicants will be informed of their invitation status, the full application receipt date, and the dates for the peer and programmatic review (Salzer, 2016b). After the reviews are complete, the program announcement is released on Grants.gov by the USAMRAA grants officer.
Landscape Documents
Two CDMRP programs—the Breast Cancer Research Program and the Ovarian Cancer Research Program—have developed landscape documents to help refine the criteria in the program announcements; as of August 2016, only the breast cancer document was available on the CDMRP website. These landscape documents provide a synopsis of the available epidemiologic data on morbidity and mortality as well as a summary of new diagnostic and treatment advances for those cancers. CDMRP stated that the research programs may project beyond the current
NOTE: HBCU/MI = historically black colleges and universities/minority institutions.
SOURCE: Vaday, 2016.
year to encompass broader themes such as those shown in Figure 4-2 for the Breast Cancer Research Program. The committee notes that this figure, which highlights research gaps, priorities, and funding opportunities (or award mechanisms) since the program’s inception, is not included in the program’s landscape document. The committee believes that including this figure in the landscape document would be useful for showing research opportunities and gaps since inception of the program and would help in developing an investment strategy. The committee further finds that only 2 of the 29 programs have developed landscape documents, that these documents vary in content, and that only one is available publicly. The committee has been informed, however, that the Gulf War Illness Research Program is in the process of developing a similar document which will be posted on the research program’s website (Salzer, 2016c) and that several other research programs are considering or are in the process of developing similar documents (Salzer, 2016d).
PRE-APPLICATION RECEIPT AND SCREENING
Once the programmatic panel has established an investment strategy and the program announcement for each award mechanism has been publicly released on the CDMRP website, pre-applications are received by CDMRP via the electronic biomedical research application portal (eBRAP). The programmatic panel is responsible for screening all pre-applications. Most CDMRP research programs require the submission of
a pre-application (Salzer, 2016b). Depending on the award mechanism, some pre-applications require relatively minimal submission materials such as a letter of intent, and applicants may submit full applications before the deadline indicated on the program announcement. Other award mechanisms require more extensive pre-application materials and have screening criteria for their review. In 2014, more than 11,000 pre-applications were received across 23 research programs, ranging from 8 for the Alcohol and Substance Abuse Disorders Research Program to 2,314 for the Breast Cancer Research Program (Salzer, 2016c).
CDMRP states that the advantages of having a pre-application step include a decrease in management costs, the ability to screen applications to determine whether they fit the intent of the award mechanism, fewer principal investigators who apply with little chance of receiving funding, a smaller number of applications to be reviewed by the peer review and programmatic panels (particularly for programs that typically receive a large number of applications), and a higher overall rate of funding of award mechanisms (Salzer, 2016b). The specific submission requirements and screening criteria for a pre-application are given in the program announcement. In general, pre-applications outline the research aims, strategy, innovation, and impact of the proposed project (Salzer, 2016c).
CDMRP reports that the program manager and peer review support contractor review each pre-application in eBRAP for compliance with the submission criteria (e.g., narrative within the page limit), but not for scientific content. At this stage, the pre-applications are also screened for any conflict of interest (COI) issues. Any programmatic panel member who is named in a pre-application (or application) other than in a cited reference is blocked from screening the pre-application in the eBRAP database and will be recused during the discussion of that pre-application at the meeting (Salzer, 2016b). All COIs are reported to the program manager, who assesses them and makes recommendations to the USAMRAA grants officer, who makes the final decision on whether a pre-application needs to be withdrawn. The program manager also provides this list to the contractor to manage COIs for pre-application screening and review (Salzer, 2016b). Applicants may appeal the grants officer’s decision for a number of reasons, such as changes in investigator or organization, but only appeals that follow the instructions laid out in the program announcement and general application instructions are permitted by the USAMRAA grants office (Salzer, 2016b).
Following the compliance and COI reviews by the program manager and peer review contractor, the programmatic panel conducts the pre-application screening. The programmatic panel support contractor, in consultation with the program manager, assigns each pre-application a primary and secondary reviewer from among the programmatic panel
members. Panel members screen the pre-applications on the basis of the criteria specified in the program announcement and make a preliminary determination as to whether a pre-application meets the criteria and the investigator should be invited to submit a full application.
After the pre-applications have been screened, the programmatic panel meets via a teleconference (or, more rarely, via an onsite meeting) to discuss and recommend the pre-applications that should be invited for a full application submission (Salzer, 2016b). During this meeting, the program manager may brief the panel on the program announcements as well as review the program’s history, award intent, focus areas, and portfolio composition, along with key elements of the pre-applications and their screening criteria (Salzer, 2016b). The program manager may also suggest the number of pre-applications that the panel might recommend be invited to submit full applications (Salzer, 2016b). Based on the discussion of each pre-application, the programmatic panel members make recommendations on whether an applicant should be invited to submit a full application.
A list of “to be invited” and “not to be invited” pre-applications is prepared by the contractor, approved by the program manager, and posted in the eBRAP system; invitation letters are subsequently sent to the selected applicants (Salzer, 2016b). In 2014, of the approximately 11,000 pre-applications that were screened, almost 6,000 full applications were received across all the research programs (see Figure 4-3). The committee
SOURCE: Salzer, 2016d.
finds that applicants who are not invited to submit full applications are not informed about why their application did not meet the screening criteria for full submission, nor are they permitted an opportunity to address the pre-application’s shortcomings and resubmit.
