3

Participants in the Review Process

The major participants in the review process are Congressionally Directed Medical Research Programs (CDMRP) leadership and staff, peer review and programmatic contractors, and the peer and programmatic reviewers. In this chapter, the committee examines the roles of each of these participants.

CDMRP staff includes both scientific and administrative staff (see Figure 3-1). Scientific staff has managerial responsibilities, including oversight of contractor staff. CDMRP administrative staff include financial and contract managers, among others. The two contractors, one for peer review and one for programmatic review, handle many aspects of the review process, including recruiting reviewers, training them, identifying potential conflicts of interest (COIs), overseeing the electronic application database, and preparing meeting documentation.

Each research program has at least one peer review panel and a programmatic panel. Members of these panels represent broad and diverse areas of expertise and experience ranging from academia (both public and private institutions), government agencies (e.g., the Department of Defense [DoD], the National Institutes of Health [NIH], and the Department of Veterans Affairs [VA]), industry (e.g., pharmaceutical, bioengineering), and consumer and advocacy organizations. The chapter begins with a description and functions of CDMRP staff and contractors and then describes the qualifications, selection process, training, length of service, COIs, and compensation for both peer review and programmatic panel members.

CDMRP PERSONNEL

About 115 onsite DoD staff and 80 contractor staff manage the research portfolios at CDMRP (Colonel Wanda Salzer, CDMRP director, personal communication, October 16, 2015). The CDMRP director is a military officer, currently an Air Force colonel and medical doctor. Each research program is led by a CDMRP program manager, who may be a military officer or a DoD civilian employee and who is responsible for the operation and oversight of one or more research programs. New program managers are assigned a senior program manager who mentors or trains them and leads them through the program manager guide (the guide was not provided to the committee) (Salzer, 2016b). CDMRP reports that about 77% of the program managers manage more than one research program (Salzer, 2016b).

Program managers have many duties, including contractor tasking and the approval of contractor work products such as rosters of proposed reviewers, training materials, peer review summaries, and meeting minutes. Program managers are also responsible for evaluating potential COIs between reviewers and applicants, preparing background materials for vision setting, and liaison and outreach activities with other federal government organizations and nongovernmental organizations. They may interact with the grants officer representative and the science officers. As the committee was not charged with evaluating the financial aspects of the contractors or the award management, these personnel and activities are not considered further in this report.

CDMRP reports that it uses an “integrated program team” approach to manage each research program and facilitate communication about and coordination of program activities (see Figure 3-1). The integrated

program team, which meets regularly (every week or biweekly), is led by the CDMRP program manager and includes other CDMRP program and support staff, such as program analysts and science officers (who are responsible for managing contracts once they are awarded) (Salzer, 2016a).

CDMRP CONTRACTORS

As of 2016, CSRA, Inc. (formerly SRA, Inc.), is the CDMRP peer review contractor, and Leidos, Inc., is the programmatic review contractor. Both contractors work closely with the program managers as part of the integrated program teams. Both contractors are tasked with identifying, recruiting, and managing scientific and consumer reviewers for the review panels they support as well as managing the meetings and preparing pre- and post-meeting materials, such as minutes. The peer review contractor is responsible for maintaining the electronic biomedical research application portal (eBRAP) for application submissions and notifications and for helpdesk support for applicant questions regarding program announcements and application submissions (CDMRP, 2011).

Among the integrated program team members from the programmatic contractor is the technical coordinator. Although the programmatic contractor recruits consumer representatives for programmatic panels, it does not have a consumer review administrator as the peer review contractor does (Salzer, 2016d). CDMRP stated that because programmatic panel members serve for at least 2 years, recruitment, coordination, and training are not needed on a yearly basis for any one program, and thus a dedicated consumer review administrator is not necessary for a programmatic panel. The committee did not receive any further information on the requirements, roles, or expertise of the programmatic contractor staff (e.g., biomedical scientist support).

The integrated program team members from the peer review contractor include the science review manager, the project manager, and the consumer review administrator (see Figure 3-1). The science review manager is the contractor’s scientific lead for a specific program and is the CDMRP program manager’s main point of contact throughout the peer review process. CDMRP reports that the science review manager provides quality assurance for contractor performance, supervises the recruitment of scientist reviewers by the scientific review officers (SROs), and monitors pre-meeting critiques, panel discussions, and summary statement writing. The science review manager also coordinates online training and webinar orientations for the scientist reviewers who are recruited by the SROs. The consumer review administrator is responsible for consumer

outreach, recruitment, training, and obtaining mentors for each research program (Salzer, 2016c).

The peer review contactor assigns SROs to specific peer review panels for which they serve as panel coordinators; SROs generally support the same program or areas of research each year. SROs, who are subcontractors to the peer review contractor and not members of the integrated program team (Salzer, 2016a), are required to have appropriate experience in health science or biological science research administration for leading scientific peer review, preferably with experience in coordinating scientific review in the particular research area under consideration (e.g., spinal cord injury) (CDMRP, 2011).

The primary responsibilities of SROs are recruiting scientist reviewers for each panel, monitoring reviewer critiques for quality and compliance with program needs, co-moderating the panel discussion with the panel chair, and preparing the summary statements. Prior to each review cycle, SROs attend an orientation webinar at which the scientific review manager, in conjunction with the CDMRP program manager, provides information on the program announcements and peer review panels.

REVIEW PANELS

Peer review and programmatic panels have many similar attributes, such as the use of scientist and consumer reviewers, their criteria for defining COIs, and reviewer compensation. Other aspects of the panels differ, such as the processes used for panel member selection and recruitment as well as the training and length of service of the panel members. Each of these aspects is discussed below, where both panels are discussed, with programmatic panels presented first because they are engaged first in the review cycle.

Panel Composition and Selection

For each research program, there is one programmatic panel and at least one peer review panel (most programs have several; see Chapter 5). Both types of panels have members with scientific and technical expertise in the research program area, and at least one (and usually more) consumer representative. In 2014, a total of 309 programmatic reviewers and 3,195 peer reviewers participated in CDMRP’s review process.

The size and membership of programmatic and peer review panels for a research program vary to some extent by year and are influenced by program-specific needs, including the available funding, the number of award mechanisms released, the expected number of applications, and the needed expertise. There are no restrictions on the number or types

of representatives (scientific, consumer, federal, or military) from one organization or on the number of different organizations that may be represented on either the programmatic panel or the peer review panels. For example, on the 2016 amyotrophic lateral sclerosis programmatic panel, there are two representatives from the ALS Association, one from the headquarters office and one from a state chapter.

The CDMRP program manager works with the contractors to help ensure that qualified people are asked to serve on the programmatic and peer review panels. Each CDMRP program manager coordinates with the contractor’s scientific review manager and other members of the integrated program team to develop detailed program-specific task order assumptions that are used by the contractors to guide the composition of the programmatic and peer review panels (Salzer, 2016a). The program manager’s task order assumptions include the award mechanisms for the program, the number of anticipated pre-applications or letters of intent (if applicable), the estimated number of full applications to be received, and the number of applications to be assigned to an individual reviewer. A program timeline identifies all relevant deadlines and milestones for each step of the program cycle. All programmatic and peer review panel members are recruited by the respective contractors and approved by the CDMRP program manager.

Programmatic and peer review panels are composed of scientist and consumer reviewers, all panels have a chair, and both panels may also include ad hoc and specialty reviewers, depending on the number and types of applications to be reviewed. All members of peer review and programmatic panels (current and past), with the exception of ad hoc and specialty reviewers, are listed on each CDMRP research program’s webpage. In the sections that follow, the committee discusses the selection and role of these panel members.

Programmatic Panels¹

Programmatic panels have a diverse membership, including academic researchers, clinicians, other scientists (e.g., researchers from the pharmaceutical and biotechnology industries), government researchers, consumers, and program specialists (e.g., administrators), often from organizations that conduct similar research (see Table 3-1). These organizations

___________________

¹ The committee notes that CDMRP has recently standardized its terminology to describe the panels that conduct vision setting and programmatic review such that all of them are now called “programmatic panels” (formerly also called integration panels, steering committees, or working groups depending on the research program) (Salzer, 2016a). The committee finds this to be a good example of standardizing aspects of the research programs.

TABLE 3-1 Number and Types of Programmatic Reviewers Across All CDMRP Programs

^a Categorizations were made by study staff and are subjective based on the reviewer’s reported affiliation on the CDMRP website.

^b The first number is the number of consumer advocates that served as programmatic reviewers for a program, the number in parenthesis is the number of different advocacy groups that are represented.

^c Programmatic reviewers for DMRDP are indicated by JPC and topic area. In 2014 there were five panels, comprising almost 150 programmatic reviewers, although many reviewers sat on more than one panel.

^d Only 2015 panel information available.

^e Not including external advisory board members.

include the many institutes of NIH as well as VA, the Food and Drug Administration (FDA), and the Centers for Disease Control and Prevention, among others, as well as nonprofit organizations such as Susan G. Komen and the National Multiple Sclerosis Society.

The programmatic panels for research programs that have a greater focus on specific military (and veteran) health conditions, such as the Spinal Cord Injury Research Program, always include representatives from the military (e.g., the Departments of the Army, Air Force, and Navy; the Defense Health Agency; and the Uniformed Services University of the Health Sciences). Research programs that are strongly aligned with DoD joint program committees (JPCs), such as the Psychological Health/Traumatic Brain Injury Research Program, or that otherwise have a strong military health focus may have programmatic panels composed almost entirely of military representatives and federal government employees (e.g., NIH, VA). Programmatic panels for those programs may also have only a few consumer representatives, as their participation is at the discretion of the JPC (Santullo, 2016); however, consumers are a part of every peer review panel for those programs. Conversely, research programs that are not heavily focused on active-duty military health, such as the Autism Research Program, may have little or no representation from the military on the programmatic panels.

The CDMRP program manager works with the programmatic panel contractor to help ensure that qualified people are asked to serve on programmatic panels. CDMRP program managers use a variety of resources to make decisions about whom to engage on the programmatic panels. These resources include literature searches, personal relationships, the consideration of a person’s participation on peer review panels, and the program manager’s attendance at scientific and advocacy meetings about the health condition (Salzer, 2016b). The CDMRP program manager may, but is not required to, reach out to other federal agencies, such as NIH, FDA, and other parts of DoD, to request that they provide a representative for the programmatic panel.

Peer Review Panels

Peer review panels are composed predominantly of scientific experts (e.g., academics, clinicians, researchers) who focus on a particular health condition or an aspect of a condition (e.g., epidemiology, genetics, or biostatistics) plus consumer reviewers representing organizations or groups that are focused on the health condition. As with programmatic panels, representation on peer review panels spans government organizations, academia, industry, and consumer organizations, with most reviewers for research programs that lack a heavy focus on military health being

from academia (see Table 3-2). For example, in 2014 the Amyotrophic Lateral Sclerosis Research Program had two peer review panels (Salzer, 2016c) with no federal representation, whereas the Psychological Health/Traumatic Brain Injury Research Program had substantial representation from both VA and DoD. Of the 25 programs, 5 did not list the peer reviewers on the CDMRP website (accessed February 10, 2016).

In contrast to programmatic panels, which typically have only small changes in membership each year, all peer review panels are recruited de novo each year. The process of selecting peer reviewers begins prior to the release of the program announcements for that fiscal year (Salzer, 2016b). On the basis of the task order assumptions described earlier, the contractor develops a reviewer specification sheet that details the number of reviewers, areas of expertise, the need for specialty reviewers, the level of education, specific degrees, diversity requirements, overall balance of consumers and scientists, and the proportion of new reviewers. The task order assumptions or reviewer specification sheet may also call for the representation of specific organizations (for example, active military, NIH, or VA reviewers). Contractor SROs recruit the scientist reviewers for their assigned peer review panels (Salzer, 2016c). All peer reviewers, including the panel chair but excluding federal government employees, are subcontractors to the peer review contractor (Salzer, 2016a).

Although there are no specific requirements for demographic diversity on either programmatic or peer review panels (Salzer, 2016a), the peer review contractor summarizes and reports to CDMRP peer reviewer demographics (gender, ethnic diversity, academic rank [junior versus more experienced], military rank, active duty status, education/degrees, and institutional association) in aggregate by program and by panel (CDMRP, 2011).

Scientist Reviewers

CDMRP reports that in the 2015 review cycle more than 2,300 scientists and clinicians served as peer reviewers and nearly 300 served as programmatic reviewers (CDMRP, 2015a). The programmatic contractor is required to recruit “forward-thinking individuals who meet to deliberate the issues and concerns unique to the individual program, propose a vision to guide the investment strategies for the upcoming year, and eventually recommend a broad-based portfolio to cure disease” (Salzer, 2016a). The programmatic review contractor stated that although it has no basic set of qualifications for scientists to serve as programmatic panel members, nevertheless panel members are selected for their “expertise, ability to represent their respective communities to aid the program in achieving its specific goals, . . . are representatives of the highest echelons

TABLE 3-2 Number and Types of Peer Reviewers Across All CDMRP Programs^a

^a Categorizations were made by study staff and are subjective based on the reviewer’s reported affiliation on the CDMRP website.

^b This list does not include four programs that were administered by CDMRP on behalf of others.

^c The first number is the number of consumer advocates who served as peer reviewers for a program, and the number in parenthesis is the number of different advocacy groups that are represented.

^d Denotes programs in which peer reviewer information was not listed for 2014.

of their respective fields, and . . . the scientific panel members generally have excellent publication and funding records” (Salzer, 2016d). During the vision setting and programmatic review meetings, scientist members of the programmatic panels are expected to actively participate in the discussions of the research program goals, the investment strategy, and the review of the applications for programmatic relevance.

The peer review contractor states that the minimum requirements to serve as a scientist peer reviewer are a doctoral degree; an academic rank (or equivalent) of assistant professor or above; being an active researcher; and being an independent researcher (Salzer, 2016d). Furthermore, the contractor is expected to recruit reviewers with the

Consumer Reviewers

Consumer reviewers are an integral component of the vision setting, peer review, and programmatic review steps in the CDMRP process. Since 1993, more than 2,000 consumers have served as peer and programmatic reviewers (Salzer, 2016a) and between one and four consumer representatives sit on all review panels. CDMRP defines consumers as “patients, survivors, family members, or caregivers of people living with a disease, injury, or condition, who serve as representatives of consumer advocacy, support, or military organizations” (CDMRP, 2015a). CDMRP reports that consumer reviewers bring a sense of urgency to all levels of decision making, provide a unique perspective on disease- or condition-related issues important to that community, and serve as liaisons between their constituencies and the scientific community (CDMRP, 2016d; Kaime et al., 2010).

Box 3-1 outlines the eligibility requirements for serving as a consumer reviewer for either a peer review or programmatic panel; however, the specific requirements to serve as a consumer reviewer vary by research program. For example, for the Breast Cancer Research Program a consumer reviewer must be a “survivor or person living with breast cancer,” whereas for the Peer Reviewed Alzheimer’s Research Program the requirement is for “persons with military service living with, or family members/caregivers of a person living with, Alzheimer’s disease (AD) or mild cognitive impairment (MCI). Persons without military service will be considered” (CDMRP, 2016e,f). Many members of the committee thought requiring that a consumer reviewer have or be a survivor of the health condition or be a family member or caretaker of such a person may

be restrictive, because many people who work for or volunteer with an advocacy organization may be familiar with more phases of the condition or have broader experience with it. However, other committee members thought the current CDMRP definition is appropriate.

The CDMRP website has several pages devoted to explaining how consumer reviewers are identified, nominated, evaluated, and selected to serve on either programmatic or peer review panels (CDMRP, 2016e). Briefly, consumer reviewers for programmatic panels are recruited by contacting advocacy and consumer organizations that cover broad aspects of the health condition including support groups, patient or provider education, health policy, and treatment, and asking them to nominate a member to participate on a panel. Consumer reviewers for programmatic panels may also be recruited from among those who have served previously as peer reviewers. The programmatic contractor states that it “selects well-known, well-established advocates for their disease” and that consumer representatives constitute 20–25% of programmatic panel membership (Salzer, 2016d).

Because all peer review panel members, including consumers, are newly recruited each year, the process for obtaining them is more involved. It is also detailed on the CDMRP website. The major difference

in the process of obtaining peer review versus programmatic consumer reviewers is the role of the peer review contractor’s consumer review administrator. This person is responsible for recruiting consumers and works closely with the CDMRP program manager to develop and implement the consumer outreach and recruitment plans for each research program, including strategies for recruiting consumers from minority groups to help achieve equity among ethnic and cultural consumer representation. The extensive nomination packages are reviewed by the consumer review administrator, and potential nominees are interviewed via teleconference to assess their knowledge, expertise, experience, and ability to represent the consumer stakeholder community (CDMRP, 2011). The consumer review administrator then selects candidates and presents them to the program manager for approval or revision (Salzer, 2016a). Once approved, the SRO and the consumer review administrator are the main points of contact for consumer reviewers during the peer review process.

The committee was not informed how the CDMRP program managers and the contractors determine which consumer organizations will be represented on either the programmatic or peer review panels. CDMRP reports that the “contractor and the program manager strive to have different points of view on the programmatic panels” (Salzer, 2016a), but the committee is unclear on how disparate consumer viewpoints are assessed or included. As many consumer and advocacy groups have diverse points of view on what research should be conducted and by whom and its relevance to unique populations (e.g., minorities), this may be an important concern.

Panel Chairs

Programmatic and peer review panel chairs work closely with the CDMRP program manager and contractors to ensure that the respective meetings run smoothly and to encourage all panel members to participate in the deliberations. The programmatic panel chair may take part in teleconferences throughout the year to discuss issues, suggest new panel members, and recommend speakers and background documents for the vision setting meeting (Salzer, 2016b). The programmatic panel chair acts as the main manager of the panel during vision setting, pre-application screening, and programmatic review; the panel chair may be either a scientist or a consumer.

At the conclusion of the programmatic review meeting, the contractor calls for nominations for a new panel chair. Current panel members nominate a chair from among themselves (including nominating the current chair). The contractor verifies that the nominee would like to be a candidate for chair. If only one person is nominated, then he or she

becomes chair for the next year; otherwise, the chair is selected by majority vote (Salzer, 2016a). The committee is unclear as to whether the vote is conducted anonymously.

Each peer review panel has a chair who is required to be a scientist with expertise in the specific panel topic and to have experience with peer review. The CDMRP program manager must approve the peer review contractor’s list of recruited chairs for each program panel (Salzer, 2016a). For programs closely aligned with a JPC, CDMRP program staff may consult with other authorities (e.g., the Army) on the list of candidate chairs before a selection is made (CDMRP, 2011). The chair works closely with the contractor SRO and may assist with the recruitment of other peer review panel members.

Specialty and Ad Hoc Reviewers

Specialty reviewers are experts in a particular scientific field such as a bioethicist, whose expertise is required to ensure the robust review of an application. Specialty reviewers are used particularly for peer review panels where the review criteria for an award mechanism may be “out of the scope of expertise of the rest of the panel” (Salzer, 2016d), such as clinical trials, research consortia, and therapeutic development awards (Kaime et al., 2010). In the case of a bioethicist, for example, he or she may review an application for a clinical trial where expertise is needed to evaluate a consent form. The decision to use specialty reviewers is made by the CDMRP program manager on the basis of the criteria for the award mechanism and communicated to the contractor science review manager via the task order assumptions. Programmatic panels rarely use specialty reviewers (Salzer, 2016d).

Both programmatic and peer review panels use ad hoc reviewers. For programmatic panels, ad hoc reviewers provide added expertise and decrease the workload of programmatic panel members when there are a large number of pre-applications or application summary statements to evaluate. Ad hoc reviewers vote as full panel members during programmatic review, but they do not participate in vision setting (Salzer, 2016d). As of 2015, more than 150 scientists and clinicians had served as ad hoc programmatic reviewers (CDMRP, 2015a).

The need for ad hoc peer reviewers depends on the expertise of the panel as “recruited by the contractor SRO and the chair” (Salzer, 2016d). The contractor consults with the program manager when such a need arises. For peer review, ad hoc panel members “participate on the panel only for their assigned applications, giving their evaluation and scores” (Salzer, 2016d). Once their applications have been presented, they are dismissed from the panel and do not contribute to the discussion of

other applications; however, their scores and critique of an application are included in the summary statement for programmatic review. Thus, ad hoc reviewers are typically scientist reviewers who are not formally panel members but who provide specific expertise for a selected number of applications.

CDMRP states that ad hoc reviewers’ names are published online (Salzer, 2016b), but as of July 2016 the committee was able to identify only four programs that listed ad hoc programmatic reviewers (breast cancer, tuberous sclerosis, prostate cancer, and multiple sclerosis) on the CDMRP website in 2015 and only one in 2016. At the committee’s request, CDMRP provided a table of the total number of peer reviewers by research program for 2014. A comparison of the table with the lists of 2014 peer reviewers on the CDMRP website showed 639 more peer reviewers in the table than on the website; the committee assumed these additional reviewers were unidentified specialty and ad hoc reviewers. The committee notes that there is little information available on the selection and use of ad hoc reviewers for any research program, although the committee did hear at its open sessions that ad hoc reviewers are commonly used.

Length of Service

Depending on the program, programmatic panel members serve terms of at least 2 years (Salzer, 2016d) and panel chairs typically serve for 1–3 years, but there is no limit on the number of terms that members or chairs can serve or on their consecutive years of service (Salzer, 2016a); a few chairs have served for as many as 4 years and up to 7 years. Terms of service for panel members begin with the vision setting meeting (training has occurred prior to the meeting) and conclude when the programmatic review meeting is over and the recommendations for funding have been made (Step 5, see Figure 2-4).

Programmatic panel members are polled annually to see if they would like to leave the panel (Salzer, 2016b) and there is annual turnover of both the scientist and consumer members of programmatic panels (see Table 3-3). The committee notes, however, that some programmatic panel

TABLE 3-3 Percentage of New Programmatic Reviewers Across All Programs by Fiscal Year

SOURCE: Salzer, 2016c.

members, both scientists and consumers, may have served on a panel for several years. Because these panels are responsible for determining program relevance and portfolio balance, some continuity of panel members each year can enhance the panels’ understanding of the mission, goals, and evolution of the program; the program’s previous investment strategies and rationale; and the prior research awards and outcomes. The service term of a programmatic panel member can be renewed if the member has exemplary prior service, offers a unique expertise and background, is at the forefront of the research or advocacy communities, or provides important historical knowledge of the program’s intent and mission (Salzer, 2016a).

The committee appreciates that for some health conditions there may be relatively few organizations that are able to provide consumer representatives for programmatic panels and thus that the turnover of consumer representatives for some research programs is necessarily low. Nevertheless, while the committee finds that the turnover in scientist reviewers is reasonable, it also finds that the overall turnover of consumer reviewers is low. Thus, the committee encourages CDMRP and its contractor to strive to recruit more consumer reviewers from new or existing organizations to improve turnover and expand participation on the programmatic panels.

Unlike the programmatic panels, peer reviewers are recruited to serve a 1-year term, but they too may serve for more than one term. Kaime et al. (2010) reported that limitations are imposed on reviewers returning for multiple years, but CDMRP did not provide additional details to the committee.

A new peer reviewer is defined as a scientist or consumer who has never served on a peer review panel for a specific program, although he or she may have served as a peer reviewer for a different program (Salzer, 2016b). The peer review contractor collects statistics on the panels, including participation on a panel as a new or returning reviewer, expertise, and demographic information. Table 3-4 shows the percentage of new peer

TABLE 3-4 Percentage of New Peer Reviewers Across All Programs by Fiscal Year

SOURCE: Salzer, 2016c.

reviewers for all programs from 2011 to 2015 as reported by CDMRP. The committee appreciates CDMRP’s effort to recruit a high percentage of new reviewers each year while maintaining some continuity in panel membership. The committee finds that this level of continuity encourages the development of a group culture in the peer review process, while at the same time allowing for an influx of new ideas and opinions to improve discussions of an application’s scientific merit and impact.

Training

As noted in Chapter 1, the training of peer reviewers and programmatic reviewers is the responsibility of the respective contractors. CDMRP does not own any of the training materials, although it does approve them, and CDMRP program managers participate in the training process (CDMRP, 2011, 2014). Both contractors were unwilling to—and CDMRP was unable to—provide any training materials to the committee, and thus the committee was unable to assess their quality. However, CDMRP staff provided some information on training for members of both panels, including the contractor performance work statements.

Programmatic Panel Members

The programmatic contractor conducts training for all new scientist and consumer members of programmatic panels, including ad hoc members. However, the contractor’s performance work statement does not include any specifics on how or when the contractor is to train programmatic reviewers or on the content of the training (CDMRP, 2014). The CDMRP program manager provides a training briefing about CDMRP and its review process to all potential candidates during the interview process (Salzer, 2016d).

After a programmatic panel candidate has agreed to serve, he or she must complete nondisclosure and confidentiality agreements. Prior to the vision setting meeting, scientist and consumer candidates attend another training session led by the contractor technical coordinator and the CDMRP program manager. The program manager presents all of the candidates with an overview of the research program, including details about the program’s history and background, its current portfolio, its vision and mission, its focus areas, and the review cycle. The program manager also reviews the award mechanisms, specifics of the program announcements, and any other relevant materials that may assist in preparing the members for their role as reviewers. The technical coordinator briefs the members on their responsibilities, COI policies, and program-

matic milestones. All reviewers are educated on the inclusion and importance of consumer participation on the CDMRP panels (Salzer, 2016c).

CDMRP states that the programmatic reviewers, whether scientific or consumer, are expected to be familiar with the research field for the program. It also states that many consumers on programmatic panels serve as leaders of their organizations; are active in the advocacy or survivor community; have previous experience in peer review, public speaking, and interacting with scientists; and have considerable knowledge about research on the health condition (Salzer, 2016c). As such, no formal mentoring of consumer panel members is offered (unlike the case for consumers serving on peer review panels) (Salzer, 2016c).

Peer Reviewers

In contrast to the situation with the programmatic contractor, the performance work statement for the peer review contractor does contain specifications regarding how and for whom the contractor is to conduct training and on the content of that training. For example, the contractor is to develop an annual training and orientation plan, including timelines, and educational materials (CDMRP, 2011). Separate training is conducted via online modules and webinars for panel chairs, scientist reviewers, ad hoc reviewers, and consumer reviewers. The webinars include an overview of the history of the research program; award mechanisms, corresponding program announcements, and peer review criteria to be used; and the logistics of the peer review panel meeting.

The contractor science review manager coordinates online training and webinar orientations for the scientist reviewers. The SRO is the primary point of contact for scientist reviewers. The contractor also determines which reviewers have completed training, and provides technology support to them for training, accessing assigned applications, and entering critiques and scores. First-time CDMRP reviewers are required to participate in pre-meeting webinars conducted by the program manager; experienced reviewers are also encouraged to attend this training, but it is not required.

All consumer reviewers participate in orientation and training which includes online and live presentations; a handbook that provides an overview of peer review, policies, procedures, and guidelines; and guidance on how to approach reviewing applications. Prior to the panel meeting, consumers participate in a brief orientation session where the program and peer review processes are discussed and where seasoned consumer reviewers from that program share their experiences. Furthermore, new consumer reviewers are mentored by an experienced consumer reviewer who has been identified as outstanding by the contractor staff, the pro-

gram manager, and the panel chair. Mentors, trained by the consumer review administrator (Salzer, 2016b), act as guides to the review process for new consumer reviewers. Mentoring is used for all panels and programs when possible. When a mentor is not available, the consumer review administrator adopts the role. The SRO also works closely with consumer reviewers to assist them with their critiques (CDMRP, 2016f). The committee finds the CDMRP’s practice of mentoring new consumer reviewers to be a good training approach which may be of use to other organizations in which consumers play substantial roles.

Conflict of Interest

There are two areas of potential COI that are addressed in the CDMRP review process: COIs for principal investigators and their institutions who submit applications to CDMRP, and COIs for programmatic and peer reviewers. COI criteria for applicants are referred to in all program announcements and general application instructions, available on the CDMRP website. The general application instructions require applications to “list all individuals other than collaborators and key personnel, who may have a COI in the review of this application” (CDMRP, 2015b). Applicants are further directed to Federal Acquisition Regulation 9.5 for more details on DoD COI policies for organizations and individuals.

COI policies and procedures for reviewers are developed by the programmatic and peer review contractors and approved by CDMRP (Salzer, 2016b). Although the contractors declined to provide any documentation about their COI policies, and the policies are not publicly available, CDMRP did supply some information on COI. Each year, all peer review and programmatic panel members are informed of lobbying restrictions and COI policies (CDMRP, 2015c), receive a reviewer handbook that details those restrictions and policies, and sign a nondisclosure or confidentiality agreement and a COI agreement (Salzer, 2016c). Reviewers are encouraged to self-report any real or perceived COI at any time in the review process. The peer review contractor also uses the eBRAP database to identify and report on any potential COIs between the individuals named in the applications and all reviewers (CDMRP, 2015d). COI policies are reviewed at each panel meeting, at which time panel members provide a signed list of any COIs. If a programmatic or peer reviewer does have a COI with an application, he or she is blocked from its review on the eBRAP database and is recused at the programmatic or peer review meeting when that application is discussed. Although all COIs are reported to the CDMRP program manager for assessment, the ultimate responsibility for COI determinations rests with the U.S. Army Medical Research Acquisition Activity (Salzer, 2016c).

A COI for programmatic panel members is defined as “any relationship with an organization where the reviewer or spouse, minor child, or partner has financial interest; the reviewer is officer, director, partner, employee (or similar); there exists any arrangement concerning reviewer’s prospective employment, financial interest, or other similar association” (Salzer, 2016c).

CDMRP provided more details on what constitutes a COI for peer reviewers (Salzer, 2016c). Potential COIs for peer reviewers are defined as follows:

• investigator on the application;
• close personal relationship with the investigator;
• active, past (within 3 years), or upcoming collaboration or co-published (within 3 years) with the investigator;
• mentor of the applicant;
• provided a letter of recommendation for the application;
• provided technical assistance to the investigator in the preparation of the application;
• may receive professional or personal (financial) gain from the outcomes of the application;
• currently employed at or negotiating prospective employment with the applicant’s institution;
• long-standing scientific differences with the investigator; or
• perceived to be or unable to provide an objective review of the application.

CDMRP reports that peer reviewers identify COIs after they read the abstracts and lists of personnel for all applications to be reviewed by their assigned panel. Any applications for which a reviewer has selected a COI category and entered a brief explanation are reviewed by the SRO prior to assigning applications to specific primary or secondary reviewers. When conflicts of interest are identified, reviewers are blocked from the review of the application, or the application is assigned to another panel, if possible. After application assignments are finalized and released to reviewers, reviewers are once more asked to confirm that there are no COIs (Salzer, 2016c).

Although the committee found the above COI information to be helpful, it is concerned that the COI definitions are not available publicly on the CDMRP website. Other organizations such as NIH and the Patient-Centered Outcomes Research Institute make their COI policies publicly available (NIH, 2015a; PCORI, 2015). The committee finds that following the example of these organizations with regard to posting COI policies

would improve the transparency of the review process and bring CDMRP more in line with other federal government funding organizations.

Compensation

Peer and programmatic reviewers who are not federal government employees (including active-duty service members) are compensated for their time and travel expenses associated with their service on CDMRP review panels. All reviewers who are not federal employees are subcontractors to either the programmatic or the peer review contractor. CDMRP informed the committee that compensation is considered to be an internal business practice of both contractors. Thus, the following information is based on responses from CDMRP to the committee’s questions.

Compensation is intended to cover the time that reviewers spend on pre-meeting activities—e.g., attending webinars and reading training materials; reviewing program announcements, applications, and critiques from other reviewers (for peer reviewers) or summary statements (for programmatic reviewers); and traveling to and from the panel meeting—and participating in the panel meeting itself. For programmatic reviewers, participation in panel meetings may require 1–6 days per review cycle, depending on the program and the award mechanisms; the corresponding time for peer reviewers is about 2.5 days. To decrease costs and time on travel, the majority of pre-application screening meetings for the programmatic panel are held by teleconference (Salzer, 2016b).

CDMRP reports that for onsite (in person) programmatic review, compensation is hourly, and for online and teleconference reviews, reviewers are compensated per application. Ad hoc programmatic reviewers are paid an hourly rate commensurate with that of the rest of the panel, whereas ad hoc peer reviewers, who are typically called in for fewer applications, are compensated per application. The compensation is the same for scientist and consumer reviewers and for panel chairs (Salzer, 2016b).