Skip to main content

Currently Skimming:

Letter Report
Pages 1-24

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 1...
... The committee was asked to evaluate the scientific merit assessment processes that are applied to directed research tasks2 funded through the HRP and to determine best practices from similar assessment processes that are used in other federal agencies; the detailed statement of task is provided in Box 1. This letter report and its recommendations are the product of a 10-member ad hoc committee, which included individuals who had 1 This study and its statement of task were derived from ongoing conversations between NASA and the IOM's Standing Committee on Aerospace Medicine and the Medicine of Extreme Environments.
From page 2...
...  What metrics should the HRP use to assess the quality of the di rected task merit review process? previously conducted research under the HRP, were familiar with the HRP's research portfolio and operations, had specific knowledge of peer review processes, or were familiar with scientific merit assessment processes used in other organizations and federal agencies, such as the Canadian Institutes of Health Research (CIHR)
From page 3...
... . The workshop was organized into four roundtable discussions that allowed the committee to explore the practices and processes of federal agencies and other organizations in identifying directed research, assessing its scientific merit, monitoring and evaluating the progress of directed tasks, and evaluating the overall directed research processes to ensure high-quality outcomes.4 The workshop also provided the committee with an additional opportunity for an open dialogue with NASA staff to further discuss the HRP merit assessment processes for directed research.
From page 4...
... . Currently, directed research tasks are reviewed using scientific merit assessment processes that are described in detail in the HRP Unique Processes, Criteria, and Guidelines (UPCG)
From page 5...
... NOTE: The process for the program-, element-, and project-led reviews are described in the HRP's Unique Processes, Criteria, and Guidelines document.
From page 6...
... The committee's criteria for rigorous scientific merit assessment processes concur with those put forth by Wood and Wessely -- the review process is "expected to be: effective .
From page 7...
... The committee's overall assessment is that the NASA scientific merit assessment processes for directed research fulfill these characteristics for the most part and are well suited for the operational requirements that they were designed to address. Where opportunities exist for improvement, suggestions are made by the committee throughout the remainder of the report, which covers the processes used to make the initial decisions on whether or not a proposal meets the definition of directed research, the scientific merit assessment processes, and quality improvement (QI)
From page 8...
... . This attention to identifying and then devoting resources to some of its immediate and specific programmatic research gaps through directed research is a strength of the HRP program and provides NASA with a well-organized and responsive research mechanism.
From page 9...
... One way to make the distinction between research tasks and supporting activities is to differentiate between those that are hypothesis-driven and those that are not. Although every task or activity has specific goals, some efforts -- particularly those associated with design, development, testing, and evaluation -- are not hypothesis-driven; they do not predict the answer to specific, original research questions but rather are supporting activities (e.g., pilot tests, data mining, literature searches)
From page 10...
... . The VA participant noted that within the VA, which conducts its research intramurally, directed research can be studies that are either large, through the Cooperative Studies Program, or relatively small in budget and short in length.
From page 11...
... can be directly approved to proceed without a formal scientific merit assessment process. Other agencies do not consider these types of tasks to be directed research that warrants peer review.
From page 12...
... The length of time required for the merit assessment process does not need to be a consideration in these decisions, as agencies agreed that external peer review or other merit assess ment processes can be accomplished quickly when needed with out compromising the quality of the review (see below)
From page 13...
... SCIENTIFIC MERIT ASSESSMENT PROCESS The committee was asked to examine the strengths and weaknesses of the HRP's scientific merit assessment process for directed research and to provide input on the current suite of options for that process. To address these topics the committee looked at the criteria used to assess scientific rigor and then at the processes used for the merit assessment of directed research.
From page 14...
... Processes Used to Conduct Scientific Merit Assessment Although the criteria used to assess scientific merit are similar among the participating agencies and organizations, as noted above, the committee found more variation in the specific processes that are used. As discussed earlier, not all organizations conduct directed research and the scope of what is considered directed research varies.
From page 15...
... The RFAs and RFPs issued by the NIH for targeted research start with a concept approved by the institute director and then, after external peer review is conducted with scoring, the final funding decisions are made by the institute or center director, with input from staff and the advisory board. Depending on the scope and nature of the research, the VA typically uses a mix of internal and external reviewers.
From page 16...
... This fits within the HRP's policy that "all investigations sponsored by the program will undergo independent scientific merit review. This includes proposals submitted in response to NASA Research Announcements, all directed study proposals, and all unsolicited proposals" (NASA, 2011b, p.
From page 17...
...  If needed, many federal agencies and other organizations can complete the scientific merit assessment process, in cluding external peer review, in a timely manner without jeopardizing the quality of the assessment. Many agencies and organizations continue to work to expedite the peer review process by exploring and implementing a variety of online and collaborative approaches.
From page 18...
... Recommendation RECOMMENDATION 3 Streamline the Merit Assessment Process for Directed Research NASA should streamline the merit assessment process for directed research consistent with a narrower definition of di rected research. Decisions regarding supporting activities should be made by the Program Scientist.
From page 19...
... Implementation efforts should  ensure that the Program Scientist has the authority to make decisions regarding supporting activities;  continue to give the Program Scientist the authority to make the final decision to proceed on directed research tasks, taking into consideration peer review findings and his or her assessment of NASA's priorities; and  expedite the merit assessment process for directed re search while also ensuring the high quality of the review process. EVALUATION AND QUALITY IMPROVEMENT At any point during the life of an individual directed research task, the Program Scientist may request a status review.
From page 20...
... Directed research tasks at the HRP have report compliance rates of approximately 75 percent for annual and final reports, which is below the reporting rates for HRP's other types of research (84-93 percent in 2009-2011)
From page 21...
... . In addition to evaluating individual research tasks, several agencies and organizations conduct QI activities that assess the policies and processes involved in identifying and reviewing research tasks.
From page 22...
... Con sistent with Recommendation 2, the HRP may want to consider expanding its documentation of the deliverables and long-term impacts of its directed research tasks, including providing follow-up on completed tasks in its annual reports and on its website, as a part of increasing the transparency of the directed research program.  Continuous QI efforts focused on the HRP merit assessment process are needed to ensure that the HRP has effective processes in place to identify directed research tasks that are feasible and valuable and that have a high probability of success.
From page 23...
... SUMMARY The committee finds that the scientific merit assessment process used by NASA's HRP for directed research is scientifically rigorous and is similar to the processes and merit criteria used by many other federal agencies and organizations for comparable types of research, including the DOD, NIH, NSF, and USDA. The committee notes the complexity of the various merit assessment pathways in the current HRP directed research program and recommends that these be streamlined into one common pathway requiring all directed research proposals to undergo
From page 24...
... Additionally, continuous QI efforts to evaluate and improve the HRP merit assessment process are needed to enable NASA to actively monitor the effectiveness of merit assessment and fund directed research that will be of the highest possible value to its mission in a timely manner. The members of the IOM Committee on the Review of NASA Human Research Program's Scientific Merit Assessment Processes appreciate the opportunity to provide input to the HRP.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.