THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
Division on Engineering and Physical Sciences
500 Fifth Street, NW Ninth Floor Washington, DC 20001 Phone: 202 334 3286 Fax: 202 334 3373 www.national-academies.org/deps
August 31, 2004
Dr. Carolyn Mercer
Ultra-Efficient Engine Technology Project Office
NASA John H.Glenn Research Center at Lewis Field
Mail Stop 60–2, Code 2100, Bldg. 60, Room 105 21000 Brookpark Road Cleveland, OH 44135–3191
Dear Dr. Mercer:
Please find attached the panel grades and associated discussion for the August 10–11, 2004 meeting of the National Research Council (NRC) Panel for the Review of Proposals for NASA’s Intelligent Propulsion Systems Foundation Technology (IPSFT) Program held at the Keck Center of the National Academies. The meeting was chaired by Dr. Alan Eckbreth, Vice President, Connecticut Academy of Science and Engineering; Dr. Karen Harwell of the Aeronautics and Space Engineering Board served as the study director. This activity was supported by Contract No. NASW-03009 between the National Academies and the National Aeronautics and Space Administration (NASA), and was performed under the direction of the Aeronautics and Space Engineering Board.
SELECTION OF PANELISTS
Panelists were selected on the basis of their expertise in the fields represented by the proposals; they came from universities, industry, and government laboratories. Panel members were screened before nomination regarding their present relationship with NASA. See Attachment A for the panel roster.
PROPOSAL EVALUATION PROCESS
All reviewers were asked to provide general comments regarding a given proposal and were asked to answer the following specific questions in order to promote discussion:
-
What will the results of this work be and how novel are they?
-
How will the results advance the state of the art and how significant will the advancement be?
-
Will the proposed approach produce the desired results? What are its strengths and weaknesses?
In addition to the principal questions above regarding the scientific and technical merit of each proposal, the reviewers were asked to provide opinions on the key personnel’s qualifications, capabilities, and related experience; the adequacy of facilities, hardware, simulation tools, etc.;
and the realism and reasonableness of the proposed project cost. These comments play a minor role in the evaluation process but have proven to be useful to program managers for other NRC proposal reviews, particularly when the proposal emanates from a source unfamiliar to them.
The merits of all proposals were discussed by at least three panel members. Most proposals were reviewed in detail by two panel members. For each proposal, a “lead” reviewer was selected based upon his or her degree of familiarity with the technical area. Each lead reviewer was provided with all the other reviews of a given proposal which were used during the discussion during the meeting.
GRADING PROCESS
Proposals were evaluated based on their scientific and technical merit. Written reviews by panelists were used as information by the panel to assist it in its task of assigning a consensus grade (adjectival rating) to each proposal. The grades reflect the overall quality of each proposal relative to the state of research in the field of the turbine engines and to the state of research, as possible, in the foundational areas being considered in the proposals (i.e., flow control, sensors, combustion, smart materials).
The grades apply to the totality of the proposal, not just to its favorable (or unfavorable) aspects. The grades are not conditional on making suggested changes to the proposal; such suggestions are meant only to aid the author in preparing a better proposal in the future. The grading system used for this review process is as follows:
Excellent: |
A comprehensive and thorough proposal of exceptional merit with one or more significant strengths. No deficiency or significant weakness exists. |
Very Good: |
A proposal having no deficiency and which demonstrates overall competence. One or more significant strengths have been found, and strengths outbalance any weaknesses that exist. |
Good: |
A proposal having no deficiency and which shows a reasonably sound response. There may be strengths or weaknesses, or both. As a whole, weaknesses not offset by strengths do not significantly detract from the offeror’s response. |
Fair: |
A proposal having no deficiency and which has one or more weaknesses. Weaknesses outbalance any strengths. |
Poor: |
A proposal that has one or more deficiencies or significant weaknesses that demonstrate a lack of overall competence or would require a major proposal revision to correct. |
I |
Lacks adequate information for determining scientific merit. |
These grades and their accompanying definitions were established by NASA and were included in NASA Research Announcements NNC04ZPR001N and NNC04ZPR002N soliciting proposals.
CONDUCT OF THE REVIEW MEETING
The panel met on August 10–11, 2004 to discuss the 100 proposals under consideration. The meeting attendance is provided in Attachment B. Panel members participated in a composition and balance discussion. As a result, a few members did not participate in the discussions or final evaluations pertaining to various proposals.
Proposals with related technical content were grouped and then reviewed by subsets of the panel, grouped by areas of expertise, in two distinct types of sessions. In the first session of each small group, the lead reviewer assigned to a given proposal provided a short oral summary of the proposal to the panel and then summarized its strengths and weaknesses. Other small group members then contributed their expertise in the form of questions and/or statements regarding the merits of the proposal. Small group leaders facilitated these discussions to ensure that all group members had an opportunity to contribute and that all proposals had a sufficient airing. These sessions for each group of proposals were attended by NASA representatives as well as NRC small groups and staff. In the second session, a closed discussion was held with only the small group and NRC staff in attendance to reach a consensus on the grade for each proposal following the definitions above. The discussion associated with the grade was recorded by the lead reviewer and group leader. Any questions related to the grading were discussed with a larger subset of the panel. This cycle was repeated several times to thoroughly evaluate the 100 proposals under consideration. Prior to breaking into small groups for the majority of the evaluation process, ten proposals were reviewed using this two-session model and grades assigned. This main session discussion provided panelists with a better understanding of the process and assignment of grades. Following this process, each lead reviewer wrote up the discussion for which he or she was responsible. The write-ups were collected by the study director for review and compilation and were then submitted to the Panel Chairman for final review and approval.
RESULTS
The panel’s consensus grades are provided in Attachments C-1 and C-2. In the former, each proposal is identified only by the NRC number; in the latter, each proposal is identified by the NRC number, the NASA proposal number, title, principal investigator, and affiliation.
The discussion summaries for the panel’s consensus grades appear in Attachment D. The discussions are not intended to systematically cover the criteria and factors involved in the ratings, but aim to advise NASA on ways in which the proposed research might be improved or take better account of other research. Each discussion cites the strengths and weaknesses of the proposal. Even poor proposals contain some good points and rather than de-motivate a researcher by concentrating on only its weaknesses, the panel tried to find something positive to say about all proposals.
National Academies
Division of Engineering and Physical Science
Aeronautics and Space Engineering Board
ATTACHMENT A, to the Letter Report of the
ASEB Panel for the Review of Proposals for NASA’s Intelligent Propulsion Systems Foundation Technology Program
Meeting of August 10–11, 2004 in Washington, DC
Panel Roster
Dr. Alan Eckbreth, Chair
Vice-President
Connecticut Academy of Science and Engineering
Hartford, CT
Dr. Dilip R.Ballal
Hans Von Chain Distinguished Professor & Director, Von Chain Fuels & Combustion Center
University of Dayton
Dayton, OH
Dr. Thomas Beutner
Program Manager for Turbulent and Rotating Flows
Air Force Office of Scientific Research
Arlington, VA
Dr. Gregory Carman
Professor, Mechanical and Aerospace Engineering and Director, Active Materials Laboratory
University of California, Los Angeles
Los Angeles, CA
Dr. Fred E.Culick
Professor of Mechanical Engineering and Jet Propulsion
California Institute of Technology
Pasadena, CA
Dr. Richard J.Goldstein
Regents’ Professor and James J.Ryan Professor, Department of Mechanical Engineering
University of Minnesota
Minneapolis, MN
Dr. Steven Gorrell
Aerospace Engineer
Air Force Research Laboratory
Dayton, OH
Dr. Kenneth C.Hall
Professor and Chair, Dept. of Mechanical Engineering & Materials Science
Duke University
Durham, NC
Mr. Ambros S.Hoffmann
Chief Engineer, Advanced Technology (retired)
Honeywell, Inc.
Phoenix, AZ
Mr. Robert Luppold
Owner and Chief Engineer
Luppold & Associates, Inc.
West Newton, PA
Dr. Lourdes Quintana Maurice
Chief Scientific and Technical Advisor for Environment
Office of Environment and Energy
Federal Aviation Administration
Washington, DC
Dr. George A.Richards
Focus Area Leader, Energy System Dynamics
Department of Energy—National Energy Technology Center
Morgantown, WV
Dr. W.Melvyn Roquemore
Senior Scientist, Combustion
Air Force Research Laboratory
Dayton, OH
Dr. Klaus Schadow
Consulting Combustion & Propulsion Expert
Schadow Technology
San Clemente, CA
Dr. Mary H.Young
Director, Sensors and Materials (retired)
HRL Laboratories
Calabasas, CA
National Academies
Division of Engineering and Physical Science
Aeronautics and Space Engineering Board
ATTACHMENT B, to the Letter Report of the
ASEB Panel for the Review of Proposals for NASA’s Intelligent Propulsion Systems Foundation Technology Program
Meeting of August 10–11, 2004 in Washington, DC
Meeting Attendance
Panelists:
Dr. Alan Eckbreth, Chair
Vice-President
Connecticut Academy of Science and Engineering
Hartford, CT
Dr. Dilip R.Ballal
Hans Von Chain Distinguished Professor & Director, Von Ohain Fuels & Combustion Center
University of Dayton
Dayton, OH
Dr. Thomas Beutner
Program Manager for Turbulent and Rotating Flows
Air Force Office of Scientific Research
Arlington, VA
Dr. Gregory Carman
Professor, Mechanical and Aerospace Engineering and Director, Active Materials Laboratory
University of California, Los Angeles
Los Angeles, CA
Dr. Fred E.Culick
Professor of Mechanical Engineering and Jet Propulsion
California Institute of Technology
Pasadena, CA
Dr. Richard J.Goldstein
Regents’ Professor and James J.Ryan Professor, Department of Mechanical Engineering
University of Minnesota
Minneapolis, MN
Dr. Steven Gorrell
Aerospace Engineer
Air Force Research Laboratory
Dayton, OH
Dr. Kenneth C.Hall
Professor and Chair, Dept. of Mechanical Engineering & Materials Science
Duke University
Durham, NC
Mr. Ambros S.Hoffmann
Chief Engineer, Advanced Technology (retired)
Honeywell, Inc.
Phoenix, AZ
Mr. Robert Luppold
Owner and Chief Engineer
Luppold & Associates, Inc.
West Newton, PA
Dr. George A.Richards
Focus Area Leader, Energy System Dynamics
Department of Energy—National Energy Technology Center
Morgantown, WV
Dr. W.Melvyn Roquemore
Senior Scientist, Combustion
Air Force Research Laboratory
Dayton, OH
Dr. Klaus Schadow
Consulting Combustion & Propulsion Expert
Schadow Technology
San Clemente, CA
Dr. Mary H.Young
Director, Sensors and Materials (retired)
HRL Laboratories
Calabasas, CA
NASA Program Staff:
Dr. Carolyn Mercer
Ms. Susan Johnson
Mr. John Rohde
Ms. Carol Ginty
Ms. Mary Jo Long-Davis
Mr. Gary Seng
Open Session Attendance:
Mr. Jeff Morris
ASEB Staff:
Dr. Karen Harwell, Senior Program Officer
Mr. Neeraj Gorkhaly, Project Assistant
National Academies
Division of Engineering and Physical Science
Aeronautics and Space Engineering Board
ATTACHMENT C-1, to the Letter Report of the
ASEB Panel for the Review of Proposals for NASA’s Intelligent Propulsion Systems Foundation Technology Program
Meeting of August 10–11, 2004, in Washington, DC
Proposals Summarized by Grade by NRC Number
Excellent
NRC-04–036
NRC-04–041
NRC-04–060
NRC-04–061
NRC-04–063
Very Good
NRC-04–001
NRC-04–004
NRC-04–013
NRC-04–014
NRC-04–015
NRC-04–020
NRC-04–023
NRC-04–028
NRC-04–029
NRC-04–030
NRC-04–034
NRC-04–037
NRC-04–038
NRC-04–039
NRC-04–045
NRC-04–047
NRC-04–048
NRC-04–051
NRC-04–054
NRC-04–055
NRC-04–056
NRC-04–059
NRC-04–068
NRC-04–070
NRC-04–075
NRC-04–082
NRC-04–100
Good
NRC-04–006
NRC-04–009
NRC-04–011
NRC-04–012
NRC-04–016
NRC-04–019
NRC-04–021
NRC-04–022
NRC-04–024
NRC-04–025
NRC-04–026
NRC-04–027
NRC-04–033
NRC-04–035
NRC-04–046
NRC-04–050
NRC-04–052
NRC-04–057
NRC-04–064
NRC-04–071
NRC-04–076
NRC-04–077
NRC-04–080
NRC-04–081
NRC-04–083
NRC-04–085
NRC-04–087
NRC-04–088
NRC-04–090
NRC-04–091
NRC-04–092
NRC-04–095
NRC-04–098
Fair
NRC-04–002
NRC-04–003
NRC-04–005
NRC-04–007
NRC-04–008
NRC-04–010
NRC-04–017
NRC-04–018
NRC-04–031
NRC-04–032
NRC-04–040
NRC-04–043
NRC-04–044
NRC-04–049
NRC-04–053
NRC-04–058
NRC-04–062
NRC-04–066
NRC-04–067
NRC-04–069
NRC-04–072
NRC-04–073
NRC-04–074
NRC-04–078
NRC-04–079
NRC-04–084
NRC-04–086
NRC-04–089
NRC-04–093
NRC-04–094
NRC-04–096
NRC-04–099
NRC-04–101
Poor
NRC-04–042
NRC-04–097
Incomplete
Information
None